Go to file

Tristan Daniël Maat 8820ce1b95 Add ningxia file dump		2022-04-09 22:37:27 +01:00
guangdong	Add typescript-language-server	2022-04-09 17:43:37 +01:00
ningxia	Add ningxia file dump	2022-04-09 22:37:27 +01:00
qinghai	Add dumped qinghai articles	2022-04-09 19:57:08 +01:00
utils	Add linkutils	2022-04-09 22:33:43 +01:00
.gitignore	Ignore article directories	2022-04-09 22:35:15 +01:00
README.md	Update page numbers for ningxia	2022-04-09 22:32:13 +01:00
flake.lock	Initial commit	2022-04-09 14:44:18 +01:00
flake.nix	Add linkutils	2022-04-09 22:33:43 +01:00

README.md

Province article scraping

A couple of scripts to scrape article text from various provinces for a text analysis university course.

We need:

Qinghai: page 14-75
Ningxia: page 11-42 (actually 8-44?)
Shanxi: page 2-18
Xinjiang: page 10-20

The websites all have subtle differences, so there's simply a folder + scripts for each (the scripts are simple enough that there's no need for deduplication or anything complex). Written in python/js where necessary for educational purposes.