tlater/scrape-yuanyuan
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
No description
26 commits 1 branch 0 tags 4.1 MiB
Find a file
Tristan Daniël Maat 70b0860e1a
Move the guangdong links
2022-04-09 23:31:52 +01:00
guangdong Move the guangdong links 2022-04-09 23:31:52 +01:00
ningxia Add ningxia file dump 2022-04-09 22:37:27 +01:00
qinghai Add dumped qinghai articles 2022-04-09 19:57:08 +01:00
shanxi Implement shanxi scraping 2022-04-09 23:06:53 +01:00
utils Implement scrape utils 2022-04-09 23:06:46 +01:00
.gitignore Ignore article directories 2022-04-09 22:35:15 +01:00
flake.lock Initial commit 2022-04-09 14:44:18 +01:00
flake.nix Add linkutils 2022-04-09 22:33:43 +01:00
README.md shanxi: Update page numbers 2022-04-09 23:06:27 +01:00

README.md

Province article scraping

A couple of scripts to scrape article text from various provinces for a text analysis university course.

We need:

Qinghai
page 14-75
Ningxia
page 11-42 (actually 8-44?)
Shanxi
page 2-18 (actually 2-20?)
Xinjiang
page 10-20

The websites all have subtle differences, so there's simply a folder + scripts for each (the scripts are simple enough that there's no need for deduplication or anything complex). Written in python/js where necessary for educational purposes.