adding China National People's Congress

[bgmello]

Fixes opensanctions/crawler-planning#170

It's currently throwing an assertion error because it's not creating an entity for each line, but I couldn't find why.

[jbothma]

Let's call this something like cn_wikidata_npc so that if we get an official source one day, we can use this name or similar for it.

[bgmello]

I'm actually using the data from NPC observer, because it was already in a google sheets . Do you want me to extract from the wikipedia or is this ok?

[jbothma]

Oh right - I pasted one table into my google sheet from wikipedia and it seems to work fine https://docs.google.com/spreadsheets/d/1WfvKmydpsyIQe-5V28meRx2miVNCxMAtbtM-taJDFNw/edit#gid=0

I think it's worth using the wikipedia data since it contains dates of birth which help a lot to disambiguate when two people have the same name.

[bgmello]

I added the data from wikipedia to the google sheets. I used the following code to extract the data:

import requests
import pandas as pd

r = requests.get('https://zh.wikipedia.org/wiki/第十四届全国人民代表大会代表名单')
dfs = pd.read_html(r.text)
final_df = []
columns = ['姓名', '政党', '性别', '民族', '出生日期', '职务', '备注']
for i, df in enumerate(dfs[1:36]):
    df.columns = df.columns.get_level_values(0)
    df = df.iloc[:, :7]
    df.columns = columns
    final_df.append(df)
    
final_df = pd.concat(final_df)
final_df.to_excel("china_national_people_congress.xlsx", index=False)

[bgmello]

I'm having trouble with the assertion because there are some cases we don't have enough information to create a unique ID. For example, there are two deputies with the name Zhang Qiang, that are male and ethnicity Han, since we don't have any other relevant information about them like birth date, we create the same ID. What we do in this case?

[jbothma]

well spotted!

It looks like one represents Jiangsu Province and one represents Jiangxi Province.

Let's include the province they represent in the ID

[jbothma]

Could you include your script for scraping the content in the crawler directory? Then it's easy to update.