{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 利用信息抽取技术搭建知识库\n",
"\n",
"在这个notebook文件中，有些模板代码已经提供给你，但你还需要实现更多的功能来完成这个项目。除非有明确要求，你无须修改任何已给出的代码。以**'【练习】'**开始的标题表示接下来的代码部分中有你需要实现的功能。这些部分都配有详细的指导，需要实现的部分也会在注释中以'TODO'标出。请仔细阅读所有的提示。\n",
"\n",
">**提示：**Code 和 Markdown 区域可通过 **Shift + Enter** 快捷键运行。此外，Markdown可以通过双击进入编辑模式。\n",
"\n",
"---\n",
"\n",
"### 让我们开始吧\n",
"\n",
"本项目的目的是结合命名实体识别、依存语法分析、实体消歧、实体统一对网站开放语料抓取的数据建立小型知识图谱。\n",
"\n",
"在现实世界中，你需要拼凑一系列的模型来完成不同的任务；举个例子，用来预测狗种类的算法会与预测人类的算法不同。在做项目的过程中，你可能会遇到不少失败的预测，因为并不存在完美的算法和模型。你最终提交的不完美的解决方案也一定会给你带来一个有趣的学习经验！\n",
"\n",
"\n",
"---\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 1：实体统一"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"实体统一做的是对同一实体具有多个名称的情况进行统一，将多种称谓统一到一个实体上，并体现在实体的属性中（可以给实体建立“别称”属性）\n",
"\n",
"例如：对“河北银行股份有限公司”、“河北银行公司”和“河北银行”我们都可以认为是一个实体，我们就可以将通过提取前两个称谓的主要内容，得到“河北银行”这个实体关键信息。\n",
"\n",
"公司名称有其特点，例如后缀可以省略、上市公司的地名可以省略等等。在data/dict目录中提供了几个词典，可供实体统一使用。\n",
"- company_suffix.txt是公司的通用后缀词典\n",
"- company_business_scope.txt是公司经营范围常用词典\n",
"- co_Province_Dim.txt是省份词典\n",
"- co_City_Dim.txt是城市词典\n",
"- stopwords.txt是可供参考的停用词\n",
"\n",
"### 练习1：\n",
"编写main_extract函数，实现对实体的名称提取“主体名称”的功能。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import jieba\n",
"import jieba.posseg as pseg\n",
"import re\n",
"import datetime\n",
"\n",
"\n",
"# 从输入的“公司名”中提取主体\n",
"def main_extract(input_str,stop_word,d_4_delete,d_city_province):\n",
" # 开始分词并处理\n",
" seg = pseg.cut(input_str)\n",
" seg_lst = remove_word(seg,stop_word,d_4_delete)\n",
" seg_lst = city_prov_ahead(seg,d_city_province)\n",
" return seg_lst\n",
"\n",
" \n",
"#TODO：实现公司名称中地名提前\n",
"def city_prov_ahead(seg,d_city_province):\n",
" city_prov_lst = []\n",
" # TODO ...\n",
" for word in seg:\n",
" if word in d_city_province:\n",
" city_prov_lst.append(word)\n",
" seg_lst = [word for word in seg if word not in city_prov_lst] \n",
" return city_prov_lst+seg_lst\n",
"\n",
"\n",
"\n",
"\n",
"#TODO：替换特殊符号\n",
"def remove_word(seg,stop_word,d_4_delete):\n",
" # TODO ...\n",
" filter_stop_word = [ word for word, flag in seg if word not in stop_word]\n",
" seg_lst = [word for word in filter_stop_word if word not in d_4_delete] \n",
" return seg_lst\n",
"\n",
"\n",
"# 初始化，加载词典\n",
"def my_initial():\n",
" fr1 = open(r\"../data/dict/co_City_Dim.txt\", encoding='utf-8')\n",
" fr2 = open(r\"../data/dict/co_Province_Dim.txt\", encoding='utf-8')\n",
" fr3 = open(r\"../data/dict/company_business_scope.txt\", encoding='utf-8')\n",
" fr4 = open(r\"../data/dict/company_suffix.txt\", encoding='utf-8')\n",
" #城市名\n",
" lines1 = fr1.readlines()\n",
" d_4_delete = []\n",
" d_city_province = [re.sub(r'(\\r|\\n)*','',line) for line in lines1]\n",
" #省份名\n",
" lines2 = fr2.readlines()\n",
" l2_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines2]\n",
" d_city_province.extend(l2_tmp)\n",
" #公司后缀\n",
" lines3 = fr3.readlines()\n",
" l3_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines3]\n",
" lines4 = fr4.readlines()\n",
" l4_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines4]\n",
" d_4_delete.extend(l4_tmp)\n",
" #get stop_word\n",
" fr = open(r'../data/dict/stopwords.txt', encoding='utf-8') \n",
" stop_word = fr.readlines()\n",
" stop_word_after = [re.sub(r'(\\r|\\n)*','',stop_word[i]) for i in range(len(stop_word))]\n",
" stop_word_after[-1] = stop_word[-1]\n",
" stop_word = stop_word_after\n",
" return d_4_delete,stop_word,d_city_province\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Building prefix dict from the default dictionary ...\n",
"Loading model from cache /var/folders/dm/8cmwvdq13cncz81zzz5vht5c0000gn/T/jieba.cache\n",
"Loading model cost 0.680 seconds.\n",
"Prefix dict has been built successfully.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"# TODO：测试实体统一用例\n",
"d_4_delete,stop_word,d_city_province = my_initial()\n",
"company_name = \"河北银行股份有限公司\"\n",
"lst = main_extract(company_name,stop_word,d_4_delete,d_city_province)\n",
"company_name = ''.join(lst) # 对公司名提取主体部分，将包含相同主体部分的公司统一为一个实体\n",
"print(company_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 2：实体识别\n",
"有很多开源工具可以帮助我们对实体进行识别。常见的有LTP、StanfordNLP、FoolNLTK等等。\n",
"\n",
"本次采用FoolNLTK实现实体识别，fool是一个基于bi-lstm+CRF算法开发的深度学习开源NLP工具，包括了分词、实体识别等功能，大家可以通过fool很好地体会深度学习在该任务上的优缺点。\n",
"\n",
"在‘data/train_data.csv’和‘data/test_data.csv’中是从网络上爬虫得到的上市公司公告，数据样例如下："
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>sentence</th>\n",
" <th>tag</th>\n",
" <th>member1</th>\n",
" <th>member2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>6461</td>\n",
" <td>与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2111</td>\n",
" <td>三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>9603</td>\n",
" <td>2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515...</td>\n",
" <td>1</td>\n",
" <td>协鑫集成科技股份有限公司</td>\n",
" <td>瑞峰（张家港）光伏科技有限公司</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3456</td>\n",
" <td>证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>8844</td>\n",
" <td>本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数...</td>\n",
" <td>1</td>\n",
" <td>广发证券股份有限公司</td>\n",
" <td>辽宁成大股份有限公司</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id sentence tag member1 \\\n",
"0 6461 与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市... 0 0 \n",
"1 2111 三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无... 0 0 \n",
"2 9603 2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515... 1 协鑫集成科技股份有限公司 \n",
"3 3456 证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限... 0 0 \n",
"4 8844 本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数... 1 广发证券股份有限公司 \n",
"\n",
" member2 \n",
"0 0 \n",
"1 0 \n",
"2 瑞峰（张家港）光伏科技有限公司 \n",
"3 0 \n",
"4 辽宁成大股份有限公司 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
"train_data.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>sentence</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>9259</td>\n",
" <td>2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>9136</td>\n",
" <td>2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>220</td>\n",
" <td>2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>9041</td>\n",
" <td>2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>10041</td>\n",
" <td>一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id sentence\n",
"0 9259 2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》\n",
"1 9136 2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先\n",
"2 220 2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》\n",
"3 9041 2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...\n",
"4 10041 一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收..."
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"test_data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"我们选取一部分样本进行标注，即train_data，该数据由5列组成。id列表示原始样本序号；sentence列为我们截取的一段关键信息；如果关键信息中存在两个实体之间有股权交易关系则tag列为1，否则为0；如果tag为1，则在member1和member2列会记录两个实体出现在sentence中的名称。\n",
"\n",
"剩下的样本没有标注，即test_data，该数据只有id和sentence两列，希望你能训练模型对test_data中的实体进行识别，并判断实体对之间有没有股权交易关系。\n",
"\n",
"### 练习2：\n",
"将每句句子中实体识别出，存入实体词典，并用特殊符号替换语句。\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# 处理test数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
"\n",
"import fool\n",
"import pandas as pd\n",
"from copy import copy\n",
"\n",
"\n",
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"test_data['ner'] = None\n",
"ner_id = 1001\n",
"ner_dict_new = {} # 存储所有实体\n",
"ner_dict_reverse_new = {} # 存储所有实体\n",
"\n",
"for i in range(len(test_data)):\n",
" sentence = copy(test_data.iloc[i, 1])\n",
" # TODO：调用fool进行实体识别，得到words和ners结果\n",
" # TODO ...\n",
" \n",
" words, ners = fool.analysis(sentence)\n",
" ners[0].sort(key=lambda x:x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type=='company' or ner_type=='person':\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
" company_main_name = main_extract(ner_name, stop_word, d_4_delete, d_city_province)\n",
" company_main_name = ''.join(lst)\n",
" if company_main_name not in ner_dict_new:\n",
" # ner_id 从 1001开始\n",
" ner_id += 1\n",
" ner_dict_new[company_main_name] = ner_id\n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end-1:]\n",
" test_data.iloc[i, -1] = sentence\n",
"\n",
"X_test = test_data[['ner']]\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"# 处理train数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
"train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
"train_data['ner'] = None\n",
"\n",
"for i in range(len(train_data)):\n",
" # 判断正负样本\n",
" if train_data.iloc[i, :]['member1'] == '0' and train_data.iloc[i, :]['member2'] == '0':\n",
" sentence = copy(train_data.iloc[i, 1])\n",
" # TODO：调用fool进行实体识别，得到words和ners结果\n",
" # TODO ...\n",
" words, ners = fool.analysis(sentence)\n",
" ners[0].sort(key=lambda x: x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type == 'company' or ner_type == 'person':\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
"\n",
" company_main_name = main_extract(ner_name, stop_word, d_4_delete, d_city_province)\n",
" company_main_name = ''.join(lst) # 对公司名提取主体部分，将包含相同主体部分的公司统一为一个实体\n",
"\n",
" if company_main_name not in ner_dict_new:\n",
" ner_id += 1\n",
" ner_dict_new[company_main_name] = ner_id\n",
"\n",
"\n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end - 1:]\n",
" train_data.iloc[i, -1] = sentence\n",
" else:\n",
" # 将训练集中正样本已经标注的实体也使用编码替换\n",
" sentence = copy(train_data.iloc[i, :]['sentence'])\n",
" for company_main_name in [train_data.iloc[i, :]['member1'], train_data.iloc[i, :]['member2']]:\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
"\n",
" company_main_name_new = main_extract(company_main_name, stop_word, d_4_delete, d_city_province)\n",
" company_main_name_new = ''.join(lst) # 对公司名提取主体部分，将包含相同主体部分的公司统一为一个实体\n",
"\n",
" if company_main_name_new not in ner_dict_new:\n",
" ner_id += 1\n",
" ner_dict_new[company_main_name_new] = ner_id\n",
"\n",
" # 在句子中用编号替换实体名\n",
" sentence = re.sub(company_main_name, ' ner_%s_ ' % (str(ner_dict_new[company_main_name_new])), sentence)\n",
" train_data.iloc[i, -1] = sentence\n",
"\n",
"ner_dict_reverse_new = {id:name for name, id in ner_dict_new.items()}\n",
" \n",
"y = train_data.loc[:, ['tag']]\n",
"train_num = len(train_data)\n",
"X_train = train_data[['ner']]\n",
"\n",
"# 将train和test放在一起提取特征\n",
"X = pd.concat([X_train, X_test])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 3：关系抽取\n",
"\n",
"\n",
"目标：借助句法分析工具，和实体识别的结果，以及文本特征，基于训练数据抽取关系，并存储进图数据库。\n",
"\n",
"本次要求抽取股权交易关系，关系为无向边，不要求判断投资方和被投资方，只要求得到双方是否存在交易关系。\n",
"\n",
"模板建立可以使用“正则表达式”、“实体间距离”、“实体上下文”、“依存句法”等。\n",
"\n",
"答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
"- info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
"- info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
"\n",
"示例：\n",
"- info_extract_entity.csv\n",
"\n",
"| 实体编号 | 实体名 |\n",
"| ------ | ------ |\n",
"| 1001 | 小王 |\n",
"| 1002 | A化工厂 |\n",
"\n",
"- info_extract_submit.csv\n",
"\n",
"| 实体1 | 实体2 |\n",
"| ------ | ------ |\n",
"| 1001 | 1003 |\n",
"| 1002 | 1001 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习3：提取文本tf-idf特征\n",
"\n",
"去除停用词，并转换成tfidf向量。"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "ModuleNotFoundError",
"evalue": "No module named 'pyltp'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-18-cce147a3de87>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_extraction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTfidfTransformer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_extraction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mCountVectorizer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpyltp\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSegmentor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;31m# 实体符号加入分词词典\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pyltp'"
]
}
],
"source": [
"# code\n",
"from sklearn.feature_extraction.text import TfidfTransformer\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"from pyltp import Segmentor\n",
"\n",
"# 实体符号加入分词词典\n",
"with open('../data/user_dict.txt', 'w', encoding='utf-8') as fw:\n",
" for v in ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']:\n",
" fw.write( v + '号企业 ni\\n')\n",
"fw.close()\n",
"\n",
"# 初始化实例\n",
"segmentor = Segmentor() \n",
"\n",
"# 加载模型，加载自定义词典\n",
"import os\n",
"LTP_DATA_DIR = 'D:\\myLTP\\ltp_data_v3.4.0'\n",
"# LTP_DATA_DIR = '/Users/Badrain/Downloads/ltp_data_v3.4.0'\n",
"cws_model_path = os.path.join(LTP_DATA_DIR, 'cws.model') # 分词模型路径，模型名称为`cws.model`\n",
"\n",
"# 加载模型，加载自定义词典\n",
"segmentor.load_with_lexicon(cws_model_path, '../data/user_dict.txt')\n",
"\n",
"# 加载停用词\n",
"fr = open(r'../data/dict/stopwords.txt', encoding='utf-8') \n",
"stop_word = fr.readlines()\n",
"stop_word = [re.sub(r'(\\r|\\n)*','',stop_word[i]) for i in range(len(stop_word))]\n",
"\n",
"# 分词\n",
"f = lambda x: ' '.join([word for word in segmentor.segment(re.sub(r'ner\\_\\d\\d\\d\\d\\_','',x)) if word not in stop_word])\n",
"\n",
"corpus=X['ner'].map(f).tolist()\n",
"\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"vectorizer = TfidfVectorizer() # 定一个tf-idf的vectorizer\n",
"X_tfidf = vectorizer.fit_transform(corpus).toarray() # 结果存放在X矩阵\n",
"print(X_tfidf)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"ename": "ModuleNotFoundError",
"evalue": "No module named 'pyltp'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-19-d9c6901987ce>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mpyltp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pyltp'"
]
}
],
"source": [
"import pyltp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习4：提取句法特征\n",
"除了词语层面的句向量特征，我们还可以从句法入手，提取一些句法分析的特征。\n",
"\n",
"参考特征：\n",
"\n",
"1、企业实体间距离\n",
"\n",
"2、企业实体间句法距离\n",
"\n",
"3、企业实体分别和关键触发词的距离\n",
"\n",
"4、实体的依存关系类别"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# -*- coding: utf-8 -*-\n",
"from pyltp import Parser\n",
"from pyltp import Segmentor\n",
"from pyltp import Postagger\n",
"import networkx as nx\n",
"import pylab\n",
"import re\n",
"import matplotlib.pyplot as plt\n",
"from pylab import mpl\n",
"from graphviz import Digraph\n",
"import numpy as np\n",
"\n",
"postagger = Postagger() # 初始化实例\n",
"\n",
"pos_model_path = os.path.join(LTP_DATA_DIR, 'pos.model')\n",
"postagger.load_with_lexicon(pos_model_path, '../data/user_dict.txt') # 加载模型\n",
"segmentor = Segmentor() # 初始化实例\n",
"segmentor.load_with_lexicon(cws_model_path, '../data/user_dict.txt') # 加载模型\n",
"\n",
"SEN_TAGS = [\"SBV\",\"VOB\",\"IOB\",\"FOB\",\"DBL\",\"ATT\",\"ADV\",\"CMP\",\"COO\",\"POB\",\"LAD\",\"RAD\",\"IS\",\"HED\"]\n",
"\n",
"\n",
"def parse(s, isGraph = False):\n",
" \"\"\"\n",
" 对语句进行句法分析，并返回句法结果\n",
" \"\"\"\n",
" tmp_ner_dict = {}\n",
" num_lst = ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']\n",
"\n",
" # print(s)\n",
"\n",
" # 将公司代码替换为特殊称谓，保证分词词性正确\n",
" for i, ner in enumerate(list(set(re.findall(r'(ner\\_\\d\\d\\d\\d\\_)', s)))):\n",
" try:\n",
" tmp_ner_dict[num_lst[i] + '号企业'] = ner\n",
" except IndexError:\n",
" # TODO：定义错误情况的输出\n",
" # TODO ...\n",
" num_lst.append(str(i))\n",
" tmp_ner_dict[num_lst[i] + '号企业'] = ner\n",
"\n",
" s = s.replace(ner, num_lst[i] + '号企业')\n",
"\n",
" # print(tmp_ner_dict)\n",
"\n",
" words = segmentor.segment(s)\n",
" tags = postagger.postag(words)\n",
" parser = Parser() # 初始化实例\n",
"\n",
" parse_model_path = os.path.join(LTP_DATA_DIR, 'parser.model')\n",
" parser.load(parse_model_path)\n",
"\n",
" arcs = parser.parse(words, tags) # 句法分析\n",
" arcs_lst = list(map(list, zip(*[[arc.head, arc.relation] for arc in arcs])))\n",
"\n",
" # 句法分析结果输出\n",
" parse_result = pd.DataFrame([[a, b, c, d] for a, b, c, d in zip(list(words), list(tags), arcs_lst[0], arcs_lst[1])],\n",
" index=range(1, len(words) + 1))\n",
" parser.release() # 释放模型\n",
"\n",
" result = []\n",
"\n",
" # 实体的依存关系类别\n",
" rely_id = [arc.head for arc in arcs] # 提取依存父节点id\n",
" relation = [arc.relation for arc in arcs] # 提取依存关系\n",
" heads = ['Root' if id == 0 else words[id - 1] for id in rely_id] # 匹配依存父节点词语\n",
" # for i in range(len(words)):\n",
" # print(relation[i] + '(' + words[i] + ', ' + heads[i] + ')')\n",
"\n",
" company_list = list(tmp_ner_dict.keys())\n",
"\n",
" str_enti_1 = \"一号企业\"\n",
" str_enti_2 = \"二号企业\"\n",
" l_w = list(words)\n",
" is_two_company = str_enti_1 in l_w and str_enti_2 in l_w\n",
" if is_two_company:\n",
" second_entity_index = l_w.index(str_enti_2)\n",
" entity_sentence_type = parse_result.iloc[second_entity_index, -1]\n",
" if entity_sentence_type in SEN_TAGS:\n",
" result.append(SEN_TAGS.index(entity_sentence_type))\n",
" else:\n",
" result.append(-1)\n",
" else:\n",
" result.append(-1)\n",
"\n",
" if isGraph:\n",
" g = Digraph('测试图片')\n",
" g.node(name='Root')\n",
" for word in words:\n",
" g.node(name=word, fontname=\"SimHei\")\n",
"\n",
" for i in range(len(words)):\n",
" if relation[i] not in ['HED']:\n",
" g.edge(words[i], heads[i], label=relation[i], fontname=\"SimHei\")\n",
" else:\n",
" if heads[i] == 'Root':\n",
" g.edge(words[i], 'Root', label=relation[i], fontname=\"SimHei\")\n",
" else:\n",
" g.edge(heads[i], 'Root', label=relation[i], fontname=\"SimHei\")\n",
" g.view()\n",
"\n",
" # 企业实体间句法距离\n",
" distance_e_jufa = 0\n",
" if is_two_company:\n",
" distance_e_jufa = shortest_path(parse_result, list(words), str_enti_1, str_enti_2, isGraph=False)\n",
" result.append(distance_e_jufa)\n",
"\n",
" # 企业实体间距离\n",
" distance_entity = 0\n",
" if is_two_company:\n",
" distance_entity = np.abs(l_w.index(str_enti_1) - l_w.index(str_enti_2))\n",
" result.append(distance_entity)\n",
"\n",
" # 企业实体分别和关键触发词的距离\n",
" key_words = [\"收购\", \"竞拍\", \"转让\", \"扩张\", \"并购\", \"注资\", \"整合\", \"并入\", \"竞购\", \"竞买\", \"支付\", \"收购价\", \"收购价格\", \"承购\", \"购得\", \"购进\",\n",
" \"购入\", \"买进\", \"买入\", \"赎买\", \"购销\", \"议购\", \"函购\", \"函售\", \"抛售\", \"售卖\", \"销售\", \"转售\"]\n",
" # TODO：*根据关键词和对应句法关系提取特征（如没有思路可以不完成）\n",
" # TODO ...\n",
"\n",
" k_w = None\n",
" for w in words:\n",
" if w in key_words:\n",
" k_w = w\n",
" break\n",
"\n",
" dis_key_e_1 = -1\n",
" dis_key_e_2 = -1\n",
"\n",
" if k_w != None and is_two_company:\n",
" k_w = str(k_w)\n",
" # print(\"k_w\", k_w)\n",
"\n",
" l_w = list(words)\n",
" # dis_key_e_1 = shortest_path(parse_result, l_w, str_enti_1, k_w)\n",
" # dis_key_e_2 = shortest_path(parse_result, l_w, str_enti_2, k_w)\n",
"\n",
" dis_key_e_1 = np.abs(l_w.index(str_enti_1) - l_w.index(k_w))\n",
" dis_key_e_2 = np.abs(l_w.index(str_enti_2) - l_w.index(k_w))\n",
"\n",
" result.append(dis_key_e_1)\n",
" result.append(dis_key_e_2)\n",
"\n",
" return result\n",
"\n",
"mpl.rcParams['font.sans-serif'] = ['SimHei'] # 指定默认字体\n",
"\n",
"def shortest_path(arcs_ret, words, source, target, isGraph = False):\n",
" \"\"\"\n",
" 求出两个词最短依存句法路径，不存在路径返回-1\n",
" arcs_ret：句法分析结果\n",
" source：实体1\n",
" target：实体2\n",
" \"\"\"\n",
" # G = nx.DiGraph()\n",
" G = nx.Graph()\n",
"\n",
" # 为这个网络添加节点...\n",
" for i in list(arcs_ret.index):\n",
" G.add_node(i)\n",
"\n",
" # TODO：在网络中添加带权中的边...（注意，我们需要的是无向边）\n",
" # TODO ...\n",
"\n",
" for i in range(len(arcs_ret)):\n",
" head = arcs_ret.iloc[i, -2]\n",
" index = i + 1 # 从1开始\n",
" G.add_edge(index, head)\n",
"\n",
" if isGraph:\n",
" nx.draw(G, with_labels=True)\n",
" plt.savefig(\"undirected_graph_2.png\")\n",
" plt.close()\n",
"\n",
" try:\n",
" # TODO：利用nx包中shortest_path_length方法实现最短距离提取\n",
" # TODO ...\n",
"\n",
" source_index = words.index(source) + 1 #从1开始\n",
" target_index = words.index(target) + 1 #从1开始\n",
" distance = nx.shortest_path_length(G, source=source_index, target=target_index)\n",
" # print(\"'%s'与'%s'在依存句法分析图中的最短距离为: %s\" % (source, target, distance))\n",
"\n",
" return distance\n",
" except:\n",
" return -1\n",
"\n",
"corpus_1 = X['ner'].tolist()\n",
"len_train_data = len(train_data)\n",
"\n",
"def get_feature(s):\n",
" \"\"\"\n",
" 汇总上述函数汇总句法分析特征与TFIDF特征\n",
" \"\"\"\n",
" # TODO：汇总上述函数汇总句法分析特征与TFIDF特征\n",
" # TODO ...\n",
" sen_feature = []\n",
" len_s = len(s)\n",
" for i in trange(len_s):\n",
" f_e = parse(s[i], isGraph = False)\n",
" sen_feature.append(f_e)\n",
"\n",
" sen_feature = np.array(sen_feature)\n",
"\n",
" features = np.concatenate((X_tfidf, sen_feature), axis= 1)\n",
"\n",
" return features\n",
"\n",
"features = []\n",
"if not is_exist_f_v:\n",
" features = get_feature(corpus_1)\n",
" np.save(f_v_s_path, features)\n",
"else:\n",
" features = np.load(f_v_s_path)\n",
"\n",
"features_train = features[:len_train_data, :]\n",
"print(features_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习5：建立分类器\n",
"\n",
"利用已经提取好的tfidf特征以及parse特征，建立分类器进行分类任务。"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# 建立分类器进行分类\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn import preprocessing\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import GridSearchCV\n",
"from sklearn.metrics import roc_auc_score\n",
"from seqeval.metrics import f1_score\n",
"from sklearn.metrics import confusion_matrix\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.metrics import classification_report\n",
"from sklearn.naive_bayes import BernoulliNB\n",
"\n",
"seed = 2019\n",
"\n",
"# y = np.array(y.values, dtype=np.float)\n",
"y = train_data.loc[:, ['tag']]\n",
"y = np.array(y.values)\n",
"y = y.reshape(-1)\n",
"Xtrain, Xtest, ytrain, ytest = train_test_split(features_train, y, test_size = 0.2, random_state = seed)\n",
"\n",
"def logistic_class(Xtrain, Xtest, ytrain, ytest):\n",
" cross_validator = KFold(n_splits=10, shuffle=True, random_state = seed)\n",
"\n",
" lr = LogisticRegression(penalty = \"l1\", solver='liblinear')\n",
"\n",
" params = {\"penalty\":[\"l1\",\"l2\"],\n",
" \"C\":[0.1,1.0,10.0,20.0,30.0,100.0]}\n",
"\n",
"\n",
" grid = GridSearchCV(estimator=lr, param_grid = params, cv=cross_validator)\n",
" grid.fit(Xtrain, ytrain)\n",
" print(\"最优参数为：\",grid.best_params_)\n",
" model = grid.best_estimator_\n",
" y_pred = model.predict(Xtest)\n",
"\n",
" y_test = [str(value) for value in ytest]\n",
" y_pred = [str(value) for value in y_pred]\n",
"\n",
"\n",
" proba_value = model.predict_proba(Xtest)\n",
" p = proba_value[:,1]\n",
" print(\"Logistic=========== ROC-AUC score: %.3f\" % roc_auc_score(y_test, p))\n",
"\n",
" report = classification_report(y_pred=y_pred,y_true=y_test)\n",
" print(report)\n",
"\n",
"\n",
" return model\n",
"\n",
"\n",
"\n",
"# TODO：保存Test_data分类结果\n",
"# 答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
"# info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
"# info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
"\n",
"s_model = logistic_class(Xtrain, Xtest, ytrain, ytest)\n",
"\n",
"features_test = features[len_train_data:, :]\n",
"y_pred_test = s_model.predict(features_test)\n",
"\n",
"l_X_test_ner = X_test.values.tolist()\n",
"\n",
"entity_dict = {}\n",
"relation_list = []\n",
"\n",
"for i, label in enumerate(y_pred_test):\n",
" if label == 1:\n",
" cur_ner_content = str(l_X_test_ner[i])\n",
"\n",
" ner_list = list(set(re.findall(r'(ner\\_\\d\\d\\d\\d\\_)',cur_ner_content)))\n",
" if len(ner_list) == 2:\n",
" # print(ner_list)\n",
" r_e_l = []\n",
" for i, ner in enumerate(ner_list):\n",
" split_list = str.split(ner, \"_\")\n",
" if len(split_list) == 3:\n",
" ner_id = int(split_list[1])\n",
"\n",
" if ner_id in ner_dict_reverse_new:\n",
" if ner_id not in entity_dict:\n",
"\n",
" company_main_name = ner_dict_reverse_new[ner_id]\n",
"\n",
" if company_main_name in dict_entity_name_unify:\n",
" entity_dict[ner_id] = company_main_name + dict_entity_name_unify[company_main_name]\n",
" else:\n",
" entity_dict[ner_id] = company_main_name\n",
"\n",
" r_e_l.append(ner_id)\n",
" if len(r_e_l) == 2:\n",
" relation_list.append(r_e_l)\n",
"\n",
"# print(entity_dict)\n",
"# print(relation_list)\n",
"entity_list = [[item[0], item[1]] for item in entity_dict.items()]\n",
"# print(entity_list)\n",
"pd_enti = pd.DataFrame(np.array(entity_list), columns=['实体编号','实体名'])\n",
"pd_enti.to_csv(\"../submit/info_extract_entity.csv\",index=0, encoding='utf_8_sig')\n",
"\n",
"pd_re = pd.DataFrame(np.array(relation_list), columns=['实体1','实体2'])\n",
"pd_re.to_csv(\"../submit/info_extract_submit.csv\",index=0,enc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习6：操作图数据库\n",
"对关系最好的描述就是用图，那这里就需要使用图数据库，目前最常用的图数据库是noe4j，通过cypher语句就可以操作图数据库的增删改查。可以参考“https://cuiqingcai.com/4778.html”。\n",
"\n",
"本次作业我们使用neo4j作为图数据库，neo4j需要java环境，请先配置好环境。\n",
"\n",
"将我们提出的实体关系插入图数据库，并查询某节点的3层投资关系，即三个节点组成的路径（如果有的话）。如果无法找到3层投资关系，请查询出任意指定节点的投资路径。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from py2neo import Node, Relationship, Graph\n",
"\n",
"graph = Graph(\n",
" \"http://localhost:7474\",\n",
" username=\"neo4j\",\n",
" # password=\"person\"\n",
" password = \"111111\"\n",
")\n",
"\n",
"#清空所有数据对象\n",
"graph.delete_all()\n",
"\n",
"for v in relation_list:\n",
" a = Node('Company', name=str(v[0]))\n",
" b = Node('Company', name=str(v[1]))\n",
"\n",
" # 本次不区分投资方和被投资方，无向图\n",
" r = Relationship(a, 'INVEST', b)\n",
" s = a | b | r\n",
" graph.create(s)\n",
" r = Relationship(b, 'INVEST', a)\n",
" s = a | b | r\n",
" graph.create(s)\n",
"\n",
"# TODO：查询某节点的3层投资关系\n",
"\n",
"import random\n",
"\n",
"result_2 = []\n",
"result_3 = []\n",
"for value in entity_list:\n",
" ner_id = value[0]\n",
" str_sql_3 = \"match data=(na:Company{{name:'{0}'}})-[:INVEST]->(nb:Company)-[:INVEST]->(nc:Company) where na.name <> nc.name return data\".format(str(ner_id))\n",
" result_3 = graph.run(str_sql_3).data()\n",
" if len(result_3) > 0:\n",
" break\n",
"\n",
"if len(result_3) > 0:\n",
" print(\"step1\")\n",
" print(result_3)\n",
"else:\n",
" print(\"step2\")\n",
" random_index = random.randint(0, len(entity_list) - 1)\n",
" random_ner_id = entity_list[random_index][0]\n",
" str_sql_2 = \"match data=(na:Company{{name:'{0}'}})-[*2]->(nb:Company) return data\".format(str(random_ner_id))\n",
" result_2 = graph.run(str_sql_2).data()\n",
" print(result_2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# TODO：查询某节点的3层投资关系\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤4：实体消歧\n",
"解决了实体识别和关系的提取，我们已经完成了一大截，但是我们提取的实体究竟对应知识库中哪个实体呢？下图中，光是“苹果”就对应了13个同名实体。\n",
"<img src=\"../image/baike2.png\", width=340, heigth=480>\n",
"\n",
"在这个问题上，实体消歧旨在解决文本中广泛存在的名称歧义问题，将句中识别的实体与知识库中实体进行匹配，解决实体歧义问题。\n",
"\n",
"\n",
"### 练习7：\n",
"匹配test_data.csv中前25条样本中的人物实体对应的百度百科URL（此部分样本中所有人名均可在百度百科中链接到）。\n",
"\n",
"利用scrapy、beautifulsoup、request等python包对百度百科进行爬虫，判断是否具有一词多义的情况，如果有的话，选择最佳实体进行匹配。\n",
"\n",
"使用URL为‘https://baike.baidu.com/item/’+人名 可以访问百度百科该人名的词条，此处需要根据爬取到的网页识别该词条是否对应多个实体，如下图：\n",
"<img src=\"../image/baike1.png\", width=440, heigth=480>\n",
"如果该词条有对应多个实体，请返回正确匹配的实体URL，例如该示例网页中的‘https://baike.baidu.com/item/陆永/20793929’。\n",
"\n",
"- 提交文件：entity_disambiguation_submit.csv\n",
"- 提交格式：第一列为实体id（与info_extract_submit.csv中id保持一致），第二列为对应URL。\n",
"- 示例：\n",
"\n",
"| 实体编号 | URL |\n",
"| ------ | ------ |\n",
"| 1001 | https://baike.baidu.com/item/陆永/20793929 |\n",
"| 1002 | https://baike.baidu.com/item/王芳/567232 |\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import jieba\n",
"import pandas as pd\n",
"\n",
"# 找出test_data.csv中前25条样本所有的人物名称，以及人物所在文档的上下文内容\n",
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"\n",
"# 存储人物以及上下文信息（key为人物ID，value为人物名称、人物上下文内容）\n",
"# person_name = {}\n",
"list_person_content = {}\n",
"\n",
"# 观察上下文的窗口大小\n",
"window = 20\n",
"\n",
"f = lambda x: ' '.join([word for word in segmentor.segment(x)])\n",
"# corpus = X_test_new['ner'].map(f).tolist()\n",
"corpus= test_data['sentence'].map(f).tolist()\n",
"vectorizer = TfidfVectorizer() # 定一个tf-idf的vectorizer\n",
"X_tfidf = vectorizer.fit_transform(corpus).toarray() # 结果存放在X矩阵\n",
"\n",
"\n",
"# 遍历前25条样本\n",
"for i in range(25):\n",
" sentence = str(copy(test_data.iloc[i, 1]))\n",
" len_sen = len(sentence)\n",
" words, ners = fool.analysis(sentence)\n",
" ners[0].sort(key=lambda x: x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type == 'person':\n",
" # TODO：提取实体的上下文\n",
" # print(\"ner_name\", ner_name)\n",
" # print(\"ner_id\", ner_dict_new[ner_name])\n",
" start_index = max(0, start - window)\n",
" end_index = min(len_sen - 1, end - 1 + window)\n",
" left_str = sentence[start_index:start]\n",
" right_str = sentence[end - 1:end_index]\n",
"\n",
" left_str = ' '.join([word for word in segmentor.segment(left_str)])\n",
" right_str = ' '.join([word for word in segmentor.segment(right_str)])\n",
" new_str = left_str + \" \" +right_str\n",
"\n",
" # new_str = left_str + right_str\n",
" # new_str = ' '.join([word for word in segmentor.segment(new_str)])\n",
"\n",
" content_vec = vectorizer.transform([new_str])\n",
"\n",
" ner_id = ner_dict_new[ner_name]\n",
" if ner_id not in list_person_content:\n",
" list_person_content[ner_id] = content_vec\n",
" \n",
"\n",
"\n",
"\n",
"# 利用爬虫得到每个人物名称对应的URL\n",
"# TODO：找到每个人物实体的词条内容。\n",
"\n",
"from requests_html import HTMLSession\n",
"from requests_html import HTML\n",
"from sklearn.metrics.pairwise import cosine_similarity\n",
"from scipy.sparse import csr_matrix\n",
"import jieba\n",
"\n",
"\n",
"list_company_names = [company for value in entity_list for company in str.split(value[1], \"|\")]\n",
"# for value in list_company_names:\n",
"# jieba.add_word(value)\n",
"\n",
"list_person_url = []\n",
"url_prefix = \"https://baike.baidu.com/item/\"\n",
"url_error = \"https://baike.baidu.com/error.html\"\n",
"\n",
"l_p_items = list(list_person_content.items())\n",
"len_items = len(l_p_items)\n",
"\n",
"# print(l_p_items)\n",
"\n",
"def get_para_vector(para_elems):\n",
" str_res = \"\"\n",
" for p_e in para_elems:\n",
" str_res += re.sub(r'(\\r|\\n)*', '', p_e.text)\n",
" str_res = ' '.join([word for word in jieba.cut(str_res)])\n",
" content_vec = vectorizer.transform([str_res])\n",
" content_vec = content_vec.toarray()[0]\n",
" return content_vec\n",
"\n",
"for index in trange(len_items):\n",
" value = l_p_items[index]\n",
"\n",
" person_id = value[0]\n",
" vector_entity = csr_matrix(value[1])\n",
"\n",
" person_name = ner_dict_reverse_new[person_id]\n",
"\n",
" session = HTMLSession()\n",
" url = url_prefix + person_name\n",
" response = session.get(url)\n",
"\n",
" url_list = []\n",
" if response.url != url_error:\n",
" para_elems = response.html.find('.para')\n",
" content_vec = get_para_vector(para_elems)\n",
" url_list.append([response.url, content_vec])\n",
"\n",
" banks = response.html.find('.polysemantList-wrapper')\n",
"\n",
" if len(banks) > 0:\n",
" banks_child = banks[0]\n",
" persion_links = list(banks_child.absolute_links)\n",
" for link in persion_links:\n",
" r_link = session.get(link)\n",
"\n",
" if r_link.url == url_error:\n",
" continue\n",
"\n",
" para_elems = r_link.html.find('.para')\n",
" content_vec = get_para_vector(para_elems)\n",
" url_list.append([r_link.url, content_vec])\n",
"\n",
" vectorizer_list = [item[1] for item in url_list]\n",
" vectorizer_list = csr_matrix(vectorizer_list)\n",
" result = list(cosine_similarity(value[1], vectorizer_list)[0])\n",
" max_index = result.index(max(result))\n",
" list_person_url.append([person_id, person_name, url_list[max_index][0]])\n",
"\n",
"print(list_person_url)\n",
"# pd.DataFrame(list_person_url).to_csv('../submit/entity_disambiguation_submit.csv', index=False)\n",
"\n",
"pd_re = pd.DataFrame(np.array(list_person_url), columns=['实体编号','名字','url'])\n",
"pd_re.to_csv(\"../submit/entity_disambiguation_submit.csv\",index=0,en"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}