{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 利用信息抽取技术搭建知识库\n",
"\n",
"在这个notebook文件中，有些模板代码已经提供给你，但你还需要实现更多的功能来完成这个项目。除非有明确要求，你无须修改任何已给出的代码。以**'【练习】'**开始的标题表示接下来的代码部分中有你需要实现的功能。这些部分都配有详细的指导，需要实现的部分也会在注释中以'TODO'标出。请仔细阅读所有的提示。\n",
"\n",
">**提示：**Code 和 Markdown 区域可通过 **Shift + Enter** 快捷键运行。此外，Markdown可以通过双击进入编辑模式。\n",
"\n",
"---\n",
"\n",
"### 让我们开始吧\n",
"\n",
"本项目的目的是结合命名实体识别、依存语法分析、实体消歧、实体统一对网站开放语料抓取的数据建立小型知识图谱。\n",
"\n",
"在现实世界中，你需要拼凑一系列的模型来完成不同的任务；举个例子，用来预测狗种类的算法会与预测人类的算法不同。在做项目的过程中，你可能会遇到不少失败的预测，因为并不存在完美的算法和模型。你最终提交的不完美的解决方案也一定会给你带来一个有趣的学习经验！\n",
"\n",
"\n",
"---\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 1：实体统一"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"实体统一做的是对同一实体具有多个名称的情况进行统一，将多种称谓统一到一个实体上，并体现在实体的属性中（可以给实体建立“别称”属性）\n",
"\n",
"例如：对“河北银行股份有限公司”、“河北银行公司”和“河北银行”我们都可以认为是一个实体，我们就可以将通过提取前两个称谓的主要内容，得到“河北银行”这个实体关键信息。\n",
"\n",
"公司名称有其特点，例如后缀可以省略、上市公司的地名可以省略等等。在data/dict目录中提供了几个词典，可供实体统一使用。\n",
"- company_suffix.txt是公司的通用后缀词典\n",
"- company_business_scope.txt是公司经营范围常用词典\n",
"- co_Province_Dim.txt是省份词典\n",
"- co_City_Dim.txt是城市词典\n",
"- stopwords.txt是可供参考的停用词\n",
"\n",
"### 练习1：\n",
"编写main_extract函数，实现对实体的名称提取“主体名称”的功能。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import jieba\n",
"import jieba.posseg as pseg\n",
"import re\n",
"import datetime\n",
"\n",
"\n",
"# 从输入的“公司名”中提取主体\n",
"def main_extract(input_str,stop_word,d_4_delete,d_city_province):\n",
" # 开始分词并处理\n",
" seg = pseg.cut(input_str)\n",
" seg_lst = remove_word(seg,stop_word,d_4_delete)\n",
" seg_lst = city_prov_ahead(seg_lst,d_city_province)\n",
" return [s.word for s in seg_lst]\n",
"\n",
" \n",
"#TODO：实现公司名称中地名提前\n",
"def city_prov_ahead(seg,d_city_province):\n",
" city_prov_lst = []\n",
" seg_lst = []\n",
" set_d_city_province = set(d_city_province)\n",
" # TODO ...\n",
" for s in seg:\n",
" if s.word in set_d_city_province:\n",
" city_prov_lst.append(s)\n",
" else:\n",
" seg_lst.append(s)\n",
" \n",
" return city_prov_lst+seg_lst\n",
"\n",
"\n",
"\n",
"\n",
"#TODO：替换特殊符号\n",
"def remove_word(seg,stop_word,d_4_delete):\n",
" # TODO ...\n",
" set_stop_word = set(stop_word)\n",
" set_d_4_delete = set(d_4_delete)\n",
" seg_lst = []\n",
" for s in seg:\n",
" if s.word in set_stop_word or s.word in set_d_4_delete:\n",
" continue\n",
" seg_lst.append(s)\n",
" return seg_lst\n",
"\n",
"\n",
"# 初始化，加载词典\n",
"def my_initial():\n",
" fr1 = open(r\"../data/dict/co_City_Dim.txt\", encoding='utf-8')\n",
" fr2 = open(r\"../data/dict/co_Province_Dim.txt\", encoding='utf-8')\n",
" fr3 = open(r\"../data/dict/company_business_scope.txt\", encoding='utf-8')\n",
" fr4 = open(r\"../data/dict/company_suffix.txt\", encoding='utf-8')\n",
" #城市名\n",
" lines1 = fr1.readlines()\n",
" d_4_delete = []\n",
" d_city_province = [re.sub(r'(\\r|\\n)*','',line) for line in lines1]\n",
" #省份名\n",
" lines2 = fr2.readlines()\n",
" l2_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines2]\n",
" d_city_province.extend(l2_tmp)\n",
" #公司后缀\n",
" lines3 = fr3.readlines()\n",
" l3_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines3]\n",
" lines4 = fr4.readlines()\n",
" l4_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines4]\n",
" d_4_delete.extend(l4_tmp)\n",
" #get stop_word\n",
" fr = open(r'../data/dict/stopwords.txt', encoding='utf-8') \n",
" stop_word = fr.readlines()\n",
" stop_word_after = [re.sub(r'(\\r|\\n)*','',stop_word[i]) for i in range(len(stop_word))]\n",
" stop_word_after[-1] = stop_word[-1]\n",
" stop_word = stop_word_after\n",
" return d_4_delete,stop_word,d_city_province\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Building prefix dict from the default dictionary ...\n",
"Loading model from cache /tmp/jieba.cache\n",
"Loading model cost 1.226 seconds.\n",
"Prefix dict has been built succesfully.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"河北银行\n"
]
}
],
"source": [
"# TODO：测试实体统一用例\n",
"d_4_delete,stop_word,d_city_province = my_initial()\n",
"company_name = \"河北银行股份有限公司\"\n",
"lst = main_extract(company_name,stop_word,d_4_delete,d_city_province)\n",
"company_name = ''.join(lst) # 对公司名提取主体部分，将包含相同主体部分的公司统一为一个实体\n",
"print(company_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 2：实体识别\n",
"有很多开源工具可以帮助我们对实体进行识别。常见的有LTP、StanfordNLP、FoolNLTK等等。\n",
"\n",
"本次采用FoolNLTK实现实体识别，fool是一个基于bi-lstm+CRF算法开发的深度学习开源NLP工具，包括了分词、实体识别等功能，大家可以通过fool很好地体会深度学习在该任务上的优缺点。\n",
"\n",
"在‘data/train_data.csv’和‘data/test_data.csv’中是从网络上爬虫得到的上市公司公告，数据样例如下："
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>sentence</th>\n",
" <th>tag</th>\n",
" <th>member1</th>\n",
" <th>member2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>6461</td>\n",
" <td>与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2111</td>\n",
" <td>三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>9603</td>\n",
" <td>2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515...</td>\n",
" <td>1</td>\n",
" <td>协鑫集成科技股份有限公司</td>\n",
" <td>瑞峰（张家港）光伏科技有限公司</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3456</td>\n",
" <td>证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>8844</td>\n",
" <td>本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数...</td>\n",
" <td>1</td>\n",
" <td>广发证券股份有限公司</td>\n",
" <td>辽宁成大股份有限公司</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id sentence tag member1 \\\n",
"0 6461 与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市... 0 0 \n",
"1 2111 三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无... 0 0 \n",
"2 9603 2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515... 1 协鑫集成科技股份有限公司 \n",
"3 3456 证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限... 0 0 \n",
"4 8844 本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数... 1 广发证券股份有限公司 \n",
"\n",
" member2 \n",
"0 0 \n",
"1 0 \n",
"2 瑞峰（张家港）光伏科技有限公司 \n",
"3 0 \n",
"4 辽宁成大股份有限公司 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
"train_data.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>sentence</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>9259</td>\n",
" <td>2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>9136</td>\n",
" <td>2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>220</td>\n",
" <td>2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>9041</td>\n",
" <td>2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>10041</td>\n",
" <td>一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id sentence\n",
"0 9259 2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》\n",
"1 9136 2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先\n",
"2 220 2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》\n",
"3 9041 2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...\n",
"4 10041 一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收..."
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"test_data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"我们选取一部分样本进行标注，即train_data，该数据由5列组成。id列表示原始样本序号；sentence列为我们截取的一段关键信息；如果关键信息中存在两个实体之间有股权交易关系则tag列为1，否则为0；如果tag为1，则在member1和member2列会记录两个实体出现在sentence中的名称。\n",
"\n",
"剩下的样本没有标注，即test_data，该数据只有id和sentence两列，希望你能训练模型对test_data中的实体进行识别，并判断实体对之间有没有股权交易关系。\n",
"\n",
"### 练习2：\n",
"将每句句子中实体识别出，存入实体词典，并用特殊符号替换语句。\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: Logging before flag parsing goes to stderr.\n",
"W0803 17:53:50.497887 140437070104320 deprecation_wrapper.py:119] From /home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/fool/predictor.py:32: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
"\n",
"W0803 17:53:50.499367 140437070104320 deprecation_wrapper.py:119] From /home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/fool/predictor.py:33: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.\n",
"\n",
"W0803 17:53:50.748158 140437070104320 deprecation_wrapper.py:119] From /home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/fool/predictor.py:53: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" ner\n",
"0 2015年1月26日， ner_1002_ 司与 ner_1001_ 峰先生签署了《附条件生...\n",
"1 2、2016年2月5日， ner_1004_ 司与 ner_1003_ 毅先\n",
"2 2015年10月26日， ner_1006_ 司与 ner_1005_ 华先生签署了附条件生...\n",
"3 2、2015年12月31日， ner_1008_ 司与 ner_1007_ 革签订了《印纪娱...\n",
"4 一、 ner_1010_ 技拟与 ner_1009_ 涛女士签订《股份转让协议》，协议约定：...\n",
"5 同日， ner_1012_ 司与 ner_1011_ 琳先生在上海签署了《股权转让协议》和《...\n",
"6 截至本公告日， ner_1014_ 林持有 ner_1013_ 司股份354，418，300...\n",
"7 本次交易完成后，上市公司将由传统的医药制造业转变成为“企业互联网服务业务为主导，制造业务为支...\n",
"8 2015年1月20日， ner_1018_ 司与 ner_1017_ 飞先生签署《万鸿集团股...\n",
"9 一、（一） ner_1020_ 司（以下简称“公司”）拟以非公开发行方式向 ner_1019...\n",
"10 2、 ner_1022_ 司于2014年12月12日与 ner_1021_ 睿先生签订了《成...\n",
"11 于2015年4月24日， ner_1024_ 司与鲜言先生签订了股权转让协议，鲜言先生按注册...\n",
"12 ner_1026_ 司于2015年11月26日与 ner_1025_ 华先\n",
"13 2015年6月17日， ner_1028_ 峰先生、 ner_1027_ 份高管资管计划分别...\n",
"14 一、 ner_1002_ 司拟向包括 ner_1001_ 峰先生在内的不超过10名特定对象非...\n",
"15 截止2016年12月31日， ner_1018_ 司已向 ner_1017_ 飞归还上述欠款\n",
"16 若公司A股股票在定价基准日至发行日期间发生派发股利、送红股、转增股本、增发新股或配股等除权、...\n",
"17 2015年6月28日， ner_1032_ 司在福建省厦门市与 ner_1031_ 斌先生签...\n",
"18 ner_1013_ 司控股股东、实际控制人 ner_1014_ 林与非关联方分别以自有资金...\n",
"19 2015年4月2日， ner_1034_ 司与 ner_1033_ 峰签署了附条件生效的《关...\n",
"20 一、（一） ner_1037_ 司（以下简称“公司”）拟与公司控股股东和实际控制人 ner_...\n",
"21 在本次非公开发行事项中， ner_1039_ 司总计向5名特定对象非公开发行股票，发行对象包...\n",
"22 一、本次交易方案概述本次交易中， ner_1042_ 份拟以发行股份及支付现金的方式购买 n...\n",
"23 2、 ner_1047_ 司（以下简称“公司”或“ ner_1044_ 技”）计划受让 ne...\n",
"24 ner_1048_ 司本次总计向五名特定对象非公开发行股票，发行对象为 ner_1049_...\n",
"25 2015年6月15日， ner_1055_ 流先生和 ner_1054_ 资分别与公司签署附...\n",
"26 ner_1026_ 司控股股东及实际控制人 ner_1025_ 华先生拟认购不低于本次非公...\n",
"27 一、（一）本次关联交易基本情况 ner_1057_ 良先生为 ner_1056_ 司控股股东...\n",
"28 一、1、 ner_1059_ 司控股股东及实际控制人 ner_1058_ 海以不低于人民币3...\n",
"29 一、（一）关联交易基本情况 ner_1061_ 司（以下简称“公司”）拟向 ner_1060...\n",
".. ...\n",
"389 股票代码:600400 股票简称: ner_1604_ 份 编号:临 2015-054 n...\n",
"390 经 ner_1606_ 司评估，并经 ner_1605_ 司备案，该煤灰池的账面价值为886...\n",
"391 10、2015 年 8 月 24 日，公司使用 10，000 万元人民币购买 ner_160...\n",
"392 3、<附生效条件的股份认购合同>的主要内容2015 年 11 月 18 日，公司与本次非公开...\n",
"393 1因 ner_1269_ 董事系北京缘竟尽⑸虾缘竟尽⒈本缘揽萍脊 ner_1537_ 司的法...\n",
"394 二、 ner_1611_ 厂项目2014年12月底， ner_1611_ 厂转为公用电厂项目...\n",
"395 公司独立董事发表了独立意见，认为: ner_1613_ 司与青海省电力设计院共同组建项目公司...\n",
"396 证券代码:600533 证券简称:栖霞建设 编号:临 2015-059 ner_1614_...\n",
"397 3、本公司与 ner_1534_ 化、 ner_1615_ 君于 2015 年 8 月 25...\n",
"398 重要内容提示●被担保人:控股子公司---- ner_1616_ 司●本次担保金额及已实际为其...\n",
"399 12 / 14 七、历史关联交易(日常关联交易除外)情况1、本次交易前 12 个月内，公司与...\n",
"400 ner_1618_ 司(“公司”)全资拥有的 ner_1617_ 司2号350兆瓦燃煤发电...\n",
"401 二、审议通过了<关于控股子公司 ner_1620_ 司与 ner_1619_ 司签订供热合作...\n",
"402 转让方: ner_1622_ 司 企业类型:有限责任公司(自然人投资或控股) 企业住所:郑州...\n",
"403 证券代码:600108 证券简称: ner_1624_ 团 公告编号:2015-068 n...\n",
"404 会议审议并通过了以下议案，形成如下决议:审议并通过了< ner_1385_ 司关于向 ner...\n",
"405 2014年，公司收购 ner_1626_ 司100%股权， ner_1247_ 司于2014...\n",
"406 申请材料显示，报告期内 ner_1165_ 流、 ner_1164_ 务董事、高级管理人员存...\n",
"407 公司使用暂时闲置募集资金 6，000 万元购买了 ner_1628_ 司 ner_1627_...\n",
"408 二、交易各方当事人情况介绍(一)北京工投基本情况公司名称: ner_1630_ 司住所:北京...\n",
"409 (依法需经批准的项目经相关部门批准后方可开展经营活动)经 ner_1632_ 所审计，201...\n",
"410 ner_1634_ 司吸收合并 ner_1633_ 司及发行股份购买资产并募集配套资金 暨...\n",
"411 回复:(一) 核查情况:1、请你公司结合行业情况、未来经营战略、盈利模式等补充披露客户集中 ...\n",
"412 三、 受让标的基本情况企业名称: ner_1637_ 司 企业类型:有限责任公司(法人独资)...\n",
"413 本次股权转让前，公司持有 ner_1638_ 险 18%的股权;本次股权转让完成后，公 司将...\n",
"414 近日，该子公司已完成工商注册登记手续，并领取了南京市工商行政管理局颁发的<企业法人营业执照>...\n",
"415 (二)本次交易构成关联交易正元投资拟认购金额不低于 13 亿元且不低于本次配套融资总额的 2...\n",
"416 证券代码:600225 证券简称: ner_1642_ 公告编号:临 2015-118 ...\n",
"417 2015年3月31日， ner_1644_ 司与 ner_1643_ 司签署了附条件生效的《...\n",
"418 三、本公司不会利用对 ner_1068_ 份控制关系损害 ner_1068_ 份及其他股东 ...\n",
"\n",
"[419 rows x 1 columns]\n"
]
}
],
"source": [
"# 处理test数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
"\n",
"import fool\n",
"import pandas as pd\n",
"from copy import copy\n",
"\n",
"\n",
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"test_data['ner'] = None\n",
"ner_id = 1001\n",
"ner_dict_new = {} # 存储所有实体\n",
"ner_dict_reverse_new = {} # 存储所有实体\n",
"set_company = set()\n",
"\n",
"for i in range(len(test_data)):\n",
" sentence = copy(test_data.iloc[i, 1])\n",
" # TODO：调用fool进行实体识别，得到words和ners结果\n",
" # TODO ...\n",
" words, ners = fool.analysis([sentence])\n",
" \n",
" ners[0].sort(key=lambda x:x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type=='company' :\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
" company_main_name = main_extract(ner_name,stop_word,d_4_delete,d_city_province)\n",
" company_main_name = ''.join(company_main_name)\n",
" if company_main_name not in ner_dict_new:\n",
" ner_dict_new[company_main_name] = ner_id\n",
" ner_dict_reverse_new[ner_dict_new[company_main_name]] = company_main_name\n",
" ner_id += 1\n",
" set_company.add(ner_id)\n",
" \n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end-1:]\n",
" elif ner_type=='person':\n",
" if ner_name not in ner_dict_new:\n",
" ner_dict_new[ner_name] = ner_id\n",
" ner_dict_reverse_new[ner_dict_new[ner_name]] = ner_name\n",
" ner_id += 1\n",
" \n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[ner_name]) + '_ ' + sentence[end-1:]\n",
" test_data.iloc[i, -1] = sentence\n",
"\n",
"X_test = test_data[['ner']]\n",
"print(X_test)\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" id sentence tag \\\n",
"0 6461 与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市... 0 \n",
"1 2111 三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无... 0 \n",
"2 9603 2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515... 1 \n",
"3 3456 证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限... 0 \n",
"4 8844 本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数... 1 \n",
"5 8722 二、被担保人基本情况被担保人:上海同济建设有限公司 住所:上海市杨浦区中山北二路 1121 ... 0 \n",
"6 9200 2014年12月12日，绵阳市国资委出具了<关于同意转让绵阳国虹通讯数码集团有限责任公司股权... 0 \n",
"7 3723 特变电工股份有限公司 2015 年第十二次临时董事会会议 证券代码:600089 证券简称:... 0 \n",
"8 3964 变更 完成后，本公司持有翔通动漫 100%的股权，翔通动漫已成为公司全资子公司。 0 \n",
"9 9547 根据上市公司与浦发银行达成的并购贷款意向函，浦发银行合肥分行愿意提供不高于50%交易金额的并... 0 \n",
"10 3177 特此公告广东四通集团股份有限公司董事会 2015 年 11 月 8 日备查文件<广东四通集团... 0 \n",
"11 7137 证券代码:600348 股票简称:阳泉煤业 编号:临 2015-056阳泉煤业(集团)股份有... 0 \n",
"12 5169 附件:上海申达股份有限公司董事会 2015 年 11 月 17 日1、 上海申达股份有限公司... 0 \n",
"13 4728 根据以上评估结果，经交易各方协商确定本次天安财险 26.96%股权交易价 格为 6，907，... 0 \n",
"14 1810 资产出售和资产购买方案简要介绍如 下:(一)资产出售公司拟向控股股东金诚实业出售双鹿实业 1... 0 \n",
"15 9598 根据京东方集团实际投资额，在2012年6月11日和2013年8月12日，内蒙古自治区人民政府... 0 \n",
"16 4788 五、本公司保证将依照西水股份章程参加股东大会，平等地行使相 应权利，承担相应义务，不利用控股... 0 \n",
"17 3104 2015 年 6 月 25 日，公司以人民币 8，000 万元闲置募集资金购买了广发银行 广... 0 \n",
"18 852 会议应参与表决董事9人，实际 表决9人，独立董事何红渠先生因公出差委托独立董事刘曙萍女士代为... 0 \n",
"19 9483 二、关联方介绍(一)关联方关系介绍根据<上海证券交易所股票上市规则>和<香港联合交易所有限公... 0 \n",
"20 9894 第五届董事会第十九次会议决议审议通过了《关于退出上海高特佳懿康投资合伙企业（有限合伙）暨关联... 1 \n",
"21 1784 公司将与浙江南浔 农村商业银行股份有限公司和孚支行及时分析和跟踪理财产品的进展情况，加强 风... 0 \n",
"22 4868 经评估，截至 2015 年 5 月 31 日，天安财险32经审计的归属母公司股东权益合计为 ... 0 \n",
"23 7 本年度上海大西洋收购天津大西洋焊接材料有限公司将其所持天津大西洋销售44%股权，支付对价8，... 1 \n",
"24 3205 变更后，中天科技持有中天宽 带 100%股权。 0 \n",
"25 8504 2、2014年11月7日公司与平安银行股份有限公司签订平安银行对公结构 性存款产品合约;资金... 0 \n",
"26 1885 13(本页无正文，为<延边石岘白麓纸业股份有限公司重大资产出售和重大资产 购买暨关联交易实施... 0 \n",
"27 8394 经合肥市工商行政管理局核准，公司全资子公司“安徽继远电网技术有限责 任公司”更名为“安徽南瑞... 0 \n",
"28 9336 四、对外投资合作合同的主要内容甲方:泉州盛翔投资管理中心(有限合伙)乙方:泉州泉晟投资有限公... 0 \n",
"29 3633 股票代码:601877 股票简称:正泰电器 编号:临 2015-068 债券代码:12208... 0 \n",
".. ... ... ... \n",
"820 8750 五、参与天堂硅谷增资扩股对公司影响天堂硅谷自成立以来，业务和资产规模快速发展，经济效益稳步提升。 0 \n",
"821 9593 五、包括贵州盘江精股份有限公司在内，被提名人兼任独立董事的境内上市公司数量未超过五家，被提名... 0 \n",
"822 6460 (二)预计增加关联交易的基本情况 类别 关联人 原预计金额 (万元) 增加预计额 (万元) ... 0 \n",
"823 1238 新疆天业股份有限公司(以下简称“公司”)收到中国证券监督管理委员会(以下 简称“中国证监会”... 0 \n",
"824 296 二、交易对方情况介绍11、公司名称:甘肃亚盛薯业有限责任公司 2、企业性质:有限责任公司 3... 0 \n",
"825 2572 公司于 2014 年 1 月 7 日与交通银行股份有限公司江苏省分行签订协议，使用 暂时闲置... 0 \n",
"826 6280 证券代码:601028 证券简称:玉龙股份 公告编号:2015-055 江苏玉龙钢管股份有限... 0 \n",
"827 236 郴州市政府旗下的郴州城投公司，是 国家开发银行湖南分行的信贷客户，在银行存在着良好的信用，符... 0 \n",
"828 3122 证券代码:603222 证券简称:济民制药 公告编号:2015-051 浙江济民制药股份有限... 0 \n",
"829 4343 ? 本次担保是否有反担保:无 ? 对外担保逾期的累计数量:零一、担保情况概述公司全资子公司保... 0 \n",
"830 8969 浙江苏泊尔股份有限公司向苏泊尔集团有限公司购买位于浙江省玉环县大麦屿街道南尤工业区部分地块，... 1 \n",
"831 2935 重要内容提示:委托理财受托方:上海浦东发展银行股份有限公司 委托理财金额: 2，000万元人... 0 \n",
"832 679 重要提示?关联交易标的名称: 钢铁产品? 关联交易金额: 2015 年销售金额预计人民币 5... 0 \n",
"833 6982 证券代码:600237 证券简称:铜峰电子 编号:临 2015-054 安徽铜峰电子股份有限... 0 \n",
"834 2808 重庆渝电工程监理咨询有限公司，成立于1998年6月，法定代表人赵 久云，工商注册号为5009... 0 \n",
"835 7122 17、公司购买理财产品名称:农业银行“汇利丰”对公定制人民币理财产品; 购买金额:人民币2亿... 0 \n",
"836 6508 (21)刘雯婧刘雯婧，女，国籍中国，身份证号 42010619880315****，住所为武... 0 \n",
"837 3221 三、关联交易标的基本情况(一)交易标的:与关联方北京缘竟尽⑸虾缘竟尽⒈本缘揽萍脊 司和缘咀什... 0 \n",
"838 95 2015年11月9日，江中药业股份有限公司将所持有的江西九州通51%股权在江西省产权交易所公... 1 \n",
"839 9340 采用成本法(资产基础法)对深圳金桔莱(母公司)全部股东权益价值进行评估，评估价值为125，1... 0 \n",
"840 942 2、平安大华基金管理有限公司(1)基本情况企业类型:有限责任公司(中外合资) 住所:深圳市福... 0 \n",
"841 866 华景公司另一股东嘉兴华泰不进行 增资，并同意我公司增资，其他权利义务不变。 0 \n",
"842 4908 如果天安财险 未能适应业务变化并及时调整风险管理政策及程序，则可能会对天安财险的业 务、经营... 0 \n",
"843 3210 变更后，中天科技持有中天合金 100%股权。 0 \n",
"844 3552 评估机构:广东中广信资产评估会计师事务所 评估基准日:2015 年 8 月 31 日 评估方... 0 \n",
"845 1225 请 你公司结合天伟化工的经营变化状况、上述增资价格与本次交易作价的差异，进一步补 充披露天伟... 0 \n",
"846 2970 公司使用暂时闲置募集资金6，483万元购买了中国农业银行股 份有限公司沈阳南湖支行“本利丰・... 0 \n",
"847 7961 (二) 本次重组尚需履行的决策程序18 本次交易尚需履行的决策和审批程序包括但不限于:1、本... 0 \n",
"848 1033 2015 年 5 月 6 日，公司以人民币 3，000 万元闲置募集资金购买了广发银行 广州... 0 \n",
"849 9282 一、：1、关联交易的基本情况广州毅昌科技股份有限公司（以下简称“公司”）拟以4617030元... 1 \n",
"\n",
" member1 member2 \\\n",
"0 0 0 \n",
"1 0 0 \n",
"2 协鑫集成科技股份有限公司 瑞峰（张家港）光伏科技有限公司 \n",
"3 0 0 \n",
"4 广发证券股份有限公司 辽宁成大股份有限公司 \n",
"5 0 0 \n",
"6 0 0 \n",
"7 0 0 \n",
"8 0 0 \n",
"9 0 0 \n",
"10 0 0 \n",
"11 0 0 \n",
"12 0 0 \n",
"13 0 0 \n",
"14 0 0 \n",
"15 0 0 \n",
"16 0 0 \n",
"17 0 0 \n",
"18 0 0 \n",
"19 0 0 \n",
"20 上海高特佳懿康投资合伙企业（有限合伙） 博雅生物制药集团股份有限公司 \n",
"21 0 0 \n",
"22 0 0 \n",
"23 大西洋 天津大西洋焊接材料有限公司 \n",
"24 0 0 \n",
"25 0 0 \n",
"26 0 0 \n",
"27 0 0 \n",
"28 0 0 \n",
"29 0 0 \n",
".. ... ... \n",
"820 0 0 \n",
"821 0 0 \n",
"822 0 0 \n",
"823 0 0 \n",
"824 0 0 \n",
"825 0 0 \n",
"826 0 0 \n",
"827 0 0 \n",
"828 0 0 \n",
"829 0 0 \n",
"830 浙江苏泊尔股份有限公司 苏泊尔集团有限公司 \n",
"831 0 0 \n",
"832 0 0 \n",
"833 0 0 \n",
"834 0 0 \n",
"835 0 0 \n",
"836 0 0 \n",
"837 0 0 \n",
"838 九州通 江中药业股份有限公司 \n",
"839 0 0 \n",
"840 0 0 \n",
"841 0 0 \n",
"842 0 0 \n",
"843 0 0 \n",
"844 0 0 \n",
"845 0 0 \n",
"846 0 0 \n",
"847 0 0 \n",
"848 0 0 \n",
"849 广州华南新材料创新园有限公司 广州毅昌科技股份有限公司 \n",
"\n",
" ner \n",
"0 与本公司关系:受同一公司控制 2， ner_1646_ 司企业类型: 有限公司注册地址: 富... \n",
"1 三、关联交易标的基本情况 1、交易标的基本情况 公司名称: ner_1648_ 司地址:无锡... \n",
"2 2016年 ner_1649_ 向 ner_1650_ 支付设备款人民币4，515，770.00元 \n",
"3 证券代码:600777 证券简称: ner_1201_ 公告编号:2015-091 ne... \n",
"4 本集团及 ner_1651_ 持有 ner_1652_ 股票的本期变动系买卖一揽子沪深300... \n",
"5 二、被担保人基本情况被担保人: ner_1654_ 司 住所:上海市杨浦区中山北二路 112... \n",
"6 2014年12月12日，绵阳市国资委出具了<关于同意转让 ner_1656_ 司股权有关事项... \n",
"7 ner_1657_ 司 2015 年第十二次临时董事会会议 证券代码:600089 证券简... \n",
"8 变更 完成后，本公司持有 ner_1659_ 100%的股权， ner_1658_ 漫已成... \n",
"9 根据上市公司与 ner_1661_ 行达成的并购贷款意向函， ner_1660_ 行愿意提供... \n",
"10 特此公告 ner_1128_ 司董事会 2015 年 11 月 8 日备查文件< ner_1... \n",
"11 证券代码:600348 股票简称: ner_1313_ 业 编号:临 2015-056 ne... \n",
"12 附件: ner_1485_ 司董事会 2015 年 11 月 17 日1、 ner_1662... \n",
"13 根据以上评估结果，经交易各方协商确定本次 ner_1166_ 险 26.96%股权交易价 格... \n",
"14 资产出售和资产购买方案简要介绍如 下:(一)资产出售公司拟向控股股东 ner_1663_ 业... \n",
"15 根据 ner_1665_ 团实际投资额，在2012年6月11日和2013年8月12日，内蒙古... \n",
"16 五、本公司保证将依照 ner_1068_ 份章程参加股东大会，平等地行使相 应权利，承担相应... \n",
"17 2015 年 6 月 25 日，公司以人民币 8，000 万元闲置募集资金购买了 ner_1... \n",
"18 会议应参与表决董事9人，实际 表决9人，独立董事 ner_1667_ 渠先生因公出差委托独立... \n",
"19 二、关联方介绍(一)关联方关系介绍根据<上海证券交易所股票上市规则>和 ner_1669_ ... \n",
"20 第五届董事会第十九次会议决议审议通过了《关于退出 ner_1670_ 暨关联交易的议案》， ... \n",
"21 公司将与 ner_1672_ 司和 ner_1100_ 行及时分析和跟踪理财产品的进展情况，... \n",
"22 经评估，截至 2015 年 5 月 31 日， ner_1166_ 险32经审计的归属母公司... \n",
"23 本年度上海 ner_1673_ 收购天津 ner_1673_ 焊接材料有限公司将其所持天津 ... \n",
"24 变更后， ner_1554_ 技持有 ner_1675_ 带 100%股权。 \n",
"25 2、2014年11月7日公司与 ner_1099_ 司签订 ner_1099_ 行对公结构 ... \n",
"26 13(本页无正文，为<延边 ner_1677_ 司重大资产出售和重大资产 购买暨关联交易实施... \n",
"27 经合肥市工商行政管理局核准，公司全资子公司“ ner_1679_ 司”更名为“ ner_16... \n",
"28 四、对外投资合作合同的主要内容甲方: ner_1681_ 心(有限合伙)乙方: ner_16... \n",
"29 股票代码:601877 股票简称: ner_1683_ 器 编号:临 2015-068 债券... \n",
".. ... \n",
"820 五、参与 ner_1223_ 谷增资扩股对公司影响 ner_1223_ 谷自成立以来，业务和... \n",
"821 五、包括 ner_2553_ 司在内，被提名人兼任独立董事的境内上市公司数量未超过五家，被提... \n",
"822 (二)预计增加关联交易的基本情况 类别 关联人 原预计金额 (万元) 增加预计额 (万元) ... \n",
"823 ner_1826_ 司(以下简称“公司”)收到中国证券监督管理委员会(以下 简称“中国证监... \n",
"824 二、交易对方情况介绍11、公司名称: ner_2557_ 司 2、企业性质:有限责任公司 3... \n",
"825 公司于 2014 年 1 月 7 日与 ner_1577_ 司 ner_2177_ 行签订协... \n",
"826 证券代码:601028 证券简称: ner_2559_ 份 公告编号:2015-055 n... \n",
"827 郴州市政府旗下的 ner_1816_ 司，是 ner_2560_ 行的信贷客户，在银行存在着... \n",
"828 证券代码:603222 证券简称: ner_2561_ 公告编号:2015-051 ne... \n",
"829 ? 本次担保是否有反担保:无 ? 对外担保逾期的累计数量:零一、担保情况概述公司全资子公司 ... \n",
"830 ner_2563_ 向 ner_2564_ 购买位于浙江省玉环县大麦屿街道南尤工业区部分地... \n",
"831 重要内容提示:委托理财受托方: ner_2086_ 司 委托理财金额: 2，000万元人民币... \n",
"832 重要提示?关联交易标的名称: 钢铁产品? 关联交易金额: 2015 年销售金额预计人民币 5... \n",
"833 证券代码:600237 证券简称: ner_2567_ 子 编号:临 2015-054 n... \n",
"834 ner_1954_ 司，成立于1998年6月，法定代表人 ner_2568_ 云，工商注册... \n",
"835 17、公司购买理财产品名称: ner_1422_ 行“ ner_1421_ 丰”对公定制人民... \n",
"836 (21) ner_2569_ 婧 ner_2569_ 婧，女，国籍中国，身份证号 42010... \n",
"837 三、关联交易标的基本情况(一)交易标的:与关联方北京缘竟尽⑸虾缘竟尽⒈本缘揽 ner_178... \n",
"838 2015年11月9日， ner_2572_ 将所持有的江西 ner_2571_ 51%股权在... \n",
"839 采用成本法(资产基础法)对 ner_2574_ 司)全部股东权益价值进行评估，评估价值为12... \n",
"840 2、 ner_2576_ 司(1)基本情况企业类型:有限责任公司(中外合资) 住所:深圳市福... \n",
"841 ner_2578_ 司另一股东 ner_2577_ 泰不进行 增资，并同意我公司增资，其他... \n",
"842 如果 ner_1166_ 险 未能适应业务变化并及时调整风险管理政策及程序，则可能会对 ne... \n",
"843 变更后， ner_1554_ 技持有 ner_2579_ 100%股权。 \n",
"844 评估机构: ner_2580_ 所 评估基准日:2015 年 8 月 31 日 评估方法:资... \n",
"845 请 你公司结合 ner_1322_ 工的经营变化状况、上述增资价格与本次交易作价的差异，进一... \n",
"846 公司使用暂时闲置募集资金6，483万元购买了 ner_2582_ 司 ner_2581_ 行... \n",
"847 (二) 本次重组尚需履行的决策程序18 本次交易尚需履行的决策和审批程序包括但不限于:1、本... \n",
"848 2015 年 5 月 6 日，公司以人民币 3，000 万元闲置募集资金购买了 ner_10... \n",
"849 一、：1、关联交易的基本情况 ner_2584_ （以下简称“公司”）拟以4617030元人... \n",
"\n",
"[850 rows x 6 columns]\n"
]
}
],
"source": [
"# 处理train数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
"train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
"train_data['ner'] = None\n",
"\n",
"for i in range(len(train_data)):\n",
" # 判断正负样本\n",
" if train_data.iloc[i,:]['member1']=='0' and train_data.iloc[i,:]['member2']=='0':\n",
" sentence = copy(train_data.iloc[i, 1])\n",
" # TODO：调用fool进行实体识别，得到words和ners结果\n",
" # TODO ...\n",
" \n",
" words, ners = fool.analysis([sentence])\n",
" ners[0].sort(key=lambda x:x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type=='company' :\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
" company_main_name = main_extract(ner_name,stop_word,d_4_delete,d_city_province)\n",
" company_main_name = ''.join(company_main_name)\n",
" if company_main_name not in ner_dict_new:\n",
" ner_dict_new[company_main_name] = ner_id\n",
" ner_dict_reverse_new[ner_dict_new[company_main_name]] = company_main_name\n",
" ner_id += 1\n",
" set_company.add(ner_id)\n",
"\n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end-1:]\n",
"\n",
" elif ner_type=='person':\n",
" if ner_name not in ner_dict_new:\n",
" ner_dict_new[ner_name] = ner_id\n",
" ner_dict_reverse_new[ner_dict_new[ner_name]] = ner_name\n",
" ner_id += 1\n",
"\n",
" # 在句子中用编号替换实体名\n",
" sentence = sentence[:start] + ' ner_' + str(ner_dict_new[ner_name]) + '_ ' + sentence[end-1:]\n",
" train_data.iloc[i, -1] = sentence\n",
" else:\n",
" # 将训练集中正样本已经标注的实体也使用编码替换\n",
" sentence = copy(train_data.iloc[i,:]['sentence'])\n",
" for company_main_name in [train_data.iloc[i,:]['member1'],train_data.iloc[i,:]['member2']]:\n",
" # TODO：调用实体统一函数，存储统一后的实体\n",
" # 并自增ner_id\n",
" # TODO ...\n",
" company_main_name = ''.join(company_main_name)\n",
" if company_main_name not in ner_dict_new:\n",
" ner_dict_new[company_main_name] = ner_id\n",
" ner_dict_reverse_new[ner_dict_new[company_main_name]] = company_main_name\n",
" ner_id += 1\n",
" set_company.add(ner_id)\n",
"\n",
"\n",
" # 在句子中用编号替换实体名\n",
" sentence = re.sub(company_main_name, ' ner_%s_ '%(str(ner_dict_new[company_main_name])), sentence)\n",
" train_data.iloc[i, -1] = sentence\n",
"print(train_data) \n",
"y = train_data.loc[:,['tag']]\n",
"train_num = len(train_data)\n",
"X_train = train_data[['ner']]\n",
"\n",
"# 将train和test放在一起提取特征\n",
"X = pd.concat([X_train, X_test])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤 3：关系抽取\n",
"\n",
"\n",
"目标：借助句法分析工具，和实体识别的结果，以及文本特征，基于训练数据抽取关系，并存储进图数据库。\n",
"\n",
"本次要求抽取股权交易关系，关系为无向边，不要求判断投资方和被投资方，只要求得到双方是否存在交易关系。\n",
"\n",
"模板建立可以使用“正则表达式”、“实体间距离”、“实体上下文”、“依存句法”等。\n",
"\n",
"答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
"- info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
"- info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
"\n",
"示例：\n",
"- info_extract_entity.csv\n",
"\n",
"| 实体编号 | 实体名 |\n",
"| ------ | ------ |\n",
"| 1001 | 小王 |\n",
"| 1002 | A化工厂 |\n",
"\n",
"- info_extract_submit.csv\n",
"\n",
"| 实体1 | 实体2 |\n",
"| ------ | ------ |\n",
"| 1001 | 1003 |\n",
"| 1002 | 1001 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习3：提取文本tf-idf特征\n",
"\n",
"去除停用词，并转换成tfidf向量。"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['公司 受 同一 公司 控制 2 司 企业 类型 有限公司 注册 地址 富阳市 东洲 街道 东洲 工业 功能 区九号路 1 号 法定 代表人 明 注册 资本 16 000万 元 经营 许可 经营 制造 高效 节能 感应 电机 ; 普通 货运', '三 关联 交易 标的 1 交易 标的 公司 名称 司 地址 无锡市 锡山区 东港镇 锡港东路 2 号 企业 类型 责任 公司 法定 代表人 燕 注册 资本 50000万 元 整 财务 公司 公司 参股 公司 成立 2008年 11月 经营 成员 单位 财务 融资 顾问 信用 鉴证 咨询 代理 业务 ; 成员 单位 交易 款项 收付 ; 批准 保险 代理 业务 ; 成员 单位 担保 ; 成员 单位 委托 贷款 委托 投资 ; 成员 单位 票据 承兑 贴现 ; 成员 单位 转账 结算 结算 清算 方案 设计 ; 吸收 成员 单位 存款 ; 成员 单位 贷款 融资 租赁 ; 同业 拆借 ; 承销 成员 单位 企业 债券 ; 金融 机构 股权 投资 ; 有价证券 投资 股票 二级 市场 投资 除外 ; 成员 单位 产品 买方 信贷', '2016年 支付 设备 款 人民币 4 515 770.00 元', '证券 代码 600777 证券 简称 公告 编号 2015-091 司 第八 届 监事会 第十一 次 会议 决议 公告 公司 监事会 全体 监事 公告 不 虚假 记载 误导性 陈述 遗漏 真实性 准确性 完整性 承担 个别 连带 责任', '集团 持有 股票 本期 变动 系 买卖 一揽子 沪 深 300 指数 成份股 致']\n",
" (0, 4103)\t0.2114565894485973\n",
" (0, 3022)\t0.1567445046668198\n",
" (0, 3469)\t0.20447702741373694\n",
" (0, 2712)\t0.22045480991774755\n",
" (0, 3894)\t0.2114565894485973\n",
" (0, 4498)\t0.22045480991774755\n",
" (0, 1855)\t0.16809558961226798\n",
" (0, 4023)\t0.15290970636043075\n",
" (0, 3754)\t0.19454004856814155\n",
" (0, 5)\t0.11077546570621795\n",
" (0, 235)\t0.16111602757740762\n",
" (0, 4136)\t0.10354667247447477\n",
" (0, 1542)\t0.11205231032692424\n",
" (0, 3241)\t0.11407096876121413\n",
" (0, 1920)\t0.22045480991774755\n",
" (0, 1880)\t0.22045480991774755\n",
" (0, 2525)\t0.17459020627859542\n",
" (0, 3953)\t0.18609203055056847\n",
" (0, 1382)\t0.4409096198354951\n",
" (0, 2470)\t0.22045480991774755\n",
" (0, 2246)\t0.1294748204889748\n",
" (0, 3248)\t0.20562135469497786\n",
" (0, 3052)\t0.13645438252383515\n",
" (0, 3705)\t0.11585878961850787\n",
" (0, 1557)\t0.11407096876121413\n",
" :\t:\n",
" (3, 1773)\t0.1524667506120705\n",
" (3, 1576)\t0.1316661833337407\n",
" (3, 3657)\t0.24603105663498262\n",
" (3, 3498)\t0.45619536537824595\n",
" (3, 3653)\t0.20244151361851392\n",
" (3, 111)\t0.2957368225980327\n",
" (3, 333)\t0.11526242758557653\n",
" (3, 3784)\t0.12716965106831488\n",
" (3, 1713)\t0.3432944013645088\n",
" (3, 3688)\t0.1074725684540511\n",
" (3, 861)\t0.26823488296435966\n",
" (3, 1540)\t0.12745407459398825\n",
" (3, 4032)\t0.2427573010643158\n",
" (3, 4091)\t0.11282000048679083\n",
" (3, 1711)\t0.059273701771436815\n",
" (4, 2718)\t0.3806528037580807\n",
" (4, 2846)\t0.3349928861087956\n",
" (4, 481)\t0.34914329741412814\n",
" (4, 1307)\t0.3806528037580807\n",
" (4, 1449)\t0.35782284493343813\n",
" (4, 2060)\t0.2959648137169803\n",
" (4, 3076)\t0.3349928861087956\n",
" (4, 2836)\t0.20790422248622678\n",
" (4, 4396)\t0.27313485489233774\n",
" (4, 3843)\t0.17437374334046382\n",
"<PandasArray>\n",
"['与本公司关系:受同一公司控制 2， ner_1646_ 司企业类型: 有限公司注册地址: 富阳市东洲街道东洲工业功能区九号路 1 号 法定代表人: ner_1645_ 明注册资本: ?16，000 万元经营范围: 许可经营项目:制造高效节能感应电机;普通货运。', '三、关联交易标的基本情况 1、交易标的基本情况 公司名称: ner_1648_ 司地址:无锡市锡山区东港镇锡港东路 2 号 企业类型:有限责任公司 法定代表人: ner_1647_ 燕注册资本:50000 万元整财务公司是公司参股公司，成立于 2008 年 11 月，经营范围为:对成员单位 办理财务和融资顾问、信用鉴证及相关的咨询、代理业务;协助成员单位实现交 易款项的收付;经批准的保险代理业务;对成员单位提供担保;办理成员单位之 间的委托贷款及委托投资;对成员单位办理票据承兑与贴现;办理成员单位之间 的内部转账结算及相应的结算、清算方案设计;吸收成员单位的存款;对成员单 位办理贷款及融资租赁;从事同业拆借;承销成员单位的企业债券;对金融机构 的股权投资;有价证券投资(股票二级市场投资除外);成员单位产品的买方信 贷。']\n",
"Length: 2, dtype: object\n"
]
}
],
"source": [
"# code\n",
"from sklearn.feature_extraction.text import TfidfTransformer \n",
"from sklearn.feature_extraction.text import CountVectorizer \n",
"from pyltp import Segmentor\n",
"import os\n",
"\n",
"\n",
"# 实体符号加入分词词典\n",
"# with open('../data/user_dict.txt', 'w') as fw:\n",
"# for v in ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']:\n",
"# fw.write( v + '号企业 ni\\n')\n",
"\n",
"with open('../data/user_dict.txt', 'w',encoding='utf-8') as fw:\n",
" for v in ner_dict_new.values():\n",
" fw.write('ner{}\\n'.format(str(v)))\n",
"\n",
"LTP_DATA_DIR = '/home/xiayulong/work/pyltp.d/models/ltp_data_v3.4.0' # ltp模型目录的路径\n",
"cws_model_path = os.path.join(LTP_DATA_DIR, 'cws.model') # 分词模型路径，模型名称为`cws.model`\n",
" \n",
"# 初始化实例\n",
"segmentor = Segmentor() \n",
"# 加载模型，加载自定义词典\n",
"segmentor.load_with_lexicon(cws_model_path, '../data/user_dict.txt') \n",
"\n",
"# 加载停用词\n",
"fr = open(r'../data/dict/stopwords.txt', encoding='utf-8') \n",
"stop_word = fr.readlines()\n",
"stop_word = [re.sub(r'(\\r|\\n)*','',stop_word[i]) for i in range(len(stop_word))]\n",
"\n",
"# 分词\n",
"# f = lambda x: ' '.join([word for word in segmentor.segment(x) if word not in stop_word and not re.findall(r'ner\\_\\d\\d\\d\\d\\_', word)])\n",
"# print(X['ner'].array[0])\n",
"# print([word for word in segmentor.segment('ner1001是个公司')])\n",
"# corpus=X['ner'].map(f).tolist()\n",
"\n",
"corpus = []\n",
"for sen in X['ner'].array:\n",
" for replace in re.findall(r'ner\\_\\d\\d\\d\\d\\_', sen):\n",
" sen = re.sub(replace,replace.replace('_',''),sen)\n",
" tmp = ' '.join([word for word in segmentor.segment(sen) if word not in stop_word and not re.findall(r'ner\\d\\d\\d\\d', word)])\n",
" corpus.append(tmp)\n",
"\n",
"\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"# TODO：提取tfidf特征\n",
"# TODO ...\n",
"tfidf = TfidfVectorizer()\n",
"vec = tfidf.fit_transform(corpus)\n",
"print(corpus[:5])\n",
"print(vec[:5])\n",
"print(X['ner'].array[:2])\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习4：提取句法特征\n",
"除了词语层面的句向量特征，我们还可以从句法入手，提取一些句法分析的特征。\n",
"\n",
"参考特征：\n",
"\n",
"1、企业实体间距离\n",
"\n",
"2、企业实体间句法距离\n",
"\n",
"3、企业实体分别和关键触发词的距离\n",
"\n",
"4、实体的依存关系类别"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"与本公司关系:受同一公司控制 2， ner_1646_ 司企业类型: 有限公司注册地址: 富阳市东洲街道东洲工业功能区九号路 1 号 法定代表人: ner_1645_ 明注册资本: ?16，000 万元经营范围: 许可经营项目:制造高效节能感应电机;普通货运。\n",
"[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0]\n",
"(1269, 4522)\n"
]
}
],
"source": [
"# -*- coding: utf-8 -*-\n",
"from pyltp import Parser\n",
"from pyltp import Segmentor\n",
"from pyltp import Postagger\n",
"import networkx as nx\n",
"import numpy as np\n",
"import pylab\n",
"import re\n",
"import os\n",
"\n",
"LTP_DATA_DIR = '/home/xiayulong/work/pyltp.d/models/ltp_data_v3.4.0' # ltp模型目录的路径\n",
"cws_model_path = os.path.join(LTP_DATA_DIR, 'cws.model') # 分词模型路径，模型名称为`cws.model`\n",
"pos_path = os.path.join(LTP_DATA_DIR, 'pos.model')\n",
"\n",
"\n",
"postagger = Postagger() # 初始化实例\n",
"postagger.load_with_lexicon(pos_path, '../data/user_dict.txt') # 加载模型\n",
"segmentor = Segmentor() # 初始化实例\n",
"segmentor.load_with_lexicon(cws_model_path, '../data/user_dict.txt') # 加载模型\n",
"parser = Parser() # 初始化实例\n",
"parser.load(os.path.join(LTP_DATA_DIR, 'parser.model')) # 加载模型\n",
"\n",
"\n",
"\n",
"# 以实体对为中心，提取特征是否会更好？\n",
"# 因为每句话中的实体数量不确定，有关系的实体数量更是不确定。以实体对为中心，可以明确观察这对实体在句法依存树中的关系，以及从实体1到实体2的路径\n",
"# 上是否有关键词，例如这里的\"支付\"，再辅以某些token级别的特征,例如某个size窗口内的tfidf的特征。这样，一句话可以生成多个样本，目标变成预测每\n",
"# 一对实体所构成的样本的关系\n",
"\n",
"\n",
"relations = [\"SBV\",\n",
" \"VOB\",\n",
" \"IOB\",\n",
" \"FOB\",\n",
" \"DBL\",\n",
" \"ATT\",\n",
" \"ADV\",\n",
" \"CMP\",\n",
" \"COO\",\n",
" \"POB\",\n",
" \"LAD\",\n",
" \"RAD\",\n",
" \"IS\",\n",
" \"HED\"]\n",
"\n",
"\n",
"def parse(s):\n",
" \"\"\"\n",
" 对语句进行句法分析，并返回句法结果\n",
" \"\"\"\n",
" tmp_ner_dict = {}\n",
" num_lst = ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']\n",
"\n",
" # 将公司代码替换为特殊称谓，保证分词词性正确\n",
"# for i, ner in enumerate(list(set(re.findall(r'(ner\\_\\d\\d\\d\\d\\_)', s)))):\n",
"# try:\n",
"# tmp_ner_dict[num_lst[i]+'号企业'] = ner\n",
"# except IndexError:\n",
"# # TODO：定义错误情况的输出\n",
"# # TODO ...\n",
" \n",
" \n",
"# s = s.replace(ner, num_lst[i]+'号企业')\n",
" for i, ner in enumerate(list(set(re.findall(r'(ner\\_\\d\\d\\d\\d\\_)', s)))):\n",
" s = re.sub(ner,ner.replace('_',''),s)\n",
" words = segmentor.segment(s)\n",
" tags = postagger.postag(words)\n",
"\n",
" arcs = parser.parse(words, tags) # 句法分析\n",
" arcs_lst = list(map(list, zip(*[[arc.head, arc.relation] for arc in arcs])))\n",
" # 句法分析结果输出\n",
" parse_result = pd.DataFrame([[a,b,c,d] for a,b,c,d in zip(list(words),list(tags), arcs_lst[0], arcs_lst[1])], index = range(1,len(words)+1))\n",
" \n",
"\n",
" # TODO：提取企业实体依存句法类型\n",
" # TODO ...\n",
" # 句子只有两个实体（俩公司或一公司一人名）\n",
" # 找前两个实体间\n",
" dict_of_relation = {relation:0 for relation in relations}\n",
" ners = list()\n",
" for i in list(parse_result.index):\n",
" word = parse_result.loc[i,0]\n",
" relation = parse_result.loc[i,3]\n",
" if re.findall(r'(ner\\d{4})', word):\n",
" if word not in ners:\n",
" ners.append(word)\n",
" ner_id = word[3:]\n",
" if ner_id.isdigit() and int(ner_id) in set_company:\n",
" dict_of_relation[relation] += 1\n",
" \n",
" count_of_relation = list(dict_of_relation.values())\n",
" \n",
" \n",
"\n",
" # 投资关系关键词\n",
" key_words = [\"收购\",\"竞拍\",\"转让\",\"扩张\",\"并购\",\"注资\",\"整合\",\"并入\",\"竞购\",\"竞买\",\"支付\",\"收购价\",\"收购价格\",\"承购\",\"购得\",\"购进\",\n",
" \"购入\",\"买进\",\"买入\",\"赎买\",\"购销\",\"议购\",\"函购\",\"函售\",\"抛售\",\"售卖\",\"销售\",\"转售\"]\n",
" # TODO：*根据关键词和对应句法关系提取特征（如没有思路可以不完成）\n",
" # TODO ...\n",
" # distances[0]: 实体之间距离\n",
" # distances[1]: 实体1和关键词之间距离\n",
" # distances[2]: 实体2和关键词之间距离\n",
" distances = [0,0,0]\n",
" if len(ners) >= 2:\n",
" # 实体1\n",
" entity1 = ners[0]\n",
" # 实体2\n",
" entity2 = ners[1]\n",
" distances[0] = shortest_path(parse_result,entity1,entity2)[0]\n",
" for i in parse_result.index:\n",
" word = parse_result.loc[i,0]\n",
" if word in key_words:\n",
" distances[1] = shortest_path(parse_result,entity1,word)[0]\n",
" distances[2] = shortest_path(parse_result,entity2,word)[0]\n",
" \n",
" \n",
"\n",
" \n",
" \n",
" # parser.release() # 释放模型\n",
" return count_of_relation + distances\n",
"\n",
"\n",
"def get_weight(type):\n",
" return 1.\n",
"\n",
"def print_graph(G):\n",
" for n, nbrs in G.adj.items():\n",
" for nbr, eattr in nbrs.items():\n",
" wt = eattr['weight']\n",
" print(f\"({n}, {nbr}, {wt:.3})\")\n",
"\n",
"\n",
"\n",
"\n",
"def shortest_path(arcs_ret, source, target):\n",
" \"\"\"\n",
" 求出两个词最短依存句法路径，不存在路径返回-1\n",
" arcs_ret：句法分析结果\n",
" source：实体1\n",
" target：实体2\n",
" \"\"\"\n",
" G=nx.DiGraph()\n",
" # 为这个网络添加节点...\n",
" for i in list(arcs_ret.index):\n",
" G.add_node(i)\n",
" # TODO：在网络中添加带权中的边...（注意，我们需要的是无向边）\n",
" # TODO ...\n",
" for cur in list(arcs_ret.index):\n",
" predecessor = arcs_ret.loc[cur,2]\n",
" if predecessor == 0:\n",
" continue\n",
" weight = get_weight(arcs_ret.loc[i,3])\n",
" G.add_edge(predecessor,cur,weight=weight)\n",
" undirected = nx.Graph(G)\n",
" try:\n",
" # TODO：利用nx包中shortest_path_length方法实现最短距离提取\n",
" # TODO ...\n",
" path = nx.shortest_path(undirected,source,target)\n",
" distance = len(path)\n",
" \n",
" return (distance,path)\n",
" except:\n",
" return (-1,None)\n",
"\n",
"\n",
"def get_feature(X,tfidf_vec):\n",
" \"\"\"\n",
" 汇总上述函数汇总句法分析特征与TFIDF特征\n",
" \"\"\"\n",
" # TODO：汇总上述函数汇总句法分析特征与TFIDF特征\n",
" # TODO ...\n",
" syntactic_features = []\n",
" for sentence in X['ner'].array:\n",
" syntactic_features.append(parse(sentence))\n",
" features = np.concatenate([tfidf_vec.toarray(), syntactic_features],axis=1)\n",
" return features\n",
"sen = X['ner'].array[0]\n",
"\n",
"result = parse(sen)\n",
"print(sen)\n",
"print(result)\n",
"print(vec.shape)\n",
"features = get_feature(X,vec)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习5：建立分类器\n",
"\n",
"利用已经提取好的tfidf特征以及parse特征，建立分类器进行分类任务。"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/model_selection/_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
" warnings.warn(CV_WARNING, FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
" FutureWarning)\n",
"/home/xiayulong/anaconda3-2019.07/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
" y = column_or_1d(y, warn=True)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'mean_fit_time': array([0.02740773, 0.02587167, 0.03985373, 0.02955047]), 'std_fit_time': array([0.00234381, 0.00048195, 0.00140094, 0.00162232]), 'mean_score_time': array([0.00247908, 0.00247018, 0.00244745, 0.00244482]), 'std_score_time': array([4.81680276e-05, 3.60999287e-05, 4.03044161e-05, 1.54933354e-05]), 'param_C': masked_array(data=[1, 1, 10, 10],\n",
" mask=[False, False, False, False],\n",
" fill_value='?',\n",
" dtype=object), 'param_penalty': masked_array(data=['l1', 'l2', 'l1', 'l2'],\n",
" mask=[False, False, False, False],\n",
" fill_value='?',\n",
" dtype=object), 'params': [{'C': 1, 'penalty': 'l1'}, {'C': 1, 'penalty': 'l2'}, {'C': 10, 'penalty': 'l1'}, {'C': 10, 'penalty': 'l2'}], 'split0_test_score': array([0.89788732, 0.88732394, 0.92605634, 0.92253521]), 'split1_test_score': array([0.93309859, 0.92957746, 0.94014085, 0.95774648]), 'split2_test_score': array([0.90070922, 0.92198582, 0.96099291, 0.94326241]), 'mean_test_score': array([0.91058824, 0.91294118, 0.94235294, 0.94117647]), 'std_test_score': array([0.01598684, 0.01840855, 0.01434018, 0.0144667 ]), 'rank_test_score': array([4, 3, 1, 2], dtype=int32)}\n",
"{'C': 10, 'penalty': 'l2'}\n",
"850\n",
"1269 850\n",
"[['2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》'\n",
" '2015年1月26日， ner_1002_ 司与 ner_1001_ 峰先生签署了《附条件生效的股份认购合同》' '李云峰' '氟化工' 1]\n",
" ['2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先'\n",
" '2、2016年2月5日， ner_1004_ 司与 ner_1003_ 毅先' '侯毅' '深圳市新纶科技股份' 0]\n",
" ['2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》'\n",
" '2015年10月26日， ner_1006_ 司与 ner_1005_ 华先生签署了附条件生效条件的《股份认购合同》' '山东华鹏玻璃'\n",
" '张德华' 1]\n",
" ['2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司非公开发行A股股票之附条件生效的股份认购协议》（以下简称“《附条件生效的股份认购协议》”）'\n",
" '2、2015年12月31日， ner_1008_ 司与 ner_1007_ 革签订了《印纪娱乐传媒股份有限公司非公开发行A股股票之附条件生效的股份认购协议》（以下简称“《附条件生效的股份认购协议》”）'\n",
" '印纪娱乐传媒' '肖文革' 1]\n",
" ['一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收购其所持有的金发小贷13.50%的股份（2，700万股），收购金额为2，837.43万元'\n",
" '一、 ner_1010_ 技拟与 ner_1009_ 涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收购其所持有的金发小贷13.50%的股份（2，700万股），收购金额为2，837.43万元'\n",
" '熊海涛' '金发科技' 0]]\n"
]
},
{
"ename": "ValueError",
"evalue": "invalid literal for int() with base 10: '李云峰'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-59-a08d1dfec342>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 53\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 54\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 56\u001b[0m \u001b[0minfo_extract_submit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mner_dict_new\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mner_dict_new\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_results\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0mdf_info_extract_entity\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minfo_extract_entity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"实体编号\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\"实体名\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mValueError\u001b[0m: invalid literal for int() with base 10: '李云峰'"
]
}
],
"source": [
"# 建立分类器进行分类\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn import preprocessing\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import GridSearchCV\n",
"\n",
"# TODO：定义需要遍历的参数\n",
"parameters = { 'penalty':('l1', 'l2'), 'C':[1, 10]}\n",
"\n",
"# TODO：选择模型\n",
"lr = LogisticRegression(random_state=0)\n",
"clf = GridSearchCV(lr, parameters)\n",
"\n",
"# TODO：利用GridSearchCV搜索最佳参数\n",
"clf.fit(features[:train_num], y[:train_num])\n",
"print(clf.cv_results_)\n",
"best_param = clf.cv_results_[\"params\"][clf.cv_results_[\"rank_test_score\"][0]-1]\n",
"print(best_param)\n",
"# print(clf.cv_results_[\"params\"])\n",
"# print(clf.cv_results_[\"rank_test_score\"])\n",
"\n",
"# TODO：对Test_data进行分类\n",
"print(train_num)\n",
"print(len(features),len(y))\n",
"result = clf.predict(features[train_num:])\n",
"result = result.reshape([len(result),1])\n",
"\n",
"\n",
"# TODO：保存Test_data分类结果\n",
"# 答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
"# info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
"# info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
"sentenceWithNerIdAndMembers = []\n",
"info_extract_entity = dict()\n",
"info_extract_submit = []\n",
"for sentenceWithNerId in X['ner'].array[train_num:]:\n",
" nerids = []\n",
" for i, ner in enumerate(list(set(re.findall(r'(ner\\_\\d\\d\\d\\d\\_)', sentenceWithNerId)))):\n",
" if ner[4:8].isdigit() and int(ner[4:8]) not in nerids:\n",
" nerids.append(int(ner[4:8]))\n",
" info_extract_entity[int(ner[4:8])] = ner_dict_reverse_new[int(ner[4:8])]\n",
" entity1 = \"0\"\n",
" entity2 = \"0\"\n",
" if len(nerids) >= 2:\n",
" entity1 = ner_dict_reverse_new[nerids[0]]\n",
" entity2 = ner_dict_reverse_new[nerids[1]]\n",
" sentenceWithNerIdAndMembers.append([sentenceWithNerId,entity1,entity2])\n",
"\n",
"test_sentences = test_data.values[:,1].reshape([len(test_data),1])\n",
"\n",
"pred_results = np.concatenate([test_sentences,sentenceWithNerIdAndMembers,result],axis=1)\n",
"print(pred_results[:5])\n",
"for i in range(len(pred_results)):\n",
" if int(pred_results[i,-1]) == 1 and int(pred_results[i,2]) != 0 and int(pred_results[i,3]) != 0:\n",
" info_extract_submit.append([ner_dict_new[pred_results[i,2]],ner_dict_new[pred_results[i,3]]])\n",
"df_info_extract_entity = pd.DataFrame(list(info_extract_entity.items()),columns=[\"实体编号\",\"实体名\"])\n",
"df_info_extract_submit = pd.DataFrame(info_extract_submit,columns=[\"实体1\",\"实体2\"])\n",
"\n",
"df_info_extract_entity.to_csv(\"../submits/df_info_extract_entity.csv\")\n",
"df_info_extract_submit.to_csv(\"../submits/df_info_extract_submit.csv\")\n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 练习6：操作图数据库\n",
"对关系最好的描述就是用图，那这里就需要使用图数据库，目前最常用的图数据库是noe4j，通过cypher语句就可以操作图数据库的增删改查。可以参考“https://cuiqingcai.com/4778.html”。\n",
"\n",
"本次作业我们使用neo4j作为图数据库，neo4j需要java环境，请先配置好环境。\n",
"\n",
"将我们提出的实体关系插入图数据库，并查询某节点的3层投资关系，即三个节点组成的路径（如果有的话）。如果无法找到3层投资关系，请查询出任意指定节点的投资路径。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"\n",
"from py2neo import Node, Relationship, Graph\n",
"\n",
"graph = Graph(\n",
" \"http://localhost:7474\", \n",
" username=\"neo4j\", \n",
" password=\"person\"\n",
")\n",
"\n",
"for v in relation_list:\n",
" a = Node('Company', name=v[0])\n",
" b = Node('Company', name=v[1])\n",
" \n",
" # 本次不区分投资方和被投资方，无向图\n",
" r = Relationship(a, 'INVEST', b)\n",
" s = a | b | r\n",
" graph.create(s)\n",
" r = Relationship(b, 'INVEST', a)\n",
" s = a | b | r\n",
" graph.create(s)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# TODO：查询某节点的3层投资关系\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 步骤4：实体消歧\n",
"解决了实体识别和关系的提取，我们已经完成了一大截，但是我们提取的实体究竟对应知识库中哪个实体呢？下图中，光是“苹果”就对应了13个同名实体。\n",
"<img src=\"../image/baike2.png\", width=340, heigth=480>\n",
"\n",
"在这个问题上，实体消歧旨在解决文本中广泛存在的名称歧义问题，将句中识别的实体与知识库中实体进行匹配，解决实体歧义问题。\n",
"\n",
"\n",
"### 练习7：\n",
"匹配test_data.csv中前25条样本中的人物实体对应的百度百科URL（此部分样本中所有人名均可在百度百科中链接到）。\n",
"\n",
"利用scrapy、beautifulsoup、request等python包对百度百科进行爬虫，判断是否具有一词多义的情况，如果有的话，选择最佳实体进行匹配。\n",
"\n",
"使用URL为‘https://baike.baidu.com/item/’+人名 可以访问百度百科该人名的词条，此处需要根据爬取到的网页识别该词条是否对应多个实体，如下图：\n",
"<img src=\"../image/baike1.png\", width=440, heigth=480>\n",
"如果该词条有对应多个实体，请返回正确匹配的实体URL，例如该示例网页中的‘https://baike.baidu.com/item/陆永/20793929’。\n",
"\n",
"- 提交文件：entity_disambiguation_submit.csv\n",
"- 提交格式：第一列为实体id（与info_extract_submit.csv中id保持一致），第二列为对应URL。\n",
"- 示例：\n",
"\n",
"| 实体编号 | URL |\n",
"| ------ | ------ |\n",
"| 1001 | https://baike.baidu.com/item/陆永/20793929 |\n",
"| 1002 | https://baike.baidu.com/item/王芳/567232 |\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import jieba\n",
"import pandas as pd\n",
"\n",
"# 找出test_data.csv中前25条样本所有的人物名称，以及人物所在文档的上下文内容\n",
"test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
"\n",
"# 存储人物以及上下文信息（key为人物ID，value为人物名称、人物上下文内容）\n",
"person_name = {}\n",
"\n",
"# 观察上下文的窗口大小\n",
"window = 10 \n",
"\n",
"# 遍历前25条样本\n",
"for i in range(25):\n",
" sentence = copy(test_data.iloc[i, 1])\n",
" words, ners = fool.analysis(sentence)\n",
" ners[0].sort(key=lambda x:x[0], reverse=True)\n",
" for start, end, ner_type, ner_name in ners[0]:\n",
" if ner_type=='person':\n",
" # TODO：提取实体的上下文\n",
" \n",
"\n",
"\n",
"\n",
"# 利用爬虫得到每个人物名称对应的URL\n",
"# TODO：找到每个人物实体的词条内容。\n",
"\n",
"# TODO：将样本中人物上下文与爬取词条结果进行对比，选择最接近的词条。\n",
"\n",
"\n",
"\n",
"# 输出结果\n",
"pd.DataFrame(result_data).to_csv('../submit/entity_disambiguation_submit.csv', index=False)\n"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}