0920知识图谱代码.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 利用信息抽取技术搭建知识库\n",
    "\n",
    "在这个notebook文件中，有些模板代码已经提供给你，但你还需要实现更多的功能来完成这个项目。除非有明确要求，你无须修改任何已给出的代码。以**'【练习】'**开始的标题表示接下来的代码部分中有你需要实现的功能。这些部分都配有详细的指导，需要实现的部分也会在注释中以'TODO'标出。请仔细阅读所有的提示。\n",
    "\n",
    ">**提示：**Code 和 Markdown 区域可通过 **Shift + Enter** 快捷键运行。此外，Markdown可以通过双击进入编辑模式。\n",
    "\n",
    "---\n",
    "\n",
    "### 让我们开始吧\n",
    "\n",
    "本项目的目的是结合命名实体识别、依存语法分析、实体消歧、实体统一对网站开放语料抓取的数据建立小型知识图谱。\n",
    "\n",
    "在现实世界中，你需要拼凑一系列的模型来完成不同的任务；举个例子，用来预测狗种类的算法会与预测人类的算法不同。在做项目的过程中，你可能会遇到不少失败的预测，因为并不存在完美的算法和模型。你最终提交的不完美的解决方案也一定会给你带来一个有趣的学习经验！\n",
    "\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤 1：实体统一"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实体统一做的是对同一实体具有多个名称的情况进行统一，将多种称谓统一到一个实体上，并体现在实体的属性中（可以给实体建立“别称”属性）\n",
    "\n",
    "例如：对“河北银行股份有限公司”、“河北银行公司”和“河北银行”我们都可以认为是一个实体，我们就可以将通过提取前两个称谓的主要内容，得到“河北银行”这个实体关键信息。\n",
    "\n",
    "公司名称有其特点，例如后缀可以省略、上市公司的地名可以省略等等。在data/dict目录中提供了几个词典，可供实体统一使用。\n",
    "- company_suffix.txt是公司的通用后缀词典\n",
    "- company_business_scope.txt是公司经营范围常用词典\n",
    "- co_Province_Dim.txt是省份词典\n",
    "- co_City_Dim.txt是城市词典\n",
    "- stopwords.txt是可供参考的停用词\n",
    "\n",
    "### 练习1：\n",
    "编写main_extract函数，实现对实体的名称提取“主体名称”的功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba\n",
    "import re\n",
    "import datetime\n",
    "\n",
    "\n",
    "#TODO：替换特殊符号\n",
    "def remove_word(seg,stop_word,d_4_delete):\n",
    "    # TODO ...\n",
    "    temp_seg_lst = [word for word in seg if word not in stop_word]\n",
    "    seg_lst = [word for word in temp_seg_lst if word not in d_4_delete]\n",
    "    return seg_lst\n",
    "\n",
    "\n",
    "#TODO：实现公司名称中地名提前\n",
    "def city_prov_ahead(seg,d_city_province):\n",
    "    # TODO ...\n",
    "    city_prov_lst = [word for word in seg if word in d_city_province]\n",
    "    seg_lst = [word for word in seg if word not in d_city_province]\n",
    "    return city_prov_lst+seg_lst\n",
    "\n",
    "\n",
    "# 从输入的“公司名”中提取主体\n",
    "def main_extract(input_str,stop_word,d_4_delete,d_city_province):\n",
    "    # 开始分词并处理\n",
    "    seg = jieba.lcut(input_str)\n",
    "    seg_lst = remove_word(seg,stop_word,d_4_delete)\n",
    "    seg_lst = city_prov_ahead(seg_lst, d_city_province)\n",
    "    return seg_lst\n",
    "\n",
    "\n",
    "# 初始化，加载词典\n",
    "def my_initial():\n",
    "    fr1 = open(r\"../data/dict/co_City_Dim.txt\", encoding='utf-8')\n",
    "    fr2 = open(r\"../data/dict/co_Province_Dim.txt\", encoding='utf-8')\n",
    "    fr3 = open(r\"../data/dict/company_business_scope.txt\", encoding='utf-8')\n",
    "    fr4 = open(r\"../data/dict/company_suffix.txt\", encoding='utf-8')\n",
    "    # 城市名\n",
    "    lines1 = fr1.readlines()\n",
    "    d_4_delete = []\n",
    "    d_city_province = [re.sub(r'(\\r|\\n)*','',line) for line in lines1]\n",
    "    # 省份名\n",
    "    lines2 = fr2.readlines()\n",
    "    l2_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines2]\n",
    "    d_city_province.extend(l2_tmp)\n",
    "    # 公司后缀\n",
    "    lines3 = fr3.readlines()\n",
    "    l3_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines3]\n",
    "    lines4 = fr4.readlines()\n",
    "    l4_tmp = [re.sub(r'(\\r|\\n)*','',line) for line in lines4]\n",
    "    d_4_delete.extend(l4_tmp)\n",
    "    # get stop_word\n",
    "    fr = open(r'../data/dict/stopwords.txt', encoding='utf-8')   \n",
    "    stop_word = fr.readlines()\n",
    "    stop_word_after = [re.sub(r'(\\r|\\n)*', '', item) for item in stop_word]\n",
    "    stop_word_after[-1] = stop_word[-1]\n",
    "    stop_word = stop_word_after\n",
    "    return d_4_delete, stop_word, d_city_province"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /var/folders/b1/pjv8l_h14w98838dm0r6qyr80000gn/T/jieba.cache\n",
      "Loading model cost 1.058 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "河北银行\n"
     ]
    }
   ],
   "source": [
    "# TODO：测试实体统一用例\n",
    "d_4_delete, stop_word, d_city_province = my_initial()\n",
    "company_name = \"河北银行股份有限公司\"\n",
    "lst = main_extract(company_name,stop_word,d_4_delete,d_city_province)\n",
    "company_name = ''.join(lst)  # 对公司名提取主体部分，将包含相同主体部分的公司统一为一个实体\n",
    "print(company_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤 2：实体识别\n",
    "有很多开源工具可以帮助我们对实体进行识别。常见的有LTP、StanfordNLP、FoolNLTK等等。\n",
    "\n",
    "本次采用FoolNLTK实现实体识别，fool是一个基于bi-lstm+CRF算法开发的深度学习开源NLP工具，包括了分词、实体识别等功能，大家可以通过fool很好地体会深度学习在该任务上的优缺点。\n",
    "\n",
    "在‘data/train_data.csv’和‘data/test_data.csv’中是从网络上爬虫得到的上市公司公告，数据样例如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "      <th>member1</th>\n",
       "      <th>member2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6461</td>\n",
       "      <td>与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2111</td>\n",
       "      <td>三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9603</td>\n",
       "      <td>2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515...</td>\n",
       "      <td>1</td>\n",
       "      <td>协鑫集成科技股份有限公司</td>\n",
       "      <td>瑞峰（张家港）光伏科技有限公司</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3456</td>\n",
       "      <td>证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8844</td>\n",
       "      <td>本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数...</td>\n",
       "      <td>1</td>\n",
       "      <td>广发证券股份有限公司</td>\n",
       "      <td>辽宁成大股份有限公司</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id                                           sentence  tag       member1  \\\n",
       "0  6461  与本公司关系:受同一公司控制 2，杭州富生电器有限公司企业类型: 有限公司注册地址: 富阳市...    0             0   \n",
       "1  2111  三、关联交易标的基本情况 1、交易标的基本情况 公司名称:红豆集团财务有限公司 公司地址:无...    0             0   \n",
       "2  9603  2016年协鑫集成科技股份有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，515...    1  协鑫集成科技股份有限公司   \n",
       "3  3456  证券代码:600777 证券简称:新潮实业 公告编号:2015-091 烟台新潮实业股份有限...    0             0   \n",
       "4  8844  本集团及广发证券股份有限公司持有辽宁成大股份有限公司股票的本期变动系买卖一揽子沪深300指数...    1    广发证券股份有限公司   \n",
       "\n",
       "           member2  \n",
       "0                0  \n",
       "1                0  \n",
       "2  瑞峰（张家港）光伏科技有限公司  \n",
       "3                0  \n",
       "4       辽宁成大股份有限公司  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9259</td>\n",
       "      <td>2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9136</td>\n",
       "      <td>2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>220</td>\n",
       "      <td>2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9041</td>\n",
       "      <td>2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10041</td>\n",
       "      <td>一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                           sentence\n",
       "0   9259      2015年1月26日，多氟多化工股份有限公司与李云峰先生签署了《附条件生效的股份认购合同》\n",
       "1   9136                      2、2016年2月5日，深圳市新纶科技股份有限公司与侯毅先\n",
       "2    220  2015年10月26日，山东华鹏玻璃股份有限公司与张德华先生签署了附条件生效条件的《股份认购合同》\n",
       "3   9041  2、2015年12月31日，印纪娱乐传媒股份有限公司与肖文革签订了《印纪娱乐传媒股份有限公司...\n",
       "4  10041  一、金发科技拟与熊海涛女士签订《股份转让协议》，协议约定：以每股1.0509元的收购价格，收..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
    "test_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们选取一部分样本进行标注，即train_data，该数据由5列组成。id列表示原始样本序号；sentence列为我们截取的一段关键信息；如果关键信息中存在两个实体之间有股权交易关系则tag列为1，否则为0；如果tag为1，则在member1和member2列会记录两个实体出现在sentence中的名称。\n",
    "\n",
    "剩下的样本没有标注，即test_data，该数据只有id和sentence两列，希望你能训练模型对test_data中的实体进行识别，并判断实体对之间有没有股权交易关系。\n",
    "\n",
    "### 练习2：\n",
    "将每句句子中实体识别出，存入实体词典，并用特殊符号替换语句。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use the retry module or similar alternatives.\n",
      "-----------words---------\n",
      "[[('我', 'r'), ('上周五', 't'), ('在', 'p'), ('国信', 'nz'), ('证券', 'n'), ('泰然九路', 'nz'), ('营业部', 'n'), ('购买', 'v'), ('了', 'y'), ('TCL', 'nz'), ('集团', 'n'), ('的', 'ude'), ('股票', 'n'), ('，', 'wd'), ('希望', 'v'), ('能', 'v'), ('挣够', 'ns'), ('和', 'c'), ('江晓涛', 'ns'), ('小盆友', 'ns'), ('一起', 's'), ('下个', 'nx'), ('月', 'n'), ('去', 'v'), ('安徽', 'ns'), ('黄山', 'ns'), ('旅游', 'vi'), ('的', 'ude'), ('钱', 'n')]]\n",
      "-----------ners----------\n",
      "[[(1, 4, 'time', '上周五'), (5, 16, 'company', '国信证券泰然九路营业部'), (19, 24, 'company', 'TCL集团'), (34, 37, 'person', '江晓涛'), (42, 45, 'time', '下个月'), (46, 50, 'location', '安徽黄山')]]\n"
     ]
    }
   ],
   "source": [
    "import fool\n",
    "\n",
    "# 测试一下fool模块的功能\n",
    "test_string = '我上周五在国信证券泰然九路营业部购买了TCL集团的股票，希望能挣够和江晓涛小盆友一起下个月去安徽黄山旅游的钱'\n",
    "words, ners = fool.analysis(test_string)\n",
    "print('-----------words---------')\n",
    "print(words)\n",
    "print('-----------ners----------')\n",
    "print(ners)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ner</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015年1月26日， ner_1002_ 司与 ner_1001_ 峰先生签署了《附条件生...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2、2016年2月5日， ner_1004_ 司与 ner_1003_ 毅先</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015年10月26日， ner_1006_ 司与 ner_1005_ 华先生签署了附条件生...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2、2015年12月31日， ner_1008_ 司与 ner_1007_ 革签订了《印纪娱...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>一、 ner_1010_ 技拟与 ner_1009_ 涛女士签订《股份转让协议》，协议约定：...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 ner\n",
       "0  2015年1月26日， ner_1002_ 司与 ner_1001_ 峰先生签署了《附条件生...\n",
       "1             2、2016年2月5日， ner_1004_ 司与 ner_1003_ 毅先\n",
       "2  2015年10月26日， ner_1006_ 司与 ner_1005_ 华先生签署了附条件生...\n",
       "3  2、2015年12月31日， ner_1008_ 司与 ner_1007_ 革签订了《印纪娱...\n",
       "4  一、 ner_1010_ 技拟与 ner_1009_ 涛女士签订《股份转让协议》，协议约定：..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 处理test数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
    "import fool\n",
    "from copy import copy\n",
    "\n",
    "import collections\n",
    "test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
    "test_data['ner'] = None\n",
    "ner_id = 1001\n",
    "ner_dict_new = {}  # 存储所有实体统一后结果与其编号对应关系\n",
    "#ner_dict_reverse_new = {}  # 存储所有实体统一后结果与其未统一前的实体名称对应关系\n",
    "\n",
    "ner_dict_reverse_new = collections.defaultdict(set)\n",
    "\n",
    "for i in range(len(test_data)):\n",
    "    sentence = copy(test_data.iloc[i, 1])\n",
    "    sentence = sentence.strip()\n",
    "    sentence = sentence.replace(' ', '')\n",
    "    \n",
    "    # TODO：调用fool进行实体识别，得到words和ners结果\n",
    "    # TODO ...\n",
    "    words, ners = fool.analysis(sentence)\n",
    "    ners[0].sort(key=lambda x:x[0], reverse=True)\n",
    "    for start, end, ner_type, ner_name in ners[0]:\n",
    "        if ner_type=='company' or ner_type=='person':\n",
    "            # TODO：调用实体统一函数，存储统一后的实体\n",
    "            # 并自增ner_id\n",
    "            # TODO ...\n",
    "            ner_lst = main_extract(ner_name,stop_word,d_4_delete,d_city_province)\n",
    "            company_main_name = ''.join(ner_lst)\n",
    "\n",
    "            # 存储所有实体统一后结果与其未统一前的实体名称对应关系\n",
    "            # if company_main_name not in ner_dict_reverse_new:\n",
    "            #     ner_dict_reverse_new[company_main_name] = [ner_name]\n",
    "            # else:\n",
    "            #     if ner_name not in ner_dict_reverse_new[company_main_name]:\n",
    "            #         ner_dict_reverse_new[company_main_name].append(ner_name)\n",
    "            ner_dict_reverse_new[company_main_name].add(ner_name)\n",
    "            # 存储所有实体统一后结果与其编号对应关系\n",
    "            if company_main_name not in ner_dict_new:\n",
    "                ner_dict_new[company_main_name] = ner_id\n",
    "                ner_id += 1\n",
    "            \n",
    "            # 在句子中用编号替换实体名\n",
    "            sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end-1:]\n",
    "    test_data.iloc[i, -1] = sentence\n",
    "\n",
    "X_test_sentence = test_data[['ner']]\n",
    "X_test_sentence.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ner</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>与本公司关系:受同一公司控制2， ner_1611_ 司企业类型:有限公司注册地址:富阳市东...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>三、关联交易标的基本情况1、交易标的基本情况公司名称: ner_1613_ 司地址:无锡市锡...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016年 ner_1614_ 有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>证券代码:600777证券简称: ner_1198_ 业公告编号:2015-091 ner_...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>本集团及 ner_1089_ 股份有限公司持有 ner_1616_ 大股份有限公司股票的本期...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 ner\n",
       "0  与本公司关系:受同一公司控制2， ner_1611_ 司企业类型:有限公司注册地址:富阳市东...\n",
       "1  三、关联交易标的基本情况1、交易标的基本情况公司名称: ner_1613_ 司地址:无锡市锡...\n",
       "2  2016年 ner_1614_ 有限公司向瑞峰（张家港）光伏科技有限公司支付设备款人民币4，...\n",
       "3  证券代码:600777证券简称: ner_1198_ 业公告编号:2015-091 ner_...\n",
       "4  本集团及 ner_1089_ 股份有限公司持有 ner_1616_ 大股份有限公司股票的本期..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 处理train数据，利用开源工具进行实体识别和并使用实体统一函数存储实体\n",
    "train_data = pd.read_csv('../data/info_extract/train_data.csv', encoding = 'gb2312', header=0)\n",
    "train_data['ner'] = None\n",
    "\n",
    "\n",
    "for i in range(len(train_data)):\n",
    "    # 判断正负样本\n",
    "    if train_data.iloc[i,:]['member1']=='0' and train_data.iloc[i,:]['member2']=='0':\n",
    "        sentence = copy(train_data.iloc[i, 1])\n",
    "        sentence = sentence.strip()\n",
    "        sentence = sentence.replace(' ', '')\n",
    "        \n",
    "        # TODO：调用fool进行实体识别，得到words和ners结果\n",
    "        # TODO ...\n",
    "        words, ners = fool.analysis(sentence)\n",
    "    \n",
    "        ners[0].sort(key=lambda x:x[0], reverse=True)\n",
    "        for start, end, ner_type, ner_name in ners[0]:\n",
    "            if ner_type=='company' or ner_type=='person':\n",
    "                # TODO：调用实体统一函数，存储统一后的实体\n",
    "                # 并自增ner_id\n",
    "                # TODO ...\n",
    "                ner_lst = main_extract(ner_name,stop_word,d_4_delete,d_city_province)\n",
    "                company_main_name = ''.join(ner_lst)\n",
    "                \n",
    "                # 存储所有实体统一后结果与其未统一前的实体名称对应关系\n",
    "                # if company_main_name not in ner_dict_reverse_new:\n",
    "                #    ner_dict_reverse_new[company_main_name] = [ner_name]\n",
    "                # else:\n",
    "                #    if ner_name not in ner_dict_reverse_new[company_main_name]:\n",
    "                #        ner_dict_reverse_new[company_main_name].add(ner_name)\n",
    "                ner_dict_reverse_new[company_main_name].add(ner_name)\n",
    "                # 存储所有实体统一后结果与其编号对应关系\n",
    "                if company_main_name not in ner_dict_new:\n",
    "                    ner_dict_new[company_main_name] = ner_id\n",
    "                    ner_id += 1\n",
    "\n",
    "                # 在句子中用编号替换实体名\n",
    "                sentence = sentence[:start] + ' ner_' + str(ner_dict_new[company_main_name]) + '_ ' + sentence[end-1:]\n",
    "        train_data.iloc[i, -1] = sentence\n",
    "    else:\n",
    "        # 将训练集中正样本已经标注的实体，直接使用编码替换(免去用工具进行实体识别步骤)\n",
    "        sentence = copy(train_data.iloc[i,:]['sentence'])\n",
    "        sentence = sentence.strip()\n",
    "        sentence = sentence.replace(' ', '')\n",
    "        \n",
    "        for ner_name in [train_data.iloc[i,:]['member1'],train_data.iloc[i,:]['member2']]:\n",
    "            # TODO：调用实体统一函数，存储统一后的实体\n",
    "            # 并自增ner_id\n",
    "            # TODO ...\n",
    "            ner_lst = main_extract(ner_name,stop_word,d_4_delete,d_city_province)\n",
    "            company_main_name = ''.join(ner_lst)\n",
    "\n",
    "            # 存储所有实体统一后结果与其未统一前的实体名称对应关系\n",
    "            # if company_main_name not in ner_dict_reverse_new:\n",
    "            #   ner_dict_reverse_new[company_main_name] = [ner_name]\n",
    "            # else:\n",
    "            #    if ner_name not in ner_dict_reverse_new[company_main_name]:\n",
    "            #        ner_dict_reverse_new[company_main_name].add(ner_name)\n",
    "            ner_dict_reverse_new[company_main_name].add(ner_name)\n",
    "            \n",
    "            # 存储所有实体统一后结果与其编号对应关系\n",
    "            if company_main_name not in ner_dict_new:\n",
    "                ner_dict_new[company_main_name] = ner_id\n",
    "                ner_id += 1\n",
    "\n",
    "            # 在句子中用编号替换实体名\n",
    "            sentence = re.sub(company_main_name, ' ner_%s_ '%(str(ner_dict_new[company_main_name])), sentence)\n",
    "        train_data.iloc[i, -1] = sentence\n",
    "        \n",
    "y_train = train_data.loc[:, ['tag']]\n",
    "train_num = len(train_data)\n",
    "X_train_sentence = train_data[['ner']]\n",
    "\n",
    "# 将train和test放在一起提取特征\n",
    "X_sentence = pd.concat([X_train_sentence, X_test_sentence])\n",
    "X_sentence.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤 3：关系抽取\n",
    "\n",
    "\n",
    "目标：借助句法分析工具，和实体识别的结果，以及文本特征，基于训练数据抽取关系，并存储进图数据库。\n",
    "\n",
    "本次要求抽取股权交易关系，关系为无向边，不要求判断投资方和被投资方，只要求得到双方是否存在交易关系。\n",
    "\n",
    "模板建立可以使用“正则表达式”、“实体间距离”、“实体上下文”、“依存句法”等。\n",
    "\n",
    "答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
    "- info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
    "- info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
    "\n",
    "示例：\n",
    "- info_extract_entity.csv\n",
    "\n",
    "| 实体编号 | 实体名 |\n",
    "| ------ | ------ |\n",
    "| 1001 | 小王 |\n",
    "| 1002 | A化工厂 |\n",
    "\n",
    "- info_extract_submit.csv\n",
    "\n",
    "| 实体1 | 实体2 |\n",
    "| ------ | ------ |\n",
    "| 1001 | 1003 |\n",
    "| 1002 | 1001 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习3：提取文本tf-idf特征\n",
    "\n",
    "去除停用词，并转换成tfidf向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(850, 4622)\n",
      "(419, 4622)\n"
     ]
    }
   ],
   "source": [
    "# code\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer  \n",
    "from pyltp import Segmentor\n",
    "\n",
    "# 实体符号加入分词词典\n",
    "with open('../data/user_dict.txt', 'w', encoding='utf-8') as fw:\n",
    "    for v in ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']:\n",
    "        fw.write( v + '号企业 ni\\n')\n",
    "\n",
    "# 初始化实例\n",
    "segmentor = Segmentor()  \n",
    "# 加载模型，加载自定义词典\n",
    "segmentor.load_with_lexicon('../data/ltp-models/ltp_data_v3.4.0/cws.model', '../data/user_dict.txt')  \n",
    "\n",
    "# 加载停用词\n",
    "fr = open(r'../data/dict/stopwords.txt', 'r', encoding='utf-8')   \n",
    "stop_word = fr.readlines()\n",
    "stop_word = [re.sub(r'(\\r|\\n)*','', word) for word in stop_word]\n",
    "\n",
    "# 分词\n",
    "# f = lambda x: ' '.join([word for word in segmentor.segment(x) if (word not in stop_word) and (not re.findall(r'ner_\\d\\d\\d\\d_', word))])\n",
    "\n",
    "def filtered_segment(s):\n",
    "    \"\"\"\n",
    "    为了避免ner_1089_这样的字符被分开，而专门写的一个分词函数\n",
    "    \"\"\"\n",
    "    tmp_ner_dict = {}\n",
    "    num_lst = ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']\n",
    "\n",
    "    # 将公司代码替换为特殊称谓，保证分词词性正确\n",
    "    for i, ner in enumerate(list(set(re.findall(r'(ner_\\d\\d\\d\\d_)', s)))):\n",
    "        try:\n",
    "            tmp_ner_dict[num_lst[i]+'号企业'] = ner\n",
    "        except IndexError:\n",
    "            # TODO：定义错误情况的输出\n",
    "            # TODO ...\n",
    "            print('替换出错!')\n",
    "            \n",
    "        s = s.replace(ner, num_lst[i]+'号企业')\n",
    "    words = segmentor.segment(s)\n",
    "    result_list = [tmp_ner_dict.get(word, word) for word in words if word not in stop_word]\n",
    "    result_segment = ' '.join(result_list)\n",
    "    return result_segment\n",
    "\n",
    "# corpus = X['ner'].map(f).tolist()\n",
    "corpus_train = X_train_sentence['ner'].map(filtered_segment).tolist()\n",
    "corpus_test = X_test_sentence['ner'].map(filtered_segment).tolist()\n",
    "\n",
    "# TODO：提取tfidf特征\n",
    "# TODO ...\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(corpus_train)\n",
    "X_test = vectorizer.transform(corpus_test)\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习4：提取句法特征\n",
    "除了词语层面的句向量特征，我们还可以从句法入手，提取一些句法分析的特征。\n",
    "\n",
    "参考特征：\n",
    "\n",
    "1、企业实体间距离\n",
    "\n",
    "2、企业实体间句法距离\n",
    "\n",
    "3、企业实体分别和关键触发词的距离\n",
    "\n",
    "4、实体的依存关系类别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/850 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 2.0, 2, 2, -1.0, -1, -1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 850/850 [17:12<00:00,  1.21s/it]\n",
      "100%|██████████| 419/419 [08:22<00:00,  1.20s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(850, 4652)\n",
      "(419, 4652)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "from pyltp import Parser\n",
    "from pyltp import Segmentor\n",
    "from pyltp import Postagger\n",
    "from itertools import combinations\n",
    "from scipy import sparse\n",
    "from tqdm import tqdm\n",
    "import networkx as nx\n",
    "import pylab\n",
    "import re\n",
    "\n",
    "postagger = Postagger() # 初始化实例\n",
    "postagger.load_with_lexicon('../data/ltp-models/ltp_data_v3.4.0/pos.model', '../data/user_dict.txt')  # 加载模型\n",
    "segmentor = Segmentor()  # 初始化实例\n",
    "segmentor.load_with_lexicon('../data/ltp-models/ltp_data_v3.4.0/cws.model', '../data/user_dict.txt')  # 加载模型\n",
    "\n",
    "\n",
    "def shortest_path(arcs_ret, source, target):\n",
    "    \"\"\"\n",
    "    求出两个词最短依存句法路径，不存在路径返回-1\n",
    "    arcs_ret：句法分析结果表格\n",
    "    source：实体1\n",
    "    target：实体2\n",
    "    \"\"\"\n",
    "    G=nx.DiGraph()\n",
    "    # 为这个网络添加节点...\n",
    "    for i in list(arcs_ret.index):\n",
    "        G.add_node(i)\n",
    "        # TODO：在网络中添加带权中的边...（注意，我们需要的是无向边）\n",
    "        # TODO ...\n",
    "        G.add_edge(arcs_ret.loc[i, 2], i)\n",
    "    \n",
    "    G.to_undirected()  # 转换成无向图\n",
    "\n",
    "    try:\n",
    "        # TODO：利用nx包中shortest_path_length方法实现最短距离提取\n",
    "        # TODO ...\n",
    "        distance = nx.shortest_path_length(G, source=source, target=target)\n",
    "        return distance\n",
    "    except:\n",
    "        return -1\n",
    "\n",
    "\n",
    "def parse(s):\n",
    "    \"\"\"\n",
    "    对语句进行句法分析，并返回句法结果\n",
    "    \"\"\"\n",
    "    tmp_ner_dict = {}\n",
    "    num_lst = ['一', '二', '三', '四', '五', '六', '七', '八', '九', '十']\n",
    "\n",
    "    # 将公司代码替换为特殊称谓，保证分词词性正确\n",
    "    for i, ner in enumerate(list(set(re.findall(r'(ner_\\d\\d\\d\\d_)', s)))):\n",
    "        try:\n",
    "            tmp_ner_dict[num_lst[i]+'号企业'] = ner\n",
    "        except IndexError:\n",
    "            # TODO：定义错误情况的输出\n",
    "            # TODO ...\n",
    "            print('替换出错!')\n",
    "            \n",
    "        s = s.replace(ner, num_lst[i]+'号企业')\n",
    "    words = segmentor.segment(s)\n",
    "    tags = postagger.postag(words)\n",
    "    parser = Parser() # 初始化实例\n",
    "    parser.load('../data/ltp-models/ltp_data_v3.4.0/parser.model')  # 加载模型\n",
    "    arcs = parser.parse(words, tags)  # 句法分析\n",
    "    arcs_lst = list(map(list, zip(*[[arc.head, arc.relation] for arc in arcs])))\n",
    "    \n",
    "    # 句法分析结果输出\n",
    "    parse_result = pd.DataFrame([[a,b,c,d] for a,b,c,d in zip(list(words), list(tags), arcs_lst[0], arcs_lst[1])], \n",
    "                                index = range(1,len(words)+1))\n",
    "    parser.release()  # 释放模型\n",
    "    \n",
    "#     print(parse_result)\n",
    "    # TODO：提取企业实体依存句法类型\n",
    "    # TODO ...\n",
    "\n",
    "    # 投资关系关键词\n",
    "    key_words = [\"收购\",\"竞拍\",\"转让\",\"扩张\",\"并购\",\"注资\",\"整合\",\"并入\",\"竞购\",\"竞买\",\"支付\",\"收购价\",\"收购价格\",\"承购\",\"购得\",\"购进\", \n",
    "                 \"购入\",\"买进\",\"买入\",\"赎买\",\"购销\",\"议购\",\"函购\",\"函售\",\"抛售\",\"售卖\",\"销售\",\"转售\"]\n",
    "    # TODO：*提取关键词和对应句法关系提取特征\n",
    "    # TODO ...\n",
    "    # 初始化企业实体的索引列表\n",
    "    ner_index_list = []\n",
    "    # 初始化关键词的索引列表\n",
    "    keyword_index_list = []\n",
    "    \n",
    "    # 遍历句法分析表格的索引\n",
    "    for idx in parse_result.index:\n",
    "        if parse_result.loc[idx, 0].endswith('号企业'):\n",
    "            ner_index_list.append(idx)\n",
    "        \n",
    "        if parse_result.loc[idx, 0] in key_words:\n",
    "            keyword_index_list.append(idx)\n",
    "    \n",
    "    # 1)在句子中的关键词数量\n",
    "    parse_feature1 = len(keyword_index_list)\n",
    "    \n",
    "    # 2)若关键词存在\n",
    "    # # 初始化判断与关键词有直接关系的'X号企业'句法类型为'S..'的数量\n",
    "    parse_feature2 = 0\n",
    "    # # 初始化判断与关键词有直接关系的'X号企业'句法类型为'.OB'的数量\n",
    "    parse_feature3 = 0\n",
    "    \n",
    "    # 遍历出现在句子中的关键词索引\n",
    "    for i in keyword_index_list:\n",
    "        # 遍历出现在句子中的实体索引\n",
    "        for j in ner_index_list:\n",
    "            # 如果实体句法类型以S开头(主语)或者OB结尾(宾语)\n",
    "            if parse_result.loc[j, 3].startswith('S') or parse_result.loc[j, 3].endswith('OB'):\n",
    "                # 若关键词对应句法关联索引为实体索引，则parse_feature2数量+1\n",
    "                if parse_result.loc[i, 2]==j:\n",
    "                    parse_feature2 += 1\n",
    "                # 若实体索引对应句法关联索引为关键词索引，则parse_feature3数量+1\n",
    "                if parse_result.loc[j, 2]==i:\n",
    "                    parse_feature3 += 1\n",
    "\n",
    "    # 3)实体与关键词之间距离的平均值，最大值和最小值\n",
    "    ner_keyword_pair_list = [(ner_index, keyword_index) for ner_index in ner_index_list for keyword_index in keyword_index_list]\n",
    "    ner_keyword_distance_list = [abs(pair[0]-pair[1]) for pair in ner_keyword_pair_list]\n",
    "\n",
    "    parse_feature4 = np.mean(ner_keyword_distance_list) if ner_keyword_distance_list else 0\n",
    "    parse_feature5 = max(ner_keyword_distance_list) if ner_keyword_distance_list else 0\n",
    "    parse_feature6 = min(ner_keyword_distance_list) if ner_keyword_distance_list else 0\n",
    "\n",
    "    # 4)实体与关键词之间句法距离的平均值，最大值和最小值\n",
    "    ner_keyword_parse_distance_list = [shortest_path(parse_result, pair[0], pair[1]) for pair in ner_keyword_pair_list]\n",
    "\n",
    "    parse_feature7 = np.mean(ner_keyword_parse_distance_list) if ner_keyword_parse_distance_list else 0\n",
    "    parse_feature8 = max(ner_keyword_parse_distance_list) if ner_keyword_parse_distance_list else 0\n",
    "    parse_feature9 = min(ner_keyword_parse_distance_list) if ner_keyword_parse_distance_list else 0\n",
    "    \n",
    "    # 5)实体之间距离的平均值，最大值和最小值\n",
    "    ner_pair_list = list(combinations(ner_index_list, 2))\n",
    "    ner_distance_list = [abs(pair[0]-pair[1]) for pair in ner_pair_list]\n",
    "\n",
    "    parse_feature10 = np.mean(ner_distance_list) if ner_distance_list else 0\n",
    "    parse_feature11 = max(ner_distance_list) if ner_distance_list else 0\n",
    "    parse_feature12 = min(ner_distance_list) if ner_distance_list else 0\n",
    "\n",
    "    # 6)实体之间句法距离的平均值，最大值和最小值\n",
    "    ner_parse_distance_list = [shortest_path(parse_result, pair[0], pair[1]) for pair in ner_pair_list]\n",
    "\n",
    "    parse_feature13 = np.mean(ner_parse_distance_list) if ner_parse_distance_list else 0\n",
    "    parse_feature14 = max(ner_parse_distance_list) if ner_parse_distance_list else 0\n",
    "    parse_feature15 = min(ner_parse_distance_list) if ner_parse_distance_list else 0\n",
    "    \n",
    "    return [parse_feature1, parse_feature2, parse_feature3, \n",
    "            parse_feature4, parse_feature5, parse_feature6,\n",
    "            parse_feature7, parse_feature8, parse_feature9,\n",
    "            parse_feature10, parse_feature11, parse_feature12,\n",
    "            parse_feature13, parse_feature14, parse_feature15]\n",
    "    \n",
    "\n",
    "#         0   1   2    3\n",
    "# 1   2015年  nt   3  ATT\n",
    "# 2      1月  nt   3  ATT\n",
    "# 3     26日  nt   9  ADV\n",
    "# 4       ，  wp   3   WP\n",
    "# 5    二号企业  ni   9  SBV\n",
    "# 6       与   p   9  ADV\n",
    "# 7    一号企业  ni   8  ATT\n",
    "# 8      先生   n   6  POB\n",
    "# 9      签署   v   0  HED\n",
    "# 10      了   u   9  RAD\n",
    "# print(parse('2016年 ner_1614_ 有限公司 瑞峰 张家港 光伏 科技 有限公司 支付 设备 款 人民币 4 515 770.00 元'))\n",
    "# print(parse('2016年 ner_1614_ 有限公司瑞峰张家港光伏科技有限公司支付设备款人民币4515770.00元'))\n",
    "\n",
    "# print(parse('资产 出售 资产 购买 方案 简要 介绍 资产 出售 公司 拟 控股 股东 ner_1625_ 出售 ner_1463_ 100% 股权'))\n",
    "print(parse('资产出售资产购买方案简要介绍资产出售公司拟控股股东 ner_1625_ 出售 ner_1463_ 100%股权'))\n",
    "\n",
    "# print(parse('集团 ner_1089_ 股份 有限公司 持有 ner_1616_ 股份 有限公司 股票 本期 变动 系 买卖 一揽子 沪 深 300 指数 成份股 致'))\n",
    "# print(parse('本集团及 ner_1089_ 股份有限公司持有 ner_1616_ 大股份有限公司股票的本期变动系买卖一揽子沪深300指数成份股所致'))\n",
    "\n",
    "\n",
    "# TODO：汇总上述函数汇总句法分析特征与TFIDF特征\n",
    "# TODO ...\n",
    "def get_extra_feature(corpus):\n",
    "    \"\"\"\n",
    "    获取语料数据的额外特征\n",
    "    \"\"\"\n",
    "    # 初始化存放所有语料数据额外特征的列表\n",
    "    extra_features_list = []\n",
    "    # 遍历语料，计算各条数据对应的额外特征，并添加到列表\n",
    "    try:\n",
    "        with tqdm(corpus) as t:\n",
    "            for i in t:\n",
    "                current_extra_features = parse(i)\n",
    "                extra_features_list.append(current_extra_features)\n",
    "        return np.array(extra_features_list)\n",
    "    except KeyboardInterrupt:\n",
    "        t.close()\n",
    "        raise\n",
    "\n",
    "\n",
    "# 对训练解和测试集分别拼接tf-idf特征与额外特征\n",
    "# # 这里直接用corpus(主要是为了去除了停用词，当然为了提取tf-idf弄成了空格分词形式)的数据进行提取额外特征\n",
    "# # (注：空格分词并会不影响提取的额外特征)\n",
    "X_train_extra_features = get_extra_feature(corpus_train)\n",
    "X_test_extra_features = get_extra_feature(corpus_test)\n",
    "\n",
    "X_train = sparse.hstack((X_train, sparse.csr_matrix(X_train_extra_features)), format='csr')\n",
    "X_test = sparse.hstack((X_test, sparse.csr_matrix(X_test_extra_features)), format='csr')\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习5：建立分类器\n",
    "\n",
    "利用已经提取好的tfidf特征以及parse特征，建立分类器进行分类任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for precision---------\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_search.py:814: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 3}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.000 (+/-0.000) for {'C': 0.001}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.201 (+/-0.802) for {'C': 0.01}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.583 (+/-0.683) for {'C': 0.1}\n",
      "0.917 (+/-0.211) for {'C': 0.3}\n",
      "0.972 (+/-0.114) for {'C': 1}\n",
      "0.989 (+/-0.044) for {'C': 3}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       722\n",
      "           1       1.00      0.95      0.97       128\n",
      "\n",
      "    accuracy                           0.99       850\n",
      "   macro avg       1.00      0.97      0.98       850\n",
      "weighted avg       0.99      0.99      0.99       850\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for recall---------\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_search.py:814: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 3}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.000 (+/-0.000) for {'C': 0.001}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.008 (+/-0.031) for {'C': 0.01}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.039 (+/-0.002) for {'C': 0.1}\n",
      "0.219 (+/-0.133) for {'C': 0.3}\n",
      "0.524 (+/-0.211) for {'C': 1}\n",
      "0.711 (+/-0.240) for {'C': 3}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       722\n",
      "           1       1.00      0.95      0.97       128\n",
      "\n",
      "    accuracy                           0.99       850\n",
      "   macro avg       1.00      0.97      0.98       850\n",
      "weighted avg       0.99      0.99      0.99       850\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for f1---------\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 3}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.000 (+/-0.000) for {'C': 0.001}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.015 (+/-0.059) for {'C': 0.01}\n",
      "0.000 (+/-0.000) for {'C': 0.003}\n",
      "0.072 (+/-0.007) for {'C': 0.1}\n",
      "0.348 (+/-0.181) for {'C': 0.3}\n",
      "0.674 (+/-0.197) for {'C': 1}\n",
      "0.822 (+/-0.167) for {'C': 3}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       722\n",
      "           1       1.00      0.95      0.97       128\n",
      "\n",
      "    accuracy                           0.99       850\n",
      "   macro avg       1.00      0.97      0.98       850\n",
      "weighted avg       0.99      0.99      0.99       850\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_search.py:814: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# 建立分类器进行分类\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# TODO：定义需要遍历的参数\n",
    "tuned_parameters ={\"C\": [0.001, 0.003, 0.01, 0.003, 0.1, 0.3, 1, 3]}\n",
    "scores = ['precision', 'recall', \"f1\"]\n",
    "\n",
    "# TODO：选择模型\n",
    "lr = LogisticRegression()\n",
    "\n",
    "# TODO：利用GridSearchCV搜索最佳参数\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s---------\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = GridSearchCV(LogisticRegression(),\n",
    "                       tuned_parameters,\n",
    "                       cv=5,\n",
    "                       scoring='%s' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_pred_train = clf.predict(X_train)\n",
    "    print(classification_report(y_train, y_pred_train))\n",
    "    print()\n",
    "\n",
    "# TODO：对Test_data进行分类\n",
    "best_model = clf.best_estimator_\n",
    "y_pred_test = best_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO：保存结果\n",
    "# 答案提交在submit目录中，命名为info_extract_submit.csv和info_extract_entity.csv。\n",
    "# info_extract_entity.csv格式为：第一列是实体编号，第二列是实体名（实体统一的多个实体名用“|”分隔）\n",
    "# info_extract_submit.csv格式为：第一列是关系中实体1的编号，第二列为关系中实体2的编号。\n",
    "\n",
    "# 实体编号对应表格\n",
    "# # 初始化结果列表\n",
    "info_extract_entity_list = []\n",
    "\n",
    "# # 遍历ner_dict_new, ner_dict_reverse_new进行生成\n",
    "for k in ner_dict_new:\n",
    "    entity_id = ner_dict_new[k]\n",
    "    entity_names = '|'.join(ner_dict_reverse_new[k])\n",
    "    info_extract_entity_list.append([entity_id, entity_names])\n",
    "\n",
    "# # 生成表格\n",
    "info_extract_entity_df = pd.DataFrame(info_extract_entity_list, columns=['实体编号', '实体名'])\n",
    "info_extract_entity_df.to_csv('../submit/info_extract_entity.csv', index=False)\n",
    "\n",
    "# 实体联系对应表格\n",
    "# # 初始化最终的结果列表\n",
    "info_extract_submit_list = []\n",
    "# # 先利用预测的标签，提取出相应的句子\n",
    "predict_positive_examples = pd.Series(corpus_test)[pd.Series(y_pred_test).astype(bool)]\n",
    "# # 遍历各个句子\n",
    "for example in predict_positive_examples:\n",
    "    # # 正则提取该句子所有的实体符号，并去重\n",
    "    current_entity_list = re.findall('ner_(\\d\\d\\d\\d)_', example)\n",
    "    current_entity_set = set(current_entity_list)\n",
    "    # 当前句子的实体组合关系\n",
    "    current_entity_pair_list = list(combinations(current_entity_set, 2))\n",
    "    # 将当前句子的实体组合关系扩充到结果列表中\n",
    "    info_extract_submit_list.extend(current_entity_pair_list)\n",
    "\n",
    "# # 生成表格\n",
    "info_extract_submit_df = pd.DataFrame(info_extract_submit_list, columns=['实体1', '实体2'])\n",
    "info_extract_submit_df.to_csv('../submit/info_extract_submit.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习6：操作图数据库\n",
    "对关系最好的描述就是用图，那这里就需要使用图数据库，目前最常用的图数据库是noe4j，通过cypher语句就可以操作图数据库的增删改查。可以参考“https://cuiqingcai.com/4778.html”。\n",
    "\n",
    "本次作业我们使用neo4j作为图数据库，neo4j需要java环境，请先配置好环境。\n",
    "\n",
    "将我们提出的实体关系插入图数据库，并查询某节点的3层投资关系，即三个节点组成的路径（如果有的话）。如果无法找到3层投资关系，请查询出任意指定节点的投资路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'py2neo'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-43f325b259db>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpy2neo\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mNode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mRelationship\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mGraph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m graph = Graph(\n\u001b[1;32m      4\u001b[0m     \u001b[0;34m\"http://localhost:7474\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0musername\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"neo4j\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'py2neo'"
     ]
    }
   ],
   "source": [
    "from py2neo import Node, Relationship, Graph\n",
    "\n",
    "graph = Graph(\n",
    "    \"http://localhost:7474\", \n",
    "    username=\"neo4j\", \n",
    "    password=\"person\"\n",
    ")\n",
    "\n",
    "for v in info_extract_submit_list:\n",
    "    a = Node('Company', name=v[0])\n",
    "    b = Node('Company', name=v[1])\n",
    "    \n",
    "    # 本次不区分投资方和被投资方，无向图\n",
    "    r = Relationship(a, 'INVEST', b)\n",
    "    s = a | b | r\n",
    "    graph.create(s)\n",
    "    r = Relationship(b, 'INVEST', a)\n",
    "    s = a | b | r\n",
    "    graph.create(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# TODO：查询某节点的3层投资关系\n",
    "graph.run(\"match data=(na:Company{name:'1018'})-[*1..3]->(nb:Company) return data\").data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤4：实体消歧\n",
    "解决了实体识别和关系的提取，我们已经完成了一大截，但是我们提取的实体究竟对应知识库中哪个实体呢？下图中，光是“苹果”就对应了13个同名实体。\n",
    "<img src=\"../image/baike2.png\" width=340 heigth=480/>\n",
    "\n",
    "在这个问题上，实体消歧旨在解决文本中广泛存在的名称歧义问题，将句中识别的实体与知识库中实体进行匹配，解决实体歧义问题。\n",
    "\n",
    "\n",
    "### 练习7：\n",
    "匹配test_data.csv中前25条样本中的人物实体对应的百度百科URL（此部分样本中所有人名均可在百度百科中链接到）。\n",
    "\n",
    "利用scrapy、beautifulsoup、request等python包对百度百科进行爬虫，判断是否具有一词多义的情况，如果有的话，选择最佳实体进行匹配。\n",
    "\n",
    "使用URL为‘https://baike.baidu.com/item/’+人名 可以访问百度百科该人名的词条，此处需要根据爬取到的网页识别该词条是否对应多个实体，如下图：\n",
    "<img src=\"../image/baike1.png\" width=440 heigth=480/>\n",
    "如果该词条有对应多个实体，请返回正确匹配的实体URL，例如该示例网页中的‘https://baike.baidu.com/item/陆永/20793929’。\n",
    "\n",
    "- 提交文件：entity_disambiguation_submit.csv\n",
    "- 提交格式：第一列为实体id（与info_extract_submit.csv中id保持一致），第二列为对应URL。\n",
    "- 示例：\n",
    "\n",
    "| 实体编号 | URL |\n",
    "| ------ | ------ |\n",
    "| 1001 | https://baike.baidu.com/item/陆永/20793929 |\n",
    "| 1002 | https://baike.baidu.com/item/王芳/567232 |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 25/25 [00:12<00:00,  2.07it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>实体编号</th>\n",
       "      <th>URL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%9D%8E%E4%BA%9...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1003</td>\n",
       "      <td>https://baike.baidu.com/item/%E4%BE%AF%E6%AF%8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1005</td>\n",
       "      <td>https://baike.baidu.com/item/%E5%BC%A0%E5%BE%B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1007</td>\n",
       "      <td>https://baike.baidu.com/item/%E8%82%96%E6%96%8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1009</td>\n",
       "      <td>https://baike.baidu.com/item/熊海涛</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1011</td>\n",
       "      <td>https://baike.baidu.com/item/%E5%AE%8B%E7%90%B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1014</td>\n",
       "      <td>https://baike.baidu.com/item/王友林</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1016</td>\n",
       "      <td>https://baike.baidu.com/item/彭聪</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1017</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1019</td>\n",
       "      <td>https://baike.baidu.com/item/%E9%A2%9C%E5%86%9...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1021</td>\n",
       "      <td>https://baike.baidu.com/item/宋睿</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1023</td>\n",
       "      <td>N/A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1025</td>\n",
       "      <td>https://baike.baidu.com/item/邓冠华</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1028</td>\n",
       "      <td>https://baike.baidu.com/item/孙锋峰</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1001</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%9D%8E%E4%BA%9...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1017</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1029</td>\n",
       "      <td>https://baike.baidu.com/item/林奇</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1031</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%B1%9F%E6%96%8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1014</td>\n",
       "      <td>https://baike.baidu.com/item/%E7%8E%8B%E5%8F%8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1033</td>\n",
       "      <td>https://baike.baidu.com/item/%E6%9E%97%E6%B5%B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1036</td>\n",
       "      <td>https://baike.baidu.com/item/郭为</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1038</td>\n",
       "      <td>https://baike.baidu.com/item/%E5%90%B4%E5%AE%8...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1041</td>\n",
       "      <td>https://baike.baidu.com/item/%E7%8E%8B%E5%88%A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1043</td>\n",
       "      <td>https://baike.baidu.com/item/周旭辉</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1049</td>\n",
       "      <td>https://baike.baidu.com/item/%E5%90%B4%E8%89%B...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    实体编号                                                URL\n",
       "0   1001  https://baike.baidu.com/item/%E6%9D%8E%E4%BA%9...\n",
       "1   1003  https://baike.baidu.com/item/%E4%BE%AF%E6%AF%8...\n",
       "2   1005  https://baike.baidu.com/item/%E5%BC%A0%E5%BE%B...\n",
       "3   1007  https://baike.baidu.com/item/%E8%82%96%E6%96%8...\n",
       "4   1009                   https://baike.baidu.com/item/熊海涛\n",
       "5   1011  https://baike.baidu.com/item/%E5%AE%8B%E7%90%B...\n",
       "6   1014                   https://baike.baidu.com/item/王友林\n",
       "7   1016                    https://baike.baidu.com/item/彭聪\n",
       "8   1017  https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9...\n",
       "9   1019  https://baike.baidu.com/item/%E9%A2%9C%E5%86%9...\n",
       "10  1021                    https://baike.baidu.com/item/宋睿\n",
       "11  1023                                                N/A\n",
       "12  1025                   https://baike.baidu.com/item/邓冠华\n",
       "13  1028                   https://baike.baidu.com/item/孙锋峰\n",
       "14  1001  https://baike.baidu.com/item/%E6%9D%8E%E4%BA%9...\n",
       "15  1017  https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9...\n",
       "16  1029                    https://baike.baidu.com/item/林奇\n",
       "17  1031  https://baike.baidu.com/item/%E6%B1%9F%E6%96%8...\n",
       "18  1014  https://baike.baidu.com/item/%E7%8E%8B%E5%8F%8...\n",
       "19  1033  https://baike.baidu.com/item/%E6%9E%97%E6%B5%B...\n",
       "20  1036                    https://baike.baidu.com/item/郭为\n",
       "21  1038  https://baike.baidu.com/item/%E5%90%B4%E5%AE%8...\n",
       "22  1041  https://baike.baidu.com/item/%E7%8E%8B%E5%88%A...\n",
       "23  1043                   https://baike.baidu.com/item/周旭辉\n",
       "24  1049  https://baike.baidu.com/item/%E5%90%B4%E8%89%B..."
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jieba\n",
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "from copy import copy\n",
    "import fool\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# 找出test_data.csv中前25条样本所有的人物名称，以及人物所在文档的上下文内容\n",
    "test_data = pd.read_csv('../data/info_extract/test_data.csv', encoding = 'gb2312', header=0)\n",
    "\n",
    "# 存储人物以及上下文信息（key为人物ID，value为人物名称、人物上下文内容）\n",
    "person_name = {}\n",
    "\n",
    "# 观察上下文的窗口大小\n",
    "window = 10  \n",
    "\n",
    "# 遍历前25条样本\n",
    "for i in range(25):\n",
    "    sentence = copy(test_data.iloc[i, 1])\n",
    "    words, ners = fool.analysis(sentence)\n",
    "    ners[0].sort(key=lambda x:x[0], reverse=True)\n",
    "    for start, end, ner_type, ner_name in ners[0]:\n",
    "        if ner_type=='person':\n",
    "            # TODO：提取实体的上下文\n",
    "            context = sentence[max(0, start-window):start] + sentence[end-1:min(len(sentence), end-1+window)]\n",
    "            person_name[i] = (ner_name, context)\n",
    "\n",
    "# print(person_name)\n",
    "\n",
    "def calculate_sentence_similarity(string1, string2):\n",
    "    \"\"\"\n",
    "    计算两个句子的相似度(基于unigram, bigram, trigram的重复程度)\n",
    "    \"\"\"\n",
    "    # unigram\n",
    "    unigram_set1 = set(string1)\n",
    "    unigram_set2 = set(string2)\n",
    "    unigram_intersection = unigram_set1 & unigram_set2\n",
    "    unigram_union = unigram_set1 | unigram_set2\n",
    "    unigram_similarity_score = len(unigram_intersection) / len(unigram_union)\n",
    "    \n",
    "    # bigram\n",
    "    bigram_set1 = {string1[idx:idx+2] for idx in range(len(string1)-1)}\n",
    "    bigram_set2 = {string2[idx:idx+2] for idx in range(len(string2)-1)}\n",
    "    bigram_intersection = bigram_set1 & bigram_set2\n",
    "    bigram_union = bigram_set1 | bigram_set2\n",
    "    bigram_similarity_score = len(bigram_intersection) / len(bigram_union)\n",
    "    \n",
    "    # trigram\n",
    "    trigram_set1 = {string1[idx:idx+3] for idx in range(len(string1)-2)}\n",
    "    trigram_set2 = {string2[idx:idx+3] for idx in range(len(string2)-2)}\n",
    "    trigram_intersection = trigram_set1 & trigram_set2\n",
    "    trigram_union = trigram_set1 | trigram_set2\n",
    "    trigram_similarity_score = len(trigram_intersection) / len(trigram_union)\n",
    "    \n",
    "    return unigram_similarity_score + bigram_similarity_score + trigram_similarity_score\n",
    "\n",
    "# 利用爬虫得到每个人物名称对应的URL\n",
    "# TODO：找到每个人物实体的词条内容。\n",
    "# # 初始化结果列表\n",
    "result_data = []\n",
    "# # 循环爬取\n",
    "\n",
    "try:\n",
    "    with tqdm(person_name.items()) as tq:\n",
    "        for k,v in tq:\n",
    "            # 拼接生成serach_url\n",
    "            current_name = v[0]\n",
    "            search_url = 'https://baike.baidu.com/item/' + current_name\n",
    "            # 获取当前人名对应实体编号\n",
    "            current_name_id = ner_dict_new[current_name]\n",
    "        #     print(current_name_id, current_name)\n",
    "            # 请求数据\n",
    "            headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}\n",
    "            web_data = requests.get(search_url, headers=headers)\n",
    "            web_data.encoding = 'utf-8'\n",
    "            # 解析数据\n",
    "            soup = BeautifulSoup(web_data.text, 'lxml')\n",
    "            # # 查找是否有百科搜到的title\n",
    "            check_title = soup.select('dd.lemmaWgt-lemmaTitle-title > h1')\n",
    "\n",
    "            if check_title:\n",
    "                # TODO：将样本中人物上下文与爬取词条结果进行对比，选择最接近的词条\n",
    "                current_context = v[1]\n",
    "                # 获取对应节点列表\n",
    "                entries_node = soup.select('li.item *')\n",
    "                # 获取词条列表\n",
    "                entries_list = [i.text for i in entries_node]\n",
    "        #         print(entries_list)\n",
    "                # 获取词条链接列表\n",
    "                entries_url_list = [search_url if i.get('class') else 'https://baike.baidu.com'+i.get('href') for i in entries_node]\n",
    "                # 判断是否获取到了词条\n",
    "                if entries_list:\n",
    "                    # # 计算所有词条与当前人物名称的相似度\n",
    "                    entries_similarity_list = [calculate_sentence_similarity(current_context, entry) for entry in entries_list]\n",
    "#                     print(entries_similarity_list)\n",
    "                    # # 获取最大相似度所在的索引\n",
    "                    max_similarity_index = np.argmax(entries_similarity_list)\n",
    "                    # # 获取最大相似度对应的词条链接\n",
    "                    max_similarity_entry_url = entries_url_list[max_similarity_index]\n",
    "                    # # 将该词条链接添加到结果列表\n",
    "                    result_data.append((current_name_id, max_similarity_entry_url))\n",
    "                else:\n",
    "                    result_data.append((current_name_id, search_url))\n",
    "            else:\n",
    "                result_data.append((current_name_id, 'N/A'))\n",
    "except KeyboardInterrupt:\n",
    "    tq.close()\n",
    "    raise\n",
    "                \n",
    "# print(result_data)\n",
    "\n",
    "# 输出结果\n",
    "result_df = pd.DataFrame(result_data, columns=['实体编号', 'URL'])\n",
    "result_df.to_csv('../submit/entity_disambiguation_submit.csv', index=False)\n",
    "result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}