added project 2 and project 5

c3304936 · 20200203141 · 450bd27a · c3304936 · c3304936 · c3304936
Commit c3304936 authored Aug 29, 2020 by 20200203141
26 changed files
--- a/Project2/code/.ipynb_checkpoints/knowledge_graph-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/knowledge_graph-checkpoint.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "对此project的想法：\n",
+    "\n",
+    "观察了此porject布置的时间点，是在“实体识别”、“句法分析”之后。应该适当温习算法原理的知识点。\n",
+    "\n",
+    "1、实体识别、依存句法等工具自己动手开发实现复杂度较高，挑选“句法结构分析”工具让学员开发，不是特别复杂，又温习了句法分析中的知识点，此处选择CYK算法让学员实现，同时又考察了动态规划的编写。（具体算法因为不知道课程内容，可以根据内容再做调整）\n",
+    "\n",
+    "2、经过挑选，选择“企业投资关系图谱”作为学员的任务，原因1：企业投资关系易于理解，模版比较好总结，适合没有训练样本的情景；原因2：企业的称谓较多而复杂，适合考察实体统一和消歧的知识点。\n",
+    "\n",
+    "3、对于实体识别和句法分析，我考虑使用stanfordnlp，但是python接口好像可配置性不强，主要就是让学员会调用，会利用调用结果。\n",
+    "对于实体消歧，我考虑使用上下文相似度进行衡量。\n",
+    "对于实体统一，我考虑考察一下学员在“企业多名称”派生上面的发散性思维。\n",
+    "由于不知道此project前几节课中涉及的相关知识点，所以先拍脑袋决定，老师如果对知识点有相关准备和资料可以给我看一下，或者听听老师的规划。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Project 1: 利用信息抽取技术搭建知识库"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "本项目的目的是结合命名实体识别、依存语法分析、实体消歧、实体统一对网站开放语料抓取的数据建立小型知识图谱。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Part1：开发句法结构分析工具"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1.1 开发工具\n",
+    "使用CYK算法，根据所提供的：非终结符集合、终结符集合、规则集，对10句句子计算句法结构。\n",
+    "\n",
+    "非终结符集合：N={S, NP, VP, PP, DT, VI, VT, NN, IN}\n",
+    "\n",
+    "终结符集合：{sleeps, saw, boy, girl, dog, telescope, the, with, in}\n",
+    "\n",
+    "规则集: R={\n",
+    "- (1) S-->NP VP 1.0\n",
+    "- (2) VP-->VI 0.3\n",
+    "- ......\n",
+    "\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1.2 计算算法复杂度\n",
+    "计算上一节开发的算法所对应的时间复杂度和空间复杂度。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Part2：在百度百科辅助下，建立“投资关系”知识图谱"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.1 实体识别\n",
+    "data目录中“baike.txt”文件记录了15个实体对应百度百科的url。\n",
+    "\n",
+    "借助开源实体识别工具并根据百度百科建立的已知实体对进行识别，对句中实体进行识别。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code\n",
+    "# 首先尝试利用开源工具分出实体\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code\n",
+    "# 在此基础上，将百度百科作为已知实体加入词典，对实体进行识别"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.2 实体消歧\n",
+    "将句中识别的实体与知识库中实体进行匹配，解决实体歧义问题。\n",
+    "可利用上下文本相似度进行识别。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code\n",
+    "# 将识别出的实体与知识库中实体进行匹配，解决识别出一个实体对应知识库中多个实体的问题。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.3 实体统一\n",
+    "对同一实体具有多个名称的情况进行统一，将多种称谓统一到一个实体上，并体现在实体的属性中（可以给实体建立“别称”属性）"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.4 关系抽取\n",
+    "借助句法分析工具，和实体识别的结果，以及正则表达式，设定模版抽取关系。从data目录中news.txt文件中的url对应的新闻提取关系并存储进图数据库。\n",
+    "\n",
+    "本次要求抽取投资关系，关系为有向边，由投资方指向被投资方。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# code\n",
+    "\n",
+    "# 最后提交文件为识别出的整个投资图谱，以及图谱中结点列表与属性。"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
--- a/Project2/code/.ipynb_checkpoints/knowledge_graph_20180107_ѧԱ-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/knowledge_graph_20180107_ѧԱ-checkpoint.ipynb
--- a/Project2/code/.ipynb_checkpoints/knowledge_graph_20181230-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/knowledge_graph_20181230-checkpoint.ipynb
--- a/Project2/code/.ipynb_checkpoints/knowledge_graph_20190331-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/knowledge_graph_20190331-checkpoint.ipynb
--- a/Project2/code/.ipynb_checkpoints/knowledge_graph_20190331_student-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/knowledge_graph_20190331_student-checkpoint.ipynb
--- a/Project2/code/.ipynb_checkpoints/starter_code-checkpoint.ipynb
+++ b/Project2/code/.ipynb_checkpoints/starter_code-checkpoint.ipynb
--- a/Project2/code/starter_code.ipynb
+++ b/Project2/code/starter_code.ipynb
--- a/Project2/data/dict/co_City_Dim.txt
+++ b/Project2/data/dict/co_City_Dim.txt
--- a/Project2/data/dict/co_Province_Dim.txt
+++ b/Project2/data/dict/co_Province_Dim.txt
+北京
+上海
+广东省
+深圳
+天津
+重庆
+江苏省
+浙江省
+四川省
+海南省
+福建省
+山东省
+江西省
+广西
+安徽省
+河北省
+河南省
+湖北省
+湖南省
+陕西省
+山西省
+黑龙江省
+辽宁省
+吉林省
+云南省
+贵州省
+甘肃省
+内蒙古
+宁夏
+西藏
+新疆
+青海省
+香港
+澳门
+台湾
+国外
+北京省
+上海省
+广东
+深圳省
+天津省
+重庆省
+江苏
+浙江
+四川
+海南
+福建
+山东
+江西
+广西省
+安徽
+河北
+河南
+湖北
+湖南
+陕西
+山西
+黑龙江
+辽宁
+吉林
+云南
+贵州
+甘肃
+内蒙古省
+宁夏省
+西藏省
+新疆省
+青海
+香港省
+澳门省
+台湾省
+国外省
--- a/Project2/data/dict/company_business_scope.txt
+++ b/Project2/data/dict/company_business_scope.txt
--- a/Project2/data/dict/company_suffix.txt
+++ b/Project2/data/dict/company_suffix.txt
+仅对非常通用的后缀进行删除
+ltd
+Ltd
+limited
+.Ltd
+Ltd.
+LTD
+LTD.
+.LTD
+Co.Ltd
+CO.LTD
+Co.LTD
+CO.Ltd
+co.ltd
+co.Ltd
+Co.
+Co
+CO.
+CO
+co.
+co
+下属
+业务
+公司
+营业部
+中心
+总部
+实验室
+研究所
+总公司
+分公司
+子公司
+设计院
+事务所
+集团
+控股
+责任
+附属
+有限
+股份
+分厂
+设计院有限公司
+国际集团
+控股集团
+附属公司
+集团公司
+责任公司
+股份公司
+贸易公司
+有限公司
--- a/Project2/data/dict/stopwords.txt
+++ b/Project2/data/dict/stopwords.txt
--- a/Project2/data/info_extract/test_data.csv
+++ b/Project2/data/info_extract/test_data.csv
--- a/Project2/data/info_extract/train_data.csv
+++ b/Project2/data/info_extract/train_data.csv
--- a/Project2/data/user_dict.txt
+++ b/Project2/data/user_dict.txt
+一号企业 ni
+二号企业 ni
+三号企业 ni
+四号企业 ni
+五号企业 ni
+六号企业 ni
+七号企业 ni
+八号企业 ni
+九号企业 ni
+十号企业 ni
--- a/Project2/image/analyse_table.jpg
+++ b/Project2/image/analyse_table.jpg
--- a/Project2/image/baike1.png
+++ b/Project2/image/baike1.png
--- a/Project2/image/baike2.png
+++ b/Project2/image/baike2.png
--- a/Project2/submit/entity_disambiguation_submit.csv
+++ b/Project2/submit/entity_disambiguation_submit.csv
+实体编号,URL
+1001,https://baike.baidu.com/item/%E6%9D%8E%E4%BA%91%E5%B3%B0/22102428#viewPageContent
+1003,https://baike.baidu.com/item/%E4%BE%AF%E6%AF%85/12795458#viewPageContent
+1005,https://baike.baidu.com/item/%E5%BC%A0%E5%BE%B7%E5%8D%8E/7002732#viewPageContent
+1007,https://baike.baidu.com/item/%E8%82%96%E6%96%87%E9%9D%A9/22791038#viewPageContent
+1009,https://baike.baidu.com/item/%E7%86%8A%E6%B5%B7%E6%B6%9B/20479589#viewPageContent
+1011,https://baike.baidu.com/item/%E5%AE%8B%E7%90%B3/16173836#viewPageContent
+1014,https://baike.baidu.com/item/王友林
+1016,https://baike.baidu.com/item/彭聪
+1017,https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9E/16542190#viewPageContent
+1019,https://baike.baidu.com/item/%E9%A2%9C%E5%86%9B/20469600#viewPageContent
+1021,https://baike.baidu.com/item/宋睿
+1023,N/A
+1025,https://baike.baidu.com/item/邓冠华
+1028,https://baike.baidu.com/item/孙锋峰
+1001,https://baike.baidu.com/item/%E6%9D%8E%E4%BA%91%E5%B3%B0/22102428#viewPageContent
+1017,https://baike.baidu.com/item/%E6%9B%B9%E9%A3%9E/16542190#viewPageContent
+1029,https://baike.baidu.com/item/林奇
+1031,https://baike.baidu.com/item/%E6%B1%9F%E6%96%8C/22653694#viewPageContent
+1014,https://baike.baidu.com/item/王友林
+1033,https://baike.baidu.com/item/%E6%9E%97%E6%B5%B7%E5%B3%B0/10781910#viewPageContent
+1036,https://baike.baidu.com/item/郭为
+1038,https://baike.baidu.com/item/%E5%90%B4%E5%AE%8F%E4%BA%AE/18593265#viewPageContent
+1041,https://baike.baidu.com/item/王利平
+1043,https://baike.baidu.com/item/周旭辉
+1049,https://baike.baidu.com/item/%E5%90%B4%E8%89%B3/20303517#viewPageContent
--- a/Project2/submit/info_extract_entity.csv
+++ b/Project2/submit/info_extract_entity.csv
--- a/Project2/submit/info_extract_submit.csv
+++ b/Project2/submit/info_extract_submit.csv
+实体1,实体2
+1007,1008
+1017,1018
+1020,1019
+1022,1021
+1040,1041
+1040,1042
+1041,1042
+1044,1043
+1044,1047
+1044,1046
+1044,1045
+1043,1047
+1043,1046
+1043,1045
+1047,1046
+1047,1045
+1046,1045
+1048,1052
+1048,1049
+1048,1050
+1048,1053
+1048,1051
+1052,1049
+1052,1050
+1052,1053
+1052,1051
+1049,1050
+1049,1053
+1049,1051
+1050,1053
+1050,1051
+1053,1051
+1055,1054
+1066,1067
+1093,1092
+1139,1140
+1149,1150
+1061,1060
+1228,1227
+1277,1276
+1336,1335
+1373,1372
+1410,1411
+1435,1434
+1456,1455
+1475,1474
+1503,1502
+1534,1533
+1547,1548
+1608,1609
--- a/project5/Readme.txt
+++ b/project5/Readme.txt
+project2_main.py python执行脚本（所有的代码位置）
+data_train.xlsx 意图分类训练数据
+stopword.txt 停用词
--- a/project5/data_train.xlsx
+++ b/project5/data_train.xlsx
--- a/project5/project2_main.py
+++ b/project5/project2_main.py
--- a/project5/stopword.txt
+++ b/project5/stopword.txt
+的
+到
+一下
+我
+，
+帮
+去
+了
+？
+吧
+。
+请
+么
--- a/project5/票务对话机器人项目说明.pdf
+++ b/project5/票务对话机器人项目说明.pdf