{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 智能问答系统（主文件）\n",
"\n",
"在这里我们来搭建一个轻量级智能问答系统，所需要的模块，包括：\n",
"- 文本预处理：这部分已经帮大家写好，只需要看看代码就可以了。\n",
"- 搭建意图识别分类器：这部分也给大家写好了，使用fastText来做的意图识别器\n",
"- 倒排表：这部分大家需要自己去创建，同时也需要考虑相似的单词（课程视频中讲过）\n",
"- 排序：基于倒排表返回的结果，我们再根据余弦相似度来计算query跟候选问题之间的相似度，最后返回相似度最高的问题的答案。这里，我们将使用BERT来表示句子的向量。 "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"KMP_DUPLICATE_LIB_OK\"]=\"TRUE\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\avaws\\anaconda3\\envs\\nlp_gensim\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"import pandas as pd\n",
"from tqdm import tqdm\n",
"import numpy as np\n",
"import pickle\n",
"import emoji\n",
"import re\n",
"import jieba\n",
"import torch\n",
"import fasttext\n",
"from sys import platform"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# 读取已经处理好的数据： 导入在preprocessor.ipynb中生成的data/question_answer_pares.pkl文件，并将其保存在变量QApares中\n",
"with open('./data/question_answer_pares.pkl','rb') as f:\n",
" QApares = pickle.load(f)\n",
"# 添加一列展示列表长度\n",
"QApares[\"num\"] = [len(QApares.question_after_preprocessing[_]) for _ in range(len(QApares.question_after_preprocessing.values))]\n",
"# 去除空列表，并重新index\n",
"QApares = QApares.loc[QApares.num > 0].reset_index(drop=True)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>question</th>\n",
" <th>answer</th>\n",
" <th>question_after_preprocessing</th>\n",
" <th>num</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>买二份有没有少点呀</td>\n",
" <td>亲亲真的不好意思我们已经是优惠价了呢小本生意请亲谅解</td>\n",
" <td>[买, 二份, 有没有, 少点]</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>那就等你们处理喽</td>\n",
" <td>好的亲退了</td>\n",
" <td>[处理]</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>那我不喜欢</td>\n",
" <td>颜色的话一般茶刀茶针和二合一的话都是红木檀和黑木檀哦</td>\n",
" <td>[喜欢]</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>不是免运费</td>\n",
" <td>本店茶具订单满99包邮除宁夏青海内蒙古海南新疆西藏满39包邮</td>\n",
" <td>[免, 运费]</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>好吃吗</td>\n",
" <td>好吃的</td>\n",
" <td>[好吃]</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86902</th>\n",
" <td>哪个比较快</td>\n",
" <td>一般都差不多的哦亲爱哒客官小店是从浙江嘉兴发货的哦一般发货后35天就能到您那边呢请您耐心等下哦</td>\n",
" <td>[比较, 快]</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86903</th>\n",
" <td>已经提交申请了谢谢</td>\n",
" <td>好的亲稍等</td>\n",
" <td>[已经, 提交, 申请, 谢谢]</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86904</th>\n",
" <td>就一些浮油你还想一张有一层油啊</td>\n",
" <td>明天我给主管看下吧</td>\n",
" <td>[想, 一张, 一层, 油]</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86905</th>\n",
" <td>他说丟了</td>\n",
" <td>好的</td>\n",
" <td>[说, 丟了]</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86906</th>\n",
" <td>辽宁营口</td>\n",
" <td>4袋包邮哦</td>\n",
" <td>[辽宁, 营口]</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>86907 rows × 4 columns</p>\n",
"</div>"
],
"text/plain": [
" question answer \\\n",
"0 买二份有没有少点呀 亲亲真的不好意思我们已经是优惠价了呢小本生意请亲谅解 \n",
"1 那就等你们处理喽 好的亲退了 \n",
"2 那我不喜欢 颜色的话一般茶刀茶针和二合一的话都是红木檀和黑木檀哦 \n",
"3 不是免运费 本店茶具订单满99包邮除宁夏青海内蒙古海南新疆西藏满39包邮 \n",
"4 好吃吗 好吃的 \n",
"... ... ... \n",
"86902 哪个比较快 一般都差不多的哦亲爱哒客官小店是从浙江嘉兴发货的哦一般发货后35天就能到您那边呢请您耐心等下哦 \n",
"86903 已经提交申请了谢谢 好的亲稍等 \n",
"86904 就一些浮油你还想一张有一层油啊 明天我给主管看下吧 \n",
"86905 他说丟了 好的 \n",
"86906 辽宁营口 4袋包邮哦 \n",
"\n",
" question_after_preprocessing num \n",
"0 [买, 二份, 有没有, 少点] 4 \n",
"1 [处理] 1 \n",
"2 [喜欢] 1 \n",
"3 [免, 运费] 2 \n",
"4 [好吃] 1 \n",
"... ... ... \n",
"86902 [比较, 快] 2 \n",
"86903 [已经, 提交, 申请, 谢谢] 4 \n",
"86904 [想, 一张, 一层, 油] 4 \n",
"86905 [说, 丟了] 2 \n",
"86906 [辽宁, 营口] 2 \n",
"\n",
"[86907 rows x 4 columns]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"QApares"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# 导入在Retrieve.ipynb中生成的data/retrieve/invertedList.pkl倒排表文件，并将其保存在变量invertedList中\n",
"with open('./data/retrieve/invertedList.pkl','rb') as f:\n",
" invertedList = pickle.load(f)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"#这一格的内容是从preprocessor.ipynb中粘贴而来，包含了数据预处理的几个关键函数\n",
"import pickle\n",
"import emoji\n",
"import re\n",
"import jieba\n",
"def clean(content):\n",
" content = emoji.demojize(content)\n",
" content = re.sub('<.*>','',content)\n",
" return content\n",
"#这一函数是用于对句子进行分词，在preprocessor.ipynb中由于数据是已经分好词的，所以我们并没有进行这一步骤，但是对于一个新的问句，这一步是必不可少的\n",
"def question_cut(content):\n",
" return list(jieba.cut(content))\n",
"def strip(wordList):\n",
" return [word.strip() for word in wordList if word.strip()!='']\n",
"with open(\"data/stopWord.json\",\"r\", encoding=\"utf-8\") as f:\n",
" stopWords = f.read().split(\"\\n\")\n",
"def rm_stop_word(wordList):\n",
" return [word for word in wordList if word not in stopWords]\n",
"\n",
"def get_retrieve_result(sentence):\n",
" '''\n",
" 输入一个句子sentence，根据倒排表进行快速检索，返回与该句子较相近的一些候选问题的index\n",
" 候选问题由包含该句子中任一单词或包含与该句子中任一单词意思相近的单词的问题索引组成\n",
" '''\n",
" sentence = clean(sentence)\n",
" sentence = question_cut(sentence)\n",
" sentence = strip(sentence)\n",
" sentence = rm_stop_word(sentence)\n",
" candidate = set()\n",
" for word in sentence:\n",
" if word in invertedList:\n",
" candidate = candidate | invertedList[word]\n",
" return candidate"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.\n"
]
}
],
"source": [
"# 加载训练好的fasttext模型用于意图识别\n",
"intention = fasttext.load_model('model/fasttext.ftz')\n",
"\n",
"def get_intention_result(sentence):\n",
" '''\n",
" 输入句子，返回意图识别结果\n",
" 入参：\n",
" sentence:输入的句子\n",
" 出参:\n",
" fasttext_label:fasttext模型的输出，共有两种结果:__label__0和__label__1。__label__0表示闲聊型，__label__1表示任务型\n",
" '''\n",
" fasttext_label = intention.predict(sentence)[0][0]\n",
" return fasttext_label"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at ./chinese_wwm_pytorch/ were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias']\n",
"- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
]
}
],
"source": [
"from transformers import BertTokenizer, BertModel\n",
"import torch\n",
"\n",
"tokenizer = BertTokenizer.from_pretrained(\"./chinese_wwm_pytorch/\")\n",
"model = BertModel.from_pretrained(\"./chinese_wwm_pytorch/\").to(\"cuda\")\n",
"\n",
"def get_bert_embedding(sentence):\n",
" inputs = tokenizer(sentence, return_tensors=\"pt\").to(\"cuda\")\n",
" outputs = model(**inputs)\n",
" outputs = outputs.last_hidden_state.mean(1)\n",
" return outputs\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def get_best_answer(sentence, candidate):\n",
" \"\"\"\n",
" sentence: 用户输入query， 已经处理好的\n",
" candidate: 通过倒排表返回的候选问题的下标列表\n",
" \n",
" 返回：最佳回复，string形式\n",
" \"\"\"\n",
" cosin_li = []\n",
" sentence_ = get_bert_embedding(sentence)\n",
" for each in tqdm(candidate):\n",
" each_ = get_bert_embedding(\" \".join(QApares[\"question_after_preprocessing\"][each])) \n",
" cosin_li.append(torch.nn.functional.cosine_similarity(each_, sentence_).to(\"cpu\").detach().numpy()[0])\n",
" max_index = np.array(cosin_li).argmax()\n",
" return QApares[\"answer\"][max_index]"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def QA(sentence):\n",
" '''\n",
" 实现一个智能客服系统，输入一个句子sentence，返回一个回答\n",
" '''\n",
" # 若意图识别结果为闲聊型，则默认返回'闲聊机器人'\n",
" if get_intention_result(sentence)=='__label__0':\n",
" return '闲聊机器人'\n",
" # 根据倒排表进行检索获得候选问题集\n",
" candidate = get_retrieve_result(sentence)\n",
" # 若候选问题集大小为0，默认返回'我不明白你在说什么'\n",
" if len(candidate)==0:\n",
" return '我不明白你在说什么'\n",
" return get_best_answer(sentence, candidate)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Building prefix dict from the default dictionary ...\n",
"Loading model from cache C:\\Users\\avaws\\AppData\\Local\\Temp\\jieba.cache\n",
"Loading model cost 0.638 seconds.\n",
"Prefix dict has been built successfully.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"发什么快递\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████████████████████████████████████████████████████| 6610/6610 [01:25<00:00, 77.25it/s]\n"
]
},
{
"data": {
"text/plain": [
"'好的亲亲'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 测试\n",
"QA('发什么快递')"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"什么时候发货呀？\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████████████████████████████████████████████████████| 7021/7021 [01:29<00:00, 78.32it/s]\n"
]
},
{
"data": {
"text/plain": [
"'亲没有优惠了哦真的抱歉呢前两天活动刚结束的'"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 测试\n",
"QA('什么时候发货呀？')"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████████████████████████████████████████████████████| 9382/9382 [02:00<00:00, 77.98it/s]\n"
]
},
{
"data": {
"text/plain": [
"'您好'"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 测试\n",
"QA('最快什么时候可以发货呢，亲？？')"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'闲聊机器人'"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 测试\n",
"QA('一二三四五六七')"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# 本来打算把所有问题的bert向量都先直接算出来的，但是显卡还是太小了，batchsize不能太大\n",
"# 那还不如直接放弃这个做法\n",
"# from transformers import BertTokenizer, BertModel\n",
"# from torch.utils.data import DataLoader, Dataset\n",
"# from torch.nn import Module\n",
"\n",
"# class MyDataset(Dataset):\n",
"# def __init__(self, df, tokenizer):\n",
"# super().__init__()\n",
"# self.data = df[\"question_after_preprocessing\"]\n",
"# self.len = len(df.values)\n",
" \n",
"# def __getitem__(self, index):\n",
"# token = tokenizer(\" \".join(self.data[index]), return_tensors=\"pt\", max_length = 64, padding=\"max_length\").to(DEVICE)\n",
"# token['input_ids'].squeeze_()\n",
"# return token\n",
"\n",
"# def __len__(self):\n",
"# return self.len\n",
"\n",
" \n",
"# class MyModel(Module):\n",
"# def __init__(self):\n",
"# super().__init__()\n",
"# self.bert = BertModel.from_pretrained(\"./chinese_wwm_pytorch/\")\n",
"# def forward(self, x):\n",
"# print(f\"x {x}\")\n",
"# out = self.bert(**x)\n",
"# print(f\"out {out}\")\n",
"# out = out.last_hidden_state.mean(1)\n",
"# return out\n",
" \n",
"# BATCHSIZE = 32\n",
"# DEVICE = \"cuda\"\n",
"\n",
"\n",
"# tokenizer = BertTokenizer.from_pretrained(\"./chinese_wwm_pytorch/\")\n",
"# myds = MyDataset(QApares, tokenizer)\n",
"# mydl = DataLoader(dataset=myds, batch_size=BATCHSIZE, shuffle=False)\n",
"# model = MyModel().to(DEVICE)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}