Research on Applying the “Shift” Concept to Deep Attention Matching
Source abstract
Abstract
The main purpose of chat robots is to realize intelligent interaction between human beings and chat robots. Generally, a complete conversation involves chat contexts. Before responding, human beings need to extract information from their chat contexts, and human beings are very good at extracting information. However, how to make the chatbots more complete in extracting appropriate contextual information has become a key issue for the chatbots in multi-round dialogues. Most research in recent years has paid attention to the point-to-point matching of responses and utterances at reciprocal granularity, such as SMN (Sequential Matching Network) and DAM (Deep Attention Matching Network), for which these parsing methods affect the effectiveness of the above networks to some extent. For example, DAM introduces an attention mechanism, which can obtain good results by parsing five levels of granularity of response and utterance through a self-attention module, and then matching the same level of granularity to mine similar spatial information in it. Based on the structure of DAM, this paper proposes an information-mining idea for improving DAM, which applies to those models with a multi-layer matching structure. Therefore, based on this idea, this paper presents two improved methods for DAM, so as to improve the accuracy of the information extraction from the contexts of the multiple round chats. The experiments show that the improved DAM has better results than that of the unimproved DAM, which are, respectively, R_2@1 increased by 0.425%, R_10@1 increased by 0.515%, R_10@2 increased by 0.341%, MAP (Mean Average Precision) increased by 0.358% in Douban data set, P@1 increased by 1.16%, R_10@1 increased by 1.17%, and these results are better than those state-of-the-art methods, and the improved methods presented in this paper can be used not only in DAM but also in other models with similar the point-to-point matching structures.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and dialogue systems · Topic Modeling · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China
References (49)
Show source-provided references
1. Turing, A.M. (2009). Parsing the Turing Test, Springer.
2. Ji, Z., Lu, Z., and Li, H. (2014). An information retrieval approach to short text conversation. arXiv.
3. Wang, H., Lu, Z., Li, H., and Chen, E. (2013, January 18–21). A dataset for research on short-text conversations. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, DC, USA.
4. The hidden information state model: A practical framework for pomdp-based spoken dialogue management10.1016/j.csl.2009.04.001 ↗
5. Zhou, X., Dong, D., Wu, H., Zhao, S., Yu, D., Tian, H., Liu, X., and Yan, R. (2016, January 1–5). Multi-view response selection for human-computer conversation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1036 ↗
6. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv.
7. Attention is all you need
8. Lowe, R., Pow, N., Serban, I., and Pineau, J. (2015). The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv.10.18653/v1/W15-4640 ↗
9. Wu, Y., Wu, W., Xing, C., Zhou, M., and Li, Z. (2016). Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. arXiv.10.18653/v1/P17-1046 ↗
10. Ritter, A., Cherry, C., and Dolan, B. (2011). Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics (ACL).
11. Pomdp-based statistical spoken dialog systems: A review10.1109/JPROC.2012.2225812 ↗
12. Banchs, R.E., and Li, H. (2012, January 8–14). Iris: A chat-oriented dialogue system based on the vector space model. Proceedings of the ACL 2012 System Demonstrations, Jeju, Republic of Korea.
13. Gu, J.C., Ling, Z.H., Zhu, X., and Liu, Q. (2019). Dually interactive matching network for personalized response selection in retrieval-based chatbots. arXiv.10.1145/3357384.3358140 ↗
14. Hu, B., Lu, Z., Li, H., and Chen, Q. (2014, January 8–13). Convolutional neural network architectures for matching natural language sentences. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
15. Kadlec, R., Schmid, M., and Kleindienst, J. (2015). Improved deep learning baselines for ubuntu corpus dialogs. arXiv.
16. Training end-to-end dialogue systems with the ubuntu dialogue corpus10.5087/dad.2017.102 ↗
17. Lu, X., Lan, M., and Wu, Y. (2018, January 26–30). Memory-based matching models for multi-turn response selection in retrieval-based chatbots. Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing, Hohhot, China.10.1007/978-3-319-99495-6_23 ↗
18. Wu, B., Wang, B., and Xue, H. (2016, January 11–16). Ranking responses oriented to conversational relevance in chat-bots. Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics, Osaka, Japan. Technical Papers.
19. Yan, R., Song, Y., and Wu, H. (2016, January 17–21). Learning to respond with deep neural networks for retrieval-based human-computer conversation system. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy.10.1145/2911451.2911542 ↗
20. Zhang, Z., Li, J., Zhu, P., Zhao, H., and Liu, G. (2018). Modeling multi-turn conversation with deep utterance aggregation. arXiv.
21. Zhou, X., Li, L., Dong, D., Liu, Y., Chen, Y., Zhao, W.X., Yu, D., and Wu, H. (2018, January 15–20). Multi-turn response selection for chatbots with deep attention matching network. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1103 ↗
22. Shang, L., Lu, Z., and Li, H. (2015). Neural responding machine for short-text conversation. arXiv.10.3115/v1/P15-1152 ↗
23. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2015). A diversity-promoting objective function for neural conversation models. arXiv.10.18653/v1/N16-1014 ↗
24. Li, J., Galley, M., Brockett, C., Spithourakis, G.P., Gao, J., and Dolan, B. (2016). A persona-based neural conversation model. arXiv.10.18653/v1/P16-1094 ↗
25. Serban, I., Klinger, T., Tesauro, G., Talamadupula, K., Zhou, B., Bengio, Y., and Courville, A. (2017, January 4–9). Multiresolution recurrent neural networks: An application to dialogue response generation. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10984 ↗
26. Serban, I., Sordoni, A., Bengio, Y., Courville, A., and Pineau, J. (2016, January 12–17). Building end-to-end dialogue systems using generative hierarchical neural network models. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.9883 ↗
27. Shang, M., Fu, Z., Peng, N., Feng, Y., Zhao, D., and Yan, R. (2018, January 13–19). Learning to converse with noisy data: Generation with calibration. Proceedings of the International Joint Conferences on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/603 ↗
28. Song, H., Zhang, W.N., Cui, Y., Wang, D., and Liu, T. (2019). Exploiting persona information for diverse generation of conversational responses. arXiv.10.24963/ijcai.2019/721 ↗
29. Vinyals, O., and Le, Q. (2015). A neural conversational model. arXiv.
30. Xing, C., Wu, W., Wu, Y., Liu, J., Huang, Y., Zhou, M., and Ma, W.Y. (2016). Topic augmented neural response generation with a joint attention mechanism. arXiv.10.1609/aaai.v31i1.10981 ↗
31. Zhu, Q., Cui, L., Zhang, W., Wei, F., and Liu, T. (2018). Retrieval-enhanced adversarial training for neural response generation. arXiv.10.18653/v1/P19-1366 ↗
32. Berant, J., Chou, A., Frostig, R., and Liang, P. (2013, January 18–21). Semantic parsing on freebase from question-answer pairs. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, DC, USA.
33. Bordes, A., Chopra, S., and Weston, J. (2014). Question answering with subgraph embeddings. arXiv.10.3115/v1/D14-1067 ↗
34. Yao, X., and Van Durme, B. (2014, January 22–27). Information extraction over structured data: Question answering with freebase. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, MD, USA.10.3115/v1/P14-1090 ↗
35. Yih, S.W.T., Chang, M.W., He, X., and Gao, J. (2015, January 28–30). Semantic parsing via staged query graph generation: Question answering with knowledge base. Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP, Beijing, China.10.3115/v1/P15-1128 ↗
36. Zhou, H., Young, T., Huang, M., Zhao, H., Xu, J., and Zhu, X. (2018, January 13). Commonsense knowledge aware conversation generation with graph attention. Proceedings of the Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/643 ↗
37. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
38. Kim, Y., Denton, C., Hoang, L., and Rush, A.M. (2017). Structured attention networks. arXiv.
39. Multi-turn response selection for chatbots with hierarchical aggregation network of multi-representation10.1109/ACCESS.2019.2934149 ↗
40. Self-attention Mechanism Fusion for Recommendation in Heterogeneous Information Network10.1007/978-981-19-3387-5_137 ↗
41. A review on the attention mechanism of deep learning10.1016/j.neucom.2021.03.091 ↗
42. What ChatGPT and generative AI mean for science10.1038/d41586-023-00340-6 ↗
43. Are ChatGPTs knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study
44. A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education10.1007/s12195-022-00754-8 ↗
45. Effective Combination of DenseNet and BiLSTM for Keyword Spotting10.1109/ACCESS.2019.2891838 ↗
46. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
47. Baeza-Yates, R., and Ribeiro-Neto, B. (1999). Modern Information Retrieval, ACM Press.
48. Voorhees, E.M. (1999, January 17–19). The trec-8 question answering track report. Proceedings of the 8th Text Retrieval Conference, Gaithersburg, Maryland. Available online: https://www.researchgate.net/publication/2888359_The_TREC-8_Question_Answering_Track_Evaluation.10.6028/NIST.SP.500-246.qa-overview ↗
49. Yu, C., Jiang, W., Zhu, D., and Li, R. (2019, January 22–24). Stacked multi-head attention for multi-turn response selection in retrieval-based chatbots. Proceedings of the IEEE 2019 Chinese Automation Congress (CAC), Hangzhou, China.10.1109/CAC48633.2019.8997176 ↗