Considering Commonsense in Solving QA: Reading Comprehension with Semantic Search and Continual Learning
Source abstract
Abstract
Unlike previous dialogue-based question-answering (QA) datasets, DREAM, multiple-choice Dialogue-based REAding comprehension exaMination dataset, requires a deep understanding of dialogue. Many problems require multi-sentence reasoning, whereas some require commonsense reasoning. However, most pre-trained language models (PTLMs) do not consider commonsense. In addition, because the maximum number of tokens that a language model (LM) can deal with is limited, the entire dialogue history cannot be included. The resulting information loss has an adverse effect on performance. To address these problems, we propose a Dialogue-based QA model with Common-sense Reasoning (DQACR), a language model that exploits Semantic Search and continual learning. We used Semantic Search to complement information loss from truncated dialogue. In addition, we used Semantic Search and continual learning to improve the PTLM’s commonsense reasoning. Our model achieves an improvement of approximately 1.5% over the baseline method and can thus facilitate QA-related tasks. It contributes toward not only dialogue-based QA tasks but also another form of QA datasets for future tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Human-Inspired AI & Computing Research Center, Korea University, Seoul 02841, Korea
Department of Computer Science and Engineering, Korea University, Seoul 02841, Korea
Funding
Ministry of Science and ICT · National Research Foundation of Korea
References (29)
Show source-provided references
1. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017). Race: Large-scale reading comprehension dataset from examinations. arXiv.10.18653/v1/D17-1082 ↗
2. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
3. Dream: A challenge data set and models for dialogue-based reading comprehension10.1162/tacl_a_00264 ↗
4. Zhao, Y., Zhang, Z., and Zhao, H. (2020). Reference knowledgeable network for machine reading comprehension. arXiv.
5. Jin, D., Gao, S., Kao, J.Y., Chung, T., and Hakkani-tur, D. (2020, January 7–12). Mmm: Multi-stage multi-task learning for multi-choice reading comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6310 ↗
6. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
7. Lim, J., Oh, D., Jang, Y., Yang, K., and Lim, H. (2020). I know what you asked: Graph path learning using AMR for commonsense reasoning. arXiv.10.18653/v1/2020.coling-main.222 ↗
8. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2018). Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv.
9. Speer, R., Chin, J., and Havasi, C. (2017, January 4–9). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11164 ↗
10. Xu, Y., Zhu, C., Xu, R., Liu, Y., Zeng, M., and Huang, X. (2020). Fusing context into knowledge graph for commonsense question answering. arXiv.10.18653/v1/2021.findings-acl.102 ↗
11. Yan, J., Raman, M., Chan, A., Zhang, T., Rossi, R., Zhao, H., Kim, S., Lipka, N., and Ren, X. (2020). Learning Contextualized Knowledge Structures for Commonsense Reasoning. arXiv.10.18653/v1/2021.findings-acl.354 ↗
12. A hybrid Persian sentiment analysis framework: Integrating dependency grammar based rules and deep neural networks10.1016/j.neucom.2019.10.009 ↗
13. Ke, W., Gao, J., Shen, H., and Cheng, X. (2022, January 21–25). ConsistSum: Unsupervised Opinion Summarization with the Consistency of Aspect, Sentiment and Semantic. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, Phoenix, AZ, USA.10.1145/3488560.3498463 ↗
14. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv.10.18653/v1/W18-5446 ↗
15. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you do not know: Unanswerable questions for SQuAD. arXiv.10.18653/v1/P18-2124 ↗
16. Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv.10.18653/v1/D18-1009 ↗
17. Joachims, T. (1996). A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization, Department of Computer Science, Carnegie Mellon University. Technical Report.
18. Robertson, S., and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond, Now Publishers Inc.10.1561/1500000019 ↗
19. Mpnet: Masked and permuted pre-training for language understanding
20. Singh, P., Lin, T., Mueller, E.T., Lim, G., Perkins, T., and Zhu, W.L. (2002). Open mind common sense: Knowledge acquisition from the general public. OTM Confederated International Conferences “On the Move to Meaningful Internet Systems”, Springer.10.1007/3-540-36124-3_77 ↗
21. Bond, F., and Foster, R. (2013, January 4–9). Linking and extending an open multilingual wordnet. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria.
22. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., and Ives, Z. (2007). Dbpedia: A nucleus for a web of open data. The Semantic Web, Springer.10.1007/978-3-540-76298-0_52 ↗
23. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
24. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., and Le, Q.V. (2019, January 8–14). Xlnet: Generalized autoregressive pretraining for language understanding. Proceedings of the Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada.
25. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
26. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv.
27. Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., and Wang, H. (2020, January 7–12). Ernie 2.0: A continual pre-training framework for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6428 ↗
28. Madotto, A., Lin, Z., Zhou, Z., Moon, S., Crook, P., Liu, B., Yu, Z., Cho, E., and Wang, Z. (2020). Continual learning in task-oriented dialogue systems. arXiv.10.18653/v1/2021.emnlp-main.590 ↗
29. Monaikul, N., Castellucci, G., Filice, S., and Rokhlenko, O. (2021, January 2–9). Continual learning for named entity recognition. Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v35i15.17600 ↗