Semantic Representation Using Sub-Symbolic Knowledge in Commonsense Reasoning
Source abstract
Abstract
The commonsense question and answering (CSQA) system predicts the right answer based on a comprehensive understanding of the question. Previous research has developed models that use QA pairs, the corresponding evidence, or the knowledge graph as an input. Each method executes QA tasks with representations of pre-trained language models. However, the ability of the pre-trained language model to comprehend completely remains debatable. In this study, adversarial attack experiments were conducted on question-understanding. We examined the restrictions on the question-reasoning process of the pre-trained language model, and then demonstrated the need for models to use the logical structure of abstract meaning representations (AMRs). Additionally, the experimental results demonstrated that the method performed best when the AMR graph was extended with ConceptNet. With this extension, our proposed method outperformed the baseline in diverse commonsense-reasoning QA tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Biomedical Text Mining and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul 02841, Korea
Human-Inspired AI and Computing Research Center, Korea University, Seoul 02841, Korea
Funding
Institute of Information & communications Technology Planning & Evaluation (IITP) · Korea government (MSIT) · MSIT (Ministry of Science and ICT), Korea · ITRC (Information Technology Research Center) · Basic Science Research Program through the National Research Foundation of Korea (NRF) · Ministry of Education
References (39)
Show source-provided references
1. Marcus, G. (2020). The next decade in ai: Four steps towards robust artificial intelligence. arXiv.
2. Berant, J., Chou, A., Frostig, R., and Liang, P. (2013, January 18–21). Semantic parsing on freebase from question-answer pairs. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.
3. Yih, W.t., He, X., and Meek, C. (2014, January 22–27). Semantic parsing for single-relation question answering. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA.10.3115/v1/P14-2105 ↗
4. Yih, S.W.t., Chang, M.W., He, X., and Gao, J. (2015, January 26–31). Semantic Parsing Via Staged Query Graph Generation: Question Answering with Knowledge Base. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1128 ↗
5. Mitra, A., and Baral, C. (2016, January 12–17). Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10354 ↗
6. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019, January 2–7). COMMONSENSEQA: A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
7. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (November, January 31). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1260 ↗
8. Singh, P., Lin, T., Mueller, E.T., Lim, G., Perkins, T., and Zhu, W.L. (2002, January 21–25). Open mind common sense: Knowledge acquisition from the general public. Proceedings of the OTM Confederated International Conferences, Rhodes, Greece.10.1007/3-540-36124-3_77 ↗
9. Lin, B.Y., Chen, X., Chen, J., and Ren, X. (2019, January 3–7). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1282 ↗
10. Lv, S., Guo, D., Xu, J., Tang, D., Duan, N., Gong, M., Shou, L., Jiang, D., Cao, G., and Hu, S. (2020, January 7–12). Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6364 ↗
11. Ma, K., Francis, J., Lu, Q., Nyberg, E., and Oltramari, A. (2019). Towards generalizable neuro-symbolic systems for commonsense question answering. arXiv.10.18653/v1/D19-6003 ↗
12. Speer, R., Chin, J., and Havasi, C. (2017, January 4–9). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11164 ↗
13. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., and Schneider, N. (2013, January 8–9). Abstract meaning representation for sembanking. Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, Sofia, Bulgaria.
14. Gross, J.L., Yellen, J., and Zhang, P. (2013). Handbook of Graph Theory, CRC Press.10.1201/b16132 ↗
15. Cai, D., and Lam, W. (2020, January 7–12). Graph transformer for graph-to-sequence learning. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
16. Densely connected graph convolutional networks for graph-to-sequence learning10.1162/tacl_a_00269 ↗
17. Vlachos, A. (2018). Guided neural language generation for abstractive summarization using Abstract Meaning Representation. arXiv.
18. Liao, K., Lebanoff, L., and Liu, F. (2018, January 20–26). Abstract Meaning Representation for Multi-Document Summarization. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA.
19. Bonial, C., Donatelli, L., Lukin, S., Tratz, S., Artstein, R., Traum, D., and Voss, C. (2019, January 1). Augmenting Abstract Meaning Representation for Human-Robot Dialogue. Proceedings of the First International Workshop on Designing Meaning Representations, Florence, Italy.10.18653/v1/W19-3322 ↗
20. Issa, F., Damonte, M., Cohen, S.B., Yan, X., and Chang, Y. (2018, January 5–6). Abstract meaning representation for paraphrase detection. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-1041 ↗
21. Wang, Y., Liu, S., Rastegar-Mojarad, M., Wang, L., Shen, F., Liu, F., and Liu, H. (2017, January 7–10). Dependency and AMR embeddings for drug-drug interaction extraction from biomedical literature. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Northbrook, IL, USA.10.1145/3107411.3107426 ↗
22. Garg, S., Galstyan, A., Hermjakob, U., and Marcu, D. (2016, January 12–17). Extracting biomolecular interactions using semantic parsing of biomedical text. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10337 ↗
23. Bonial, C., Hwang, J., Bonn, J., Conger, K., Babko-Malaya, O., and Palmer, M. (2015, January 14). English propbank annotation guidelines. Proceedings of the Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado, Boulder, CO, USA.
24. Meyer, C.M., and Gurevych, I. (2021, April 01). Wiktionary: A New Rival for Expert-Built Lexicons? Exploring the Possibilities of Collaborative Lexicography, Available online: https://academic.oup.com/book/27204/chapter-abstract/196665268?redirectedFrom=fulltext.
25. WordNet: A lexical database for English10.1145/219717.219748 ↗
26. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., and Ives, Z. (2007, January 11–15). Dbpedia: A nucleus for a web of open data. Proceedings of the Semantic Web, Busan, Korea.10.1007/978-3-540-76298-0_52 ↗
27. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
28. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
29. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., and Le, Q.V. (2019, January 8–14). Xlnet: Generalized autoregressive pretraining for language understanding. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
30. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019, January 6–9). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
31. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2020, January 26–30). Electra: Pre-training text encoders as discriminators rather than generators. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
32. Cai, D., and Lam, W. (2020, January 5–10). AMR Parsing via Graph-Sequence Iterative Inference. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.119 ↗
33. Shi, P., and Lin, J. (2019). Simple BERT Models for Relation Extraction and Semantic Role Labeling. arXiv.
34. Lim, J., Oh, D., Jang, Y., Yang, K., and Lim, H.S. (2020, January 8–13). I Know What You Asked: Graph Path Learning using AMR for Commonsense Reasoning. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.222 ↗
35. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
36. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (, January 25–29). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1179 ↗
37. Shaw, P., Uszkoreit, J., and Vaswani, A. (2018, January 1–6). Self-Attention with Relative Position Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-2074 ↗
38. Salton, G., Ross, R., and Kelleher, J. (December, January 27). Attentive language models. Proceedings of the Eighth International Joint Conference on Natural Language Processing, Taipei, China.
39. Semantic role labeling