Document Re-Ranking Model for Machine-Reading and Comprehension
Source abstract
Abstract
Recently, the performance of machine-reading and comprehension (MRC) systems has been significantly enhanced. However, MRC systems require high-performance text retrieval models because text passages containing answer phrases should be prepared in advance. To improve the performance of text retrieval models underlying MRC systems, we propose a re-ranking model, based on artificial neural networks, that is composed of a query encoder, a passage encoder, a phrase modeling layer, an attention layer, and a similarity network. The proposed model learns degrees of associations between queries and text passages through dot products between phrases that constitute questions and passages. In experiments with the MS-MARCO dataset, the proposed model demonstrated higher mean reciprocal ranks (MRRs), 0.8%p–13.2%p, than most of the previous models, except for the models based on BERT (a pre-trained language model). Although the proposed model demonstrated lower MRRs than the BERT-based models, it was approximately 8 times lighter and 3.7 times faster than the BERT-based models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Information Retrieval and Search Behavior · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Artificial Intelligence, Konkuk University, Seoul 05029, Korea
Computer Science and Engineering & Artificial Intelligence, Konkuk University, Seoul 05029, Korea
Funding
Konkuk University
References (26)
Show source-provided references
1. GF-Net: Improving machine reading comprehension with feature gates10.1016/j.patrec.2019.10.030 ↗
2. Wang, W., Yang, N., Wei, F., Chang, B., and Zhou, M. (August, January 30). Gated Self-Matching Networks for Reading Comprehension and Question Answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada.
3. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020, January 27–30). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Proceedings of the 8th International Conference on Learning Representations 2020 (ICLR), Addis Ababa, Ethiopia.
4. Chen, D., Fisch, A., Weston, J., and Bordes, A. (August, January 30). Reading Wikipedia to Answer Open-Domain Questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada.
5. Lee, J., Yun, S., Kim, H., Ko, M., and Kang, J. (November, January 31). Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1053 ↗
6. Kratzwald, B., and Feuerriegel, S. (November, January 31). Adaptive Document Retrieval for Deep Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1055 ↗
7. Xiong, C., Dai, Z., Callan, J., Liu, Z., and Power, R. (2017, January 7–11). End-to-End Neural Ad-hoc Ranking with Kernel Pooling. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan.10.1145/3077136.3080809 ↗
8. Guo, J., Fan, Y., Ai, Q., and Croft, W.B. (2016, January 24–28). A Deep Relevance Matching Model for Ad-hoc Retrieval. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA.10.1145/2983323.2983769 ↗
9. Dai, Z., Xiong, C., Callan, J., and Liu, Z. (2018, January 5–9). Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining—WSDM, Marina Del Rey, CA, USA.10.1145/3159652.3159659 ↗
10. Mitra, B., and Craswell, N. (2019). An Updated Duet Model for Passage Re-Ranking. arXiv.
11. Alaparthi, C. (2019, January 12). Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks. Proceedings of the 2nd Workshop on Humanizing AI(HAI) at IJCAI’19, Macao, China.
12. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
13. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
14. Enriching Word Vectors with Subword Information10.1162/tacl_a_00051 ↗
15. Cover, T.M. (1999). Elements of Information Theory, John Wiley & Sons.
16. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014, January 8–13). Empirical evaluation of gated recurrent neural networks on sequence modeling. Proceedings of the NIPS 2014 Workshop on Deep Learning NeurIPS, Montreal, QC, Canada.
17. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014, January 23–25). A Convolutional Neural Network for Modelling Sentences. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA.10.3115/v1/P14-1062 ↗
18. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, L. (2017, January 4–7). Attention Is All You Need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
19. Zhou, X., Wan, X., and Xiao, J. (2016, January 1–5). Attention-based LSTM Network for Cross-Lingual Sentiment Classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1024 ↗
20. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017, January 7–11). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1070 ↗
21. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
22. Anserini: Reproducible ranking baselines using lucene
23. Kingma, D., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
24. Craswell, N. (2009). Mean Reciprocal Rank. Encyclopedia of Database Systems, Springer Science and Business Media LLC.10.1007/978-0-387-39940-9_488 ↗
25. Nogueira, R., and Cho, K. (2019). Passage Re-Ranking with BERT. arXiv.
26. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. (Long and Short Papers).