Recognizing Textual Inference in Mongolian Bar Exam Questions
Source abstract
Abstract
This paper examines how to apply deep learning techniques to Mongolian bar exam questions. Several approaches that utilize eight different fine-tuned transformer models were demonstrated for recognizing textual inference in Mongolian bar exam questions. Among eight different models, the fine-tuned bert-base-multilingual-cased obtained the best accuracy of 0.7619. The fine-tuned bert-base-multilingual-cased was capable of recognizing “contradiction”, with a recall of 0.7857 and an F1 score of 0.7674; it recognized “entailment” with a precision of 0.7750, a recall of 0.7381, and an F1 score of 0.7561. Moreover, the fine-tuned bert-large-mongolian-uncased showed balanced performance in recognizing textual inference in Mongolian bar exam questions, thus achieving a precision of 0.7561, a recall of 0.7381, and an F1 score of 0.7470 for recognizing “contradiction”.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Shiga, Japan
Research Organization of Science and Technology, Ritsumeikan University, Kusatsu 525-8577, Shiga, Japan
College of Information Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Shiga, Japan
Funding
JSPS KAKENHI
References (78)
Show source-provided references
1. (2024, January 09). World Development Indicators. Available online: https://databank.worldbank.org/source/world-development-indicators.
2. (2024, January 09). Mid-Term Evaluation’s Discussion of the Medium-Term Action Plan “New Development”. Available online: https://parliament.mn/n/gico.
3. Shagdarsuren, T. (2001). Study of Mongolian Scripts (Graphic Study of Grammatology, Urlakh Erdem Khevleliin Gazar. [2nd ed.].
4. Svantesson, J., Tsendina, A., Karlsson, A., and Franzén, V. (2005). The phonology of Mongolian, Oxford University Press. [1st ed.].
5. Advances in natural language processing10.1126/science.aaa8685 ↗
6. A survey on deep learning: Algorithms, techniques, and applications
7. Recent trends in deep learning based natural language processing [Review article]10.1109/MCI.2018.2840738 ↗
8. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
9. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
10. Bowman, S.R., Angeli, G., Potts, C., and Manning, C.D. (2015, January 17–21). A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1075 ↗
11. Williams, A., Nangia, N., and Bowman, S. (2018, January 1–6). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), New Orleans, LA, USA.10.18653/v1/N18-1101 ↗
12. (2024, January 18). Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt.
13. (2024, January 18). Competition on Legal Information Extraction/Entailment (COLIEE). Available online: https://sites.ualberta.ca/~rabelo/COLIEE2023/.
14. HUKB at the COLIEE 2022 Statute Law Task
15. Yoshioka, M., and Aoki, Y. (2023, January 19). HUKB at COLIEE 2023 Statute Law Task. Proceedings of the Tenth Competition on Legal Information Extraction/Entailment (COLIEE’2023), Braga, Portugal.
16. Legal Textual Entailment Using Ensemble of Rule-Based and BERT-Based Method with Data Augmentation by Related Article Generation
17. Bui, Q., Do, T., Le, K., Nguyen, D.H., Nguyen, H., Pham, T., and Nguyen, L. (2023, January 19). JNLP COLIEE-2023: Data Argumentation and Large Language Model for Legal Case Retrieval and Entailment. Proceedings of the Tenth Competition on Legal Information Extraction/Entailment (COLIEE’2023), Braga, Portugal.
18. Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., and Brahma, S. (2023). Scaling instruction-finetuned language models. arXiv.
19. Tay, Y., Dehghani, M., Tran, V.Q., Garcia, X., Wei, J., Wang, X., Chung, H., Shakeri, S., Bahri, D., and Schuster, T. (2023). UL2: Unifying Language Learning Paradigms. arXiv.
20. (2024, January 18). Flan-Alpaca: Instruction Tuning from Humans and Machines. Available online: https://huggingface.co/declare-lab/flan-alpaca-xxl.
21. Onaga, T., Fujita, M., and Kano, Y. (2023, January 19). Japanese Legal Bar Problem Solver Focusing on Person Names. Proceedings of the Tenth Competition on Legal Information Extraction/Entailment (COLIEE’2023), Braga, Portugal.
22. (2024, January 18). Luke-Japanese. Available online: https://huggingface.co/studio-ousia/luke-japanese-base-lite.
23. Nguyen, C., Nguyen, P., Tran, T., Nguyen, D., Trieu, A., Pham, T., Dang, A., and Nguyen, M. (2023, January 19). CAPTAIN at COLIEE 2023: Efficient Methods for Legal Information Retrieval and Entailment Tasks. Proceedings of the Tenth Competition on Legal Information Extraction/Entailment (COLIEE’2023), Braga, Portugal.
24. Nogueira, R., Jiang, Z., Pradeep, R., and Lin, J. (2020, January 16–20). Document Ranking with a Pretrained Sequence-to-Sequence Model. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.10.18653/v1/2020.findings-emnlp.63 ↗
25. Ariunaa, O., and Munkhjargal, Z. (2020, January 5–7). Sentiment Analysis for Mongolian Tweets with RNN. Proceedings of the 16th International Conference on IIHMSP in Conjunction with the 13th International Conference on FITAT, Ho Chi Minh City, Vietnam.
26. Cyrillic Word Error Program Based on Machine Learning10.5564/jimdt.v4i1.2661 ↗
27. Dashdorj, Z., Munkhbayar, T., and Grigorev, S. (2023). Deep learning model for Mongolian Citizens’ Feedback Analysis using Word Vector Embeddings. arXiv.
28. A Study on the Appropriate Size of the Mongolian General Corpus10.5121/ijnlc.2023.12302 ↗
29. Lkhagvasuren, G., Rentsendorj, J., and Namsrai, O.E. (2020, January 5–7). Mongolian Part-of-Speech Tagging with Neural Networks. Proceedings of the 16th International Conference on IIHMSP in Conjunction with the 13th International Conference on FITAT, Ho Chi Minh City, Vietnam.
30. Jaimai, P., and Chimeddorj, O. (2009, January 6–7). Part of speech tagging for Mongolian corpus. Proceedings of the 7th Workshop on Asian Language Resources (ACL-IJCNLP2009), Suntec, Singapore.10.3115/1690299.1690314 ↗
31. Ivanov, B., Musa, M., and Dulamragchaa, U. (2020, January 5–7). Mongolian Spelling Error Correction Using Word NGram Method. Proceedings of the 16th International Conference on IIHMSP in Conjunction with the 13th International Conference on FITAT, Ho Chi Minh City, Vietnam.
32. Batsuren, K., Ganbold, A., Chagnaa, A., and Giunchiglia, F. (2019, January 23–27). Building the Mongolian Wordnet. Proceedings of the 10th Global Wordnet Conference, Wroclaw, Poland.
33. Bataa, B., and Altangerel, K. (2012, January 18–21). Word sense disambiguation in Mongolian language. Proceedings of the 7th International Forum on Strategic Technology (IFOST), Tomsk, Russia.10.1109/IFOST.2012.6357625 ↗
34. PC-KIMMO-based description of Mongolian morphology10.3745/JIPS.2005.1.1.041 ↗
35. Dulamragchaa, U., Chadraabal, S., Ivanov, B., and Baatarkhuu, M. (2017, January 15–17). Mongolian language morphology and its database structure. Proceedings of the International Conference Green Informatics (ICGI2017), Fuzhou, China.10.1109/ICGI.2017.56 ↗
36. Chagnaa, A., and Adiyatseren, B. (2011, January 6–9). Two Level Rules for Mongolian Language. Proceedings of the 7th International Conference Multimedia, Information Technology and its Applications (MITA2011), Ulaanbaatar, Mongolia.
37. Munkhjargal, Z., Chagnaa, A., and Jaimai, P. (2016, January 28–30). Morphological Transducer for Mongolian. Proceedings of the International Conference Computational Collective Intelligence (ICCCI 2016), Halkidiki, Greece.10.1007/978-3-319-45246-3_52 ↗
38. Mongolian Phrase Generation and Morphological Analysis based on Phonological and Morphological Constraints10.5715/jnlp.12.5_185 ↗
39. A lemmatization method for Mongolian and its application to indexing for information retrieval10.1016/j.ipm.2009.01.008 ↗
40. Performance Analysis of Optimized Content Extraction for Cyrillic Mongolian Learning Text Materials in the Database10.4236/jcc.2016.410009 ↗
41. Munkhjargal, Z., Bella, G., Chagnaa, A., and Giunchiglia, F. (2015, January 14–17). Named Entity Recognition for Mongolian Language. Proceedings of the International Conference Text, Speech, and Dialogue (TSD2015), Pilsen, Czech Republic.10.1007/978-3-319-24033-6_28 ↗
42. Khaltar, B.O., Fujii, A., and Ishikawa, T. (2006, January 17–18). Extracting Loanwords from Mongolian Corpora and Producing a Japanese-Mongolian Bilingual Dictionary. Proceedings of the 21st International Conference Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia.10.3115/1220175.1220258 ↗
43. Extracting Loanwords from Modern Mongolian Corpora and Producing a Japanese-Mongolian Bilingual Dictionary
44. Lkhagvasuren, G., and Rentsendorj, J. (2020, January 5–7). Open Information Extraction for Mongolian Language. Proceedings of the 16th International Conference on IIHMSP in conjunction with the 13th International Conference on FITAT, Ho Chi Minh City, Vietnam.
45. Damiran, Z., and Altangerel, K. (2014, January 12–14). Author Identification: An Experiment based on Mongolian Literature using Decision Tree. Proceedings of the 7th International Conference Ubi-Media Computing and Workshops (UMEDIA), Ulaanbaatar, Mongolia.10.1109/U-MEDIA.2014.22 ↗
46. Damiran, Z., and Altangerel, K. (1, January 28). Text classification experiments on Mongolian language. Proceedings of the 8th International Forum on Strategic Technology (IFOST2013), Ulaanbaatar, Mongolia.10.1109/IFOST.2013.6616875 ↗
47. Mongolian to Japanese Machine Translation System
48. Enkhbayar, S., Utsuro, T., and Sato, S. (2004, January 16–18). Japanese-Mongolian Machine Translation of Functional Expressions. Proceedings of the 10th Annual Conference of Japanese Association for Natural Language Processing (NLP2004), Tokyo, Japan. (In Japanese).
49. (2024, January 18). BERT Pretrained Models on Mongolian Datasets. Available online: https://github.com/tugstugi/mongolian-bert/.
50. (2024, January 18). Mongolian RoBERTa Base. Available online: https://huggingface.co/bayartsogt/mongolian-roberta-base.
51. (2024, January 18). Mongolian RoBERTa Large. Available online: https://huggingface.co/bayartsogt/mongolian-roberta-large.
52. (2024, January 18). ALBERT Pretrained Model on Mongolian Datasets. Available online: https://github.com/bayartsogt-ya/albert-mongolian/.
53. (2024, January 18). Mongolian GPT2. Available online: https://huggingface.co/bayartsogt/mongolian-gpt2.
54. (2024, January 18). Mongolian Text Classification. Available online: https://github.com/sharavsambuu/mongolian-text-classification.
55. (2024, January 18). BERT-BASE-MONGOLIAN-CASED. Available online: https://huggingface.co/tugstugi/bert-base-mongolian-cased.
56. (2024, January 18). BERT-LARGE-MONGOLIAN-UNCASED. Available online: https://huggingface.co/tugstugi/bert-large-mongolian-uncased.
57. (2024, January 18). mongolian-bert-ner. Available online: https://github.com/enod/mongolian-bert-ner.
58. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020, January 5–10). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.747 ↗
59. (2024, January 18). BERT. Available online: https://github.com/google-research/bert/.
60. (2024, January 18). The Mongolian Wordnet (MonWN). Available online: https://github.com/kbatsuren/monwn.
61. (2024, January 18). Khalkha Mongolian (khk) Paradigms. Available online: https://github.com/unimorph/khk.
62. (2024, January 18). MorphyNet. Available online: https://github.com/kbatsuren/MorphyNet.
63. (2024, January 18). Eduge.mn Dataset. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/eduge.csv.gz.
64. (2024, January 18). News.rar. Available online: https://disk.yandex.ru/d/z5e3MVnKvFvF6w.
65. (2024, January 18). CC-100: Monolingual Datasets from Web Crawl Data. Available online: https://data.statmt.org/cc-100/.
66. (2024, January 18). Mongolian Government Agency—11-11.mn Dataset. Available online: https://www.kaggle.com/datasets/enqush/mongolian-government-agency-1111mn-dataset.
67. (2024, January 18). Mongolian NER Dataset. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/NER_v1.0.json.gz.
68. (2024, January 18). Mongolian_personal_names.csv.gz. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/mongolian_personal_names.csv.gz.
69. (2024, January 18). Mongolian_clan_names.csv.gz. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/mongolian_clan_names.csv.gz.
70. (2024, January 18). Mongolian_company_names.csv.gz. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/mongolian_company_names.csv.gz.
71. (2024, January 18). Districts.csv. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/districts.csv.
72. (2024, January 18). Countries.csv. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/countries.csv.
73. (2024, January 18). Mongolian_abbreviations.csv. Available online: https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/mongolian_abbreviations.csv.
74. (2024, January 18). Unified Legal Information System. Available online: https://legalinfo.mn/.
75. (2024, January 18). The Database of Mongolian Court Orders. Available online: https://shuukh.mn/.
76. (2024, January 18). Mongolian Bar Association. Available online: https://www.mglbar.mn/index.
77. (2024, January 18). BERT-LARGE-MONGOLIAN-CASED. Available online: https://huggingface.co/tugstugi/bert-large-mongolian-cased.
78. (2024, January 18). BERT-BASE-MONGOLIAN-UNCASED. Available online: https://huggingface.co/tugstugi/bert-base-mongolian-uncased.