Knowing the Words, Missing the Meaning: Evaluating LLMs’ Cultural Understanding Through Sino-Korean Words and Four-Character Idioms
Source abstract
Abstract
This study proposes a new benchmark to evaluate the cultural understanding and natural language processing capabilities of large language models based on Sino-Korean words and four-character idioms. Those are essential linguistic and cultural assets in Korea. Reflecting the official question types of the Korean Hanja Proficiency Test, we constructed four question categories—four-character idioms, synonyms, antonyms, and homophones—and systematically compared the performance of GPT-based and non-GPT LLMs. GPT-4o showed the highest accuracy and explanation quality. However, challenges remain in distinguishing the subtle nuances of individual characters and in adapting to uniquely Korean meanings as opposed to standard Chinese character interpretations. Our findings reveal a gap in LLMs’ understanding of Korea-specific Hanja culture and underscore the need for evaluation tools reflecting these cultural distinctions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Translation Studies and Practices · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of English Language and Literature, Kyungpook National University, Daegu 41566, Republic of Korea
Department of Korean Literature in Chinese Character, Kyungpook National University, Daegu 41566, Republic of Korea
References (41)
Show source-provided references
1. The Harmony of Native Korean Words and Sino-Korean Words
2. Analysis of Sino-Korean Words and Chinese Characters in Elementary School Textbooks
3. Analysis of Lexical Composition and Investigation of Chinese Characters’ Word-Formation Ability through Registered Vocabulary in Korean Dictionaries
4. A Comparative Analysis of the Story-Based Idioms between Chinese and Korean
5. A Comparative Study of Aspect of Acceptance of Chinese Characters in Vietnam and Korea—Focused on Vietnamese Case
6. Lee, K.B., and Lee, K.D. (1982). Lectures on Korean History, Vol. 1: Ancient Period, Iljogak.
7. The Reception Period and Early Settlement Process of Chinese Characters (1)
8. A Study on the Early Settlement Process of Classical Chinese (2)—The Situation Before Its Introduction
9. Lee, K.M. (1998). New Edition of Introduction to the History of the Korean Language, Taehaksa.
10. A Study on the Early Settlement Process of Classical Chinese (3)
11. Sino-Korean Words in Idu Materials from the Unified Silla Period
12. Actual State and Direction of Chinese Characters’ Education
13. A Study on the Cultural Characteristics of Sino-Korean Words—Focusing on Contrast with Chinese and Japanese10.31889/kll.2020.09.192.5 ↗
14. Construction and Challenges of a Chinese Character Word Database for Four East Asian Languages
15. A Comparative Study on the Morphological Differences of Four-Character Idioms in Japanese, Chinese, and Korean
16. Ma, Y. (2023). A Study on the Semantic Change of Sino-Korean Word: Focus on the Homograph Between Korean and Chinese in Kukhanhoeeo. [Ph.D. Thesis, Seoul National University].
17. Naous, T., Ryan, M.J., Ritter, A., and Xu, W. (2023). Having beer after prayer? measuring cultural bias in large language models. arXiv.10.18653/v1/2024.acl-long.862 ↗
18. Ramesh, K., Sitaram, S., and Choudhury, M. (2023). Fairness in language models beyond English: Gaps and challenges. arXiv.10.18653/v1/2023.findings-eacl.157 ↗
19. History, development, and principles of large language models: An introductory survey10.1007/s43681-024-00583-7 ↗
20. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2025, January 06). Improving Language Understanding by Generative Pre-Training. Available online: https://www.mikecaptain.com/resources/pdf/GPT-1.pdf.
21. Language models are unsupervised multitask learners
22. Language models are few-shot learners
23. Training language models to follow instructions with human feedback
24. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
25. OpenAI (2024). Hello GPT-4o, OpenAI.
26. OpenAI (2024). GPT-4 Models Documentation, OpenAI.
27. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., and McKinnon, C. (2022). Constitutional ai: Harmlessness from ai feedback. arXiv.
28. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv.
29. Conneau, A., Lample, G., Rinott, R., Williams, A., Bowman, S.R., Schwenk, H., and Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. arXiv.10.18653/v1/D18-1269 ↗
30. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., and Axmed, M. (2023). Mega: Multilingual evaluation of generative ai. arXiv.10.18653/v1/2023.emnlp-main.258 ↗
31. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages
32. Romero, D., Lyu, C., Wibowo, H.A., Lynn, T., Hamed, I., Kishore, A.N., Mandal, A., Dragonetti, A., Abzaliev, A., and Tonja, A.L. (2024). Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv.
33. Nayak, S., Jain, K., Awal, R., Reddy, S., van Steenkiste, S., Hendricks, L.A., Staczak, K., and Agrawal, A. (2024). Benchmarking vision language models for cultural understanding. arXiv.10.18653/v1/2024.emnlp-main.329 ↗
34. Schneider, F., Holtermann, C., Biemann, C., and Lauscher, A. (2025). GIMMICK–Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking. arXiv.
35. Zheng, C., Huang, M., and Sun, A. (2019). ChID: A large-scale Chinese IDiom dataset for cloze test. arXiv.10.18653/v1/P19-1075 ↗
36. Chinese idiom paraphrasing10.1162/tacl_a_00572 ↗
37. Son, G., Lee, H., Kim, S., Kim, H., Lee, J., Yeom, J.W., Jung, J., Kim, J.W., and Kim, S. (2023). Hae-rae bench: Evaluation of korean knowledge in language models. arXiv.
38. Kim, E., Suk, J., Oh, P., Yoo, H., Thorne, J., and Oh, A. (2024). CLIcK: A benchmark dataset of cultural and linguistic intelligence in Korean. arXiv.
39. KoBBQ: Korean bias benchmark for question answering10.1162/tacl_a_00661 ↗
40. Jang, S., Lee, S., and Yu, H. (2024). KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark. arXiv.
41. Wang, X., Yeo, J., Lim, J.H., and Kim, H. (2024). KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context. arXiv.