WCC-JC: A Web-Crawled Corpus for Japanese-Chinese Neural Machine Translation
Source abstract
Abstract
Currently, there are only a limited number of Japanese-Chinese bilingual corpora of a sufficient amount that can be used as training data for neural machine translation (NMT). In particular, there are few corpora that include spoken language such as daily conversation. In this research, we attempt to construct a Japanese-Chinese bilingual corpus of a certain scale by crawling the subtitle data of movies and TV series from the websites. We calculated the BLEU scores of the constructed WCC-JC (Web Crawled Corpus—Japanese and Chinese) and the other compared corpora. We also manually evaluated the translation results using the translation model trained on the WCC-JC to confirm the quality and effectiveness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Engineering, Shenyang Ligong University, Shenyang 110159, China
Zhuzhou CRRC Times Electric Co., Ltd., Zhuzhou 412001, China
Faculty of Engineering, Gifu University, Gifu 501-1193, Japan
School of Electrical and Information Engineering, Hunan University of Technology, Zhuzhou 412007, China
Funding
Educational Department of Liaoning Province · Shenyang Ligong University
References (33)
Show source-provided references
1. Nakazawa, T., Yaguchi, M., Uchimoto, K., Utiyama, M., Sumita, E., Kurohashi, S., and Isahara, H. (2016, January 23–28). ASPEC: Asian Scientific Paper Excerpt Corpus. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia.
2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.F. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
3. Stefanowitsch, A. (2020). Corpus Linguistics, Language Science Press. Number 7 in Textbooks in Language Sciences.
4. Xu, Y., and Cao, D. (2002). A Study on the Development and Application of Chinese-Japanese Bilingual Corpus: A Collection of Papers, Foreign Language Teaching And Research Press. (In Chinese).
5. Zhang, Y., Uchimoto, K., Ma, Q., and Isahara, H. (2005, January 13–15). Building an Annotated Japanese-Chinese Parallel Corpus—A Part of NICT Multilingual Corpora. Proceedings of the Machine Translation Summit X: Papers, Phuket, Thailand.
6. Koehn, P. (2005, January 13–15). Europarl: A Parallel Corpus for Statistical Machine Translation. Proceedings of the Machine Translation Summit X: Papers, Phuket, Thailand.
7. Lavecchia, C., Smaili, K., and Langlois, D. (2007, January 29–30). Building a bilingual dictionary from movie subtitles based on inter-lingual triggers. Proceedings of the Translating and the Computer 29, London, UK.10.21437/Interspeech.2007-651 ↗
8. The WaCky wide Web: A collection of very large linguistically processed Web-crawled corpora10.1007/s10579-009-9081-4 ↗
9. Smith, J.R., Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. (2013, January 4–9). Dirt Cheap Web-Scale Parallel Text from the Common Crawl. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria.
10. Chu, C., Nakazawa, T., and Kurohashi, S. (2014, January 26–31). Constructing a Chinese—Japanese Parallel Corpus from Wikipedia. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
11. Sojka, P., Horák, A., Kopeček, I., and Pala, K. (2014, January 8–12). Aranea: Yet Another Family of (Comparable) Web Corpora. Proceedings of the Text, Speech and Dialogue, Brno, Czech Republic.10.1007/978-3-319-10816-2 ↗
12. Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016, January 23–28). The United Nations Parallel Corpus v1.0. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia.
13. Pryzant, R., Chung, Y., Jurafsky, D., and Britz, D. (2018, January 7–12). JESC: Japanese-English Subtitle Corpus. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
14. Lison, P., Tiedemann, J., and Kouylekov, M. (2018, January 7–12). OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
15. Park, J., and Hai, Z. (2019, January 13–15). Korean-to-Chinese Machine Translation using Chinese Character as Pivot Clue. Proceedings of the 33rd Pacific Asia Conference on Language, Information and Computation, Hakodate, Japan.
16. Morishita, M., Suzuki, J., and Nagata, M. (2020, January 11–16). JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.
17. Lai, G., Dai, Z., and Yang, Y. (2020). Unsupervised Parallel Corpus Mining on Web Data. arXiv.
18. Hasan, T., Bhattacharjee, A., Samin, K., Hasan, M., Basak, M., Rahman, M.S., and Shahriyar, R. (2020, January 16–20). Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.207 ↗
19. Comparing Web-crawled and traditional corpora10.1007/s10579-020-09487-4 ↗
20. Serrat Roozen, I., and Martínez Martínez, J.M. (2020, January 11–16). EMPAC: An English-Spanish Corpus of Institutional Subtitles. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.
21. Nakazawa, T., Li, L., and Rikters, M. (2021). Construction of a Corpus for Business Scene Dialogue and Issues in Dialogue Translation, The Association for Natural Language Processing (Japan). (In Japanese).
22. Liu, B., and Huang, L. (2021). ParaMed: A parallel corpus for English-Chinese translation in the biomedical domain. BMC Med. Inform. Decis. Mak., 21.10.1186/s12911-021-01621-8 ↗
23. Dou, Z.Y., and Neubig, G. (2021, January 19–23). Word Alignment by Fine-tuning Embeddings on Parallel Corpora. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online.10.18653/v1/2021.eacl-main.181 ↗
24. Schwenk, H., Chaudhary, V., Sun, S., Gong, H., and Guzmán, F. (2021, January 19–23). WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online.10.18653/v1/2021.eacl-main.115 ↗
25. Zhang, J., and Matsumoto, T. (2017, January 5–7). Improving character-level Japanese-Chinese neural machine translation with radicals as an additional input feature. Proceedings of the 2017 International Conference on Asian Language Processing (IALP), Singapore.10.1109/IALP.2017.8300572 ↗
26. Zhang, J., and Matsumoto, T. (2019, January 15–17). Character Decomposition for Japanese-Chinese Character-Level Neural Machine Translation. Proceedings of the 2019 International Conference on Asian Language Processing (IALP), Shanghai, China.10.1109/IALP48816.2019.9037677 ↗
27. Zhang, J., and Matsumoto, T. (2019). Corpus Augmentation for Neural Machine Translation with Chinese-Japanese Parallel Corpora. Appl. Sci., 9.10.3390/app9102036 ↗
28. Li, X., Meng, Y., Sun, X., Han, Q., Yuan, A., and Li, J. (August, January 28). Is Word Segmentation Necessary for Deep Learning of Chinese Representations?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
29. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019, January 3–5). fairseq: A Fast, Extensible Toolkit for Sequence Modeling. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, MI, USA.10.18653/v1/N19-4009 ↗
30. Luong, M.T., Pham, H., and Manning, C.D. (2015, January 17–21). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1166 ↗
31. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
32. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Virtual.10.3115/1073083.1073135 ↗
33. Utiyama, M., and Isahara, H. (2007, January 10–14). A Japanese-English patent parallel corpus. Proceedings of the Machine Translation Summit XI: Papers, Copenhagen, Denmark.