esCorpius-m: A Massive Multilingual Crawling Corpus with a Focus on Spanish
Source abstract
Abstract
In recent years, transformer-based models have played a significant role in advancing language modeling for natural language processing. However, they require substantial amounts of data and there is a shortage of high-quality non-English corpora. Some recent initiatives have introduced multilingual datasets obtained through web crawling. However, there are notable limitations in the results for some languages, including Spanish. These datasets are either smaller compared to other languages or suffer from lower quality due to insufficient cleaning and deduplication. In this paper, we present esCorpius-m, a multilingual corpus extracted from around 1 petabyte of Common Crawl data. It is the most extensive corpus for some languages with such a level of high-quality content extraction, cleanliness, and deduplication. Our data curation process involves an efficient cleaning pipeline and various deduplication methods that maintain the integrity of document and paragraph boundaries. We also ensure compliance with EU regulations by retaining both the source web page URL and the WARC shared origin URL.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Data Quality and Management · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
LHF Labs, 48007 Bilbao, Spain
Department of Mathematics, Universidad Autónoma de Madrid, 28049 Madrid, Spain
School of Informatics, University of Edinburgh, Edinburgh EH8 9YL, UK
Department of Software Engineering, University of Granada, 18071 Granada, Spain
Research Centre for Information and Communications Technologies (CITIC-UGR), 18071 Granada, Spain
Funding
MCIN/AEI/10.13039/501100011033 · European Union “NextGenerationEU/PRTR”
References (49)
Show source-provided references
1. Gutiérrez-Fandiño, A., Pérez-Fernández, D., Armengol-Estapé, J., Griol, D., and Callejas, Z. (2022, January 14–16). esCorpius: A Massive Spanish Crawling Corpus. Proceedings of the IberSPEECH 2022 Conference, Granada, Spain.10.21437/IberSPEECH.2022-26 ↗
2. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2022). On the Opportunities and Risks of Foundation Models. arXiv.
3. Exploring the frontiers of deep learning and natural language processing: A comprehensive overview of key challenges and emerging trends10.1016/j.nlp.2023.100026 ↗
4. OECD (2023). AI Language Models: Technological, Socio-Economic and Policy Considerations, OECD Publishing.
5. CTRAN: CNN-Transformer-based network for natural language understanding10.1016/j.engappai.2023.107013 ↗
6. Towards better Chinese-centric neural machine translation for low-resource languages10.1016/j.csl.2023.101566 ↗
7. Efficient dynamic feature adaptation for cross language sentiment analysis with biased adversarial training10.1016/j.knosys.2023.110957 ↗
8. Incorporation of company-related factual knowledge into pre-trained language models for stock-related spam tweet filtering10.1016/j.eswa.2023.121021 ↗
9. A comprehensive review of State-of-The-Art methods for Java code generation from Natural Language Text10.1016/j.nlp.2023.100013 ↗
10. Goswamy, T., Singh, I., Barkati, A., and Modi, A. (2020, January 8–13). Adapting a Language Model for Controlled Affective Text Generation. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.251 ↗
11. Natural language understanding for argumentative dialogue systems in the opinion building domain10.1016/j.knosys.2022.108318 ↗
12. McTear, M. (2020). Conversational AI. Dialogue Systems, Conversational Agents, and Chatbots, Morgan and Claypool Publishers.10.1007/978-3-031-02176-3 ↗
13. TxLASM: A novel language agnostic summarization model for text documents10.1016/j.eswa.2023.121433 ↗
14. Pre-trained language models with domain knowledge for biomedical extractive summarization10.1016/j.knosys.2022.109460 ↗
15. Multilingual personalized hashtag recommendation for low resource Indic languages using graph-based deep neural network10.1016/j.eswa.2023.121188 ↗
16. Franco, M., Gaggi, O., and Palazzi, C.E. (2023, January 4–8). Analyzing the Use of Large Language Models for Content Moderation with ChatGPT Examples. Proceedings of the 3rd International Workshop on Open Challenges in Online Social Networks (OASIS’23), Rome, Italy.10.1145/3599696.3612895 ↗
17. SWSNL: Semantic Web Search Using Natural Language10.1016/j.eswa.2012.12.070 ↗
18. Hao, S., Tan, B., Tang, K., Ni, B., Shao, X., Zhang, H., Xing, E., and Hu, Z. (2023, January 9–14). BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, Toronto, ON, Canada.10.18653/v1/2023.findings-acl.309 ↗
19. Wang, C., Liu, X., and Song, D. (2020). Language Models are Open Knowledge Graphs. arXiv.
20. ChatGPT for good? On opportunities and challenges of large language models for education10.1016/j.lindif.2023.102274 ↗
21. A methodological approach to enable natural language interaction in an Intelligent Tutoring System10.1016/j.csl.2023.101516 ↗
22. Revolutionizing Healthcare with ChatGPT: An Early Exploration of an AI Language Model’s Impact on Medicine at Large and its Role in Pediatric Surgery10.1016/j.jpedsurg.2023.07.008 ↗
23. Modified Hierarchical-Attention Network model for legal judgment predictions10.1016/j.datak.2023.102203 ↗
24. Peña, A., Morales, A., Fierrez, J., Serna, I., Ortega-Garcia, J., Puente, Í., Córdova, J., and Córdova, G. (2023, January 21–26). Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs. Proceedings of the Document Analysis and Recognition Conference—ICDAR 2023 Workshops, San Jose, CA, USA.10.1007/978-3-031-41498-5_2 ↗
25. Employing large language models in survey research10.1016/j.nlp.2023.100020 ↗
26. Constructing and analyzing domain-specific language model for financial text mining10.1016/j.ipm.2022.103194 ↗
27. Liu, S., Peng, C., Wang, C., Chen, X., and Song, S. (2023, January 23). icsBERTs: Optimizing Pre-trained Language Models in Intelligent Customer Service. Proceedings of the International Neural Network Society Workshop on Deep Learning Innovations and Applications (INNS DLIA’23), Gold Coast, Australia.10.1016/j.procs.2023.08.150 ↗
28. Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. (2023). Challenges and Applications of Large Language Models. arXiv.
29. Language Models are Few-Shot Learners
30. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
31. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2022). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv.
32. OpenAI (2023). GPT-4 Technical Report. arXiv.
33. A Survey of the Usages of Deep Learning for Natural Language Processing10.1109/TNNLS.2020.2979670 ↗
34. A Survey on Bias and Fairness in Machine Learning
35. Deep learning in clinical natural language processing: A methodical review10.1093/jamia/ocz200 ↗
36. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021, January 6–11). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Virtual.10.18653/v1/2021.naacl-main.41 ↗
37. Sarti, G., and Nissim, M. (2022). IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation. arXiv.
38. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, A., and Gallé, M. (2023). BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv.
39. Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A.V., Le Scao, T., Von Werra, L., Mou, C., González Ponferrada, E., and Nguyen, H. (2023). The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. arXiv.
40. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets10.1162/tacl_a_00447 ↗
41. El-Kishky, A., Chaudhary, V., Guzmán, F., and Koehn, P. (2020, January 16–20). CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online.10.18653/v1/2020.emnlp-main.480 ↗
42. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3–10). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, FAccT ’21, Virtual.10.1145/3442188.3445922 ↗
43. Abadji, J., Ortiz Suarez, P., Romary, L., and Sagot, B. (2022, January 20–25). Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. Proceedings of the 13th Language Resources and Evaluation Conference, Marseille, France.
44. Wenzek, G., Lachaux, M.A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020, January 11–16). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.
45. Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., Forcada, M.L., Kamran, A., Kirefu, F., and Koehn, P. (2020, January 5–10). ParaCrawl: Web-Scale Acquisition of Parallel Corpora. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual.10.18653/v1/2020.acl-main.417 ↗
46. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
47. Enriching Word Vectors with Subword Information10.1162/tacl_a_00051 ↗
48. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. (2022, January 22–27). Deduplicating Training Data Makes Language Models Better. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Dublin, Ireland.10.18653/v1/2022.acl-long.577 ↗
49. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv.