ArabicEduCrawler: AI-Assisted Focused Crawling and Corpus Construction for Arabic Educational Web Content
Source abstract
Abstract
Arabic natural language processing (NLP) faces major difficulties due to the language’s rich morphological structure and the scarcity of high-quality datasets, especially for educational material distributed across diverse online platforms. Many existing large-scale corpus construction methods depend on extensive web crawling followed by substantial post-processing. This process may introduce irrelevant or low-quality data and often fails to represent the target domain adequately. As a result, a robust approach to developing corpora tailored for domain-sensitive educational NLP systems and linguistic depth is critical, as most current resources are inadequate. This paper presents ArabicEduCrawler, an AI-assisted focused crawling framework designed to improve the acquisition, discovery, and organization of Arabic educational web content. The framework integrates domain-aware source selection, in-crawl Arabic language detection using FastText, large language model (LLM)-assisted XPath extraction, and metadata retrieval to support corpus quality and traceability. Its two-layer architecture combines dynamic web crawling using Scrapy-Playwright with advanced NLP processing, including automatic linguistic annotation with GateNLP and Stanza and a sentence-aware chunking strategy designed for transformer-compatible token limits. Experiments across four major Arabic educational domains resulted in the creation of the Arabic Educational Web Corpus (AraEdu-WC), which consists of 101,770 documents segmented into approximately 286 k text chunks, with more than 50 million tokens, 289,778 sentences, and nearly 3.5 million named entities. The system achieved a harvest ratio of 95.25%, indicating its effectiveness in filtering and retaining relevant content. The sentence-aware chunking evaluation showed consistent improvements in top-ranked retrieval, achieving the highest Hit Rate@10 and MRR@10 across all four embedding models. In particular, the multilingual-E5-large model achieved a Hit Rate@10 of 70%, Precision@10 of 18%, and MRR@10 of 57%. These findings demonstrate that the proposed approach provides an effective balance between crawl efficiency, language purity, and content richness, offering a high-quality Arabic educational corpus for downstream NLP and retrieval research.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Web Data Mining and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technology, Faculty of Computing & Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia
References (71)
Show source-provided references
1. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv.
2. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv.
3. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2021). Scaling language models: Methods, analysis & insights from training gopher. arXiv.
4. Common Crawl Foundation (2026, April 19). Common Crawl. Available online: https://commoncrawl.org/.
5. Wenzek, G., Lachaux, M.A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020, January 11–16). CCNet: Extracting high quality monolingual datasets from web crawl data. Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France.
6. Exploring the limits of transfer learning with a unified text-to-text transformer
7. Suárez, P.J.O., Sagot, B., and Romary, L. (2019). Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. Proceedings of the 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Leibniz-Institut für Deutsche Sprache.
8. The fineweb datasets: Decanting the web for the finest text data at scale
9. Hawasly, M., Mohiuddin, M.T., Mubarak, H., and Boughorbel, S. (2025, January 8–9). ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training. Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China.10.18653/v1/2025.arabicnlp-main.36 ↗
10. LanguageCrawl: A generic tool for building language models upon common Crawl10.1007/s10579-021-09551-7 ↗
11. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. (2021, January 7–11). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.98 ↗
12. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets10.1162/tacl_a_00447 ↗
13. Remus, S., and Biemann, C. (2016, January 23–28). Domain-Specific Corpus Expansion with Focused Webcrawling. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia.10.63317/5mymb5ybxjmt ↗
14. Antoun, W., Baly, F., and Hajj, H. (2020, January 11–16). AraBERT: Transformer-based Model for Arabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, Marseille, France.
15. Abdallah, A., Kasem, M., Abdalla, M., Mahmoud, M., Elkasaby, M., Elbendary, Y., and Jatowt, A. (2024, January 14–18). Arabicaqa: A comprehensive dataset for arabic question answering. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA.10.1145/3626772.3657889 ↗
16. Bhatia, G., Nagoudi, E.M.B., El Mekki, A., Alwajih, F., and Abdul-Mageed, M. (May, January 29). Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, NM, USA.10.18653/v1/2025.findings-naacl.263 ↗
17. Web Crawling10.1561/1500000017 ↗
18. Liu, B., and Menczer, F. (2011). Web Crawling. Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data, Springer.10.1007/978-3-642-19460-3 ↗
19. Zhai, C., and Massung, S. (2016). Text Data Management and Analysis: A Practical Introduction to Information Retrieval and Text Mining, Association for Computing Machinery and Morgan & Claypool.10.1145/2915031 ↗
20. A Survey of Web Crawlers for Information Retrieval10.1002/widm.1218 ↗
21. Manning, C., Raghavan, P., and Schuetze, H. (2009). Introduction to Information Retrieval, Cambridge University Press.
22. Diouf, R., Sarr, E.N., Sall, O., Birregah, B., Bousso, M., and Mbaye, S.N. (2019, January 9–12). Web Scraping: State-of-the-Art and Areas of Application. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9005594 ↗
23. Ayuso, E., Dumfeh Brogya, M.S., Kumar Ahlawat, V., and Sain, M. (2024, January 6–7). From Manual to Machine: How AI Is Redefining Web Scraping for Superior Efficiency: A Literature Review. Proceedings of the 2024 International Conference on Communication, Control, and Intelligent Systems (CCIS), Mathura, India.10.1109/CCIS63231.2024.10931912 ↗
24. Obeid, O., Zalmout, N., Khalifa, S., Taji, D., Oudah, M., Alhafni, B., Inoue, G., Eryani, F., Erdmann, A., and Habash, N. (2020, January 13–15). CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France.
25. Habash, N., Soudi, A., and Buckwalter, T. (2007). On arabic transliteration. Arabic Computational Morphology: Knowledge-Based and Empirical Methods, Springer.
26. AlKhalil Morpho Sys 2: A robust Arabic morpho-syntactic analyzer10.1016/j.jksuci.2016.05.002 ↗
27. Diacritic-based matching of arabic words
28. Arabic morphological analysis techniques
29. AraCovTexFinder: Leveraging the transformer-based language model for Arabic COVID-19 text identification10.1016/j.engappai.2024.107987 ↗
30. Development of Focused Crawlers for Building Large Punjabi News Corpus10.5614/itbj.ict.res.appl.2021.15.3.1 ↗
31. Zhang, J., Su, K., Tian, Y., and Matsumoto, T. (2024). WCC-EC 2.0: Enhancing neural machine translation with a 1.6 M+ web-crawled English-Chinese parallel corpus. Electronics, 13.10.3390/electronics13071381 ↗
32. de Jesus, G., and Nunes, S.S. (2024, January 20–25). Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy.10.63317/3bqdt48jvtjm ↗
33. Corpulyzer: A novel framework for building low resource language corpora10.1109/ACCESS.2021.3049793 ↗
34. Marquard, C., and Suleman, H. (2023, January 17–19). Focused Crawling for Automated IsiXhosa Corpus Building. Proceedings of the South African Institute of Computer Scientists and Information Technologists, Pretoria, South Africa.10.1007/978-3-031-39652-6_2 ↗
35. Humkinar: Construction of a large scale web repository and information system for low resource Urdu language10.1109/ACCESS.2024.3454706 ↗
36. Fahrudin, T.M., Funabiki, N., Brata, K.C., Naing, I., Aung, S.T., Muhaimin, A., and Prasetya, D.A. (2025). An improved reference paper collection system using web scraping with three enhancements. Future Internet, 17.10.3390/fi17050195 ↗
37. A web scraping app for smart literature search of the keywords10.7717/peerj-cs.2384 ↗
38. Ampadi Ramachandran, R., Tell, L.A., Rai, S., Millagaha Gedara, N.I., Xu, X., Riviere, J.E., and Jaberi-Douraki, M. (2023). An Automated Customizable Live Web Crawler for Curation of Comparative Pharmacokinetic Data: An Intelligent Compilation of Research-Based Comprehensive Article Repository. Pharmaceutics, 15.10.3390/pharmaceutics15051384 ↗
39. Constructing a multilingual e-learning ontology through web crawling and scraping10.17762/ijcnis.v15i3.6241 ↗
40. Wang, Y. (2022, January 16–18). Visual Analysis of Finance Courses on Chinese University MOOC Platform Based on Web Crawler. Proceedings of the 6th International Conference on Digital Technology in Education, Hangzhou, China.10.1145/3568739.3568792 ↗
41. Aljemazi, M.A., and Khder, M.A. (2022, January 22–23). Octobot-Web Scarping towards retrieving Google Scholar Data. Proceedings of the 2022 ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS), Virtual.10.1109/ICETSIS55481.2022.9888872 ↗
42. Alrashed, S., Khizbullin, D., and Pugh, D.R. (2024). Fineweb-Edu-Ar: Machine-translated corpus to support Arabic small language models. arXiv.
43. Aloui, M., Chouikhi, H., Chaabane, G., Kchaou, H., and Dhaouadi, C. (2024). 101 billion arabic words dataset. arXiv.
44. El-Hmed, S.B.A. (2026, February 27). Alukah Network. Available online: https://www.alukah.net.
45. Adab Foundation (2026, February 27). World Encyclopedia of Arabic Literature. Available online: https://www.adab.com.
46. Al-Maktaba Al-Shamela Foundation (2026, February 27). Al-Maktaba al-Shamela, Digital Library. Available online: https://shamela.ws.
47. Wikipedia Contributors (2026, February 27). Arabic Wikipedia. Available online: https://ar.wikipedia.org.
48. World Wide Web Consortium (W3C) (2026, April 19). XML Path Language (XPath) Version 1.0. Available online: https://www.w3.org/TR/1999/REC-xpath-19991116/.
49. Evaluating web scraping performance using XPath, CSS selector, regular expression, and HTML DOM with multiprocessing technical applications
50. Automatic XPath generation agents for vertical websites by LLMs10.1007/s44443-025-00071-w ↗
51. Li, Y., Wang, B., and Luan, X. (2024). Xpath agent: An efficient xpath programming agent based on llm for web crawler. arXiv.
52. Scrapy Developers (2026, April 19). Architecture Overview. Available online: https://docs.scrapy.org/en/latest/topics/architecture.html.
53. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016). FastText.zip: Compressing text classification models. arXiv.
54. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2026, April 19). fastText Language Identification, 2024. Available online: https://github.com/facebookresearch/fastText.
55. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
56. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023). The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv.
57. Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., and Elazar, Y. (2024, January 11–16). Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.10.18653/v1/2024.acl-long.840 ↗
58. Alsubhi, J., Alahmadi, M.D., Alhusayni, A., Aldailami, I., Hamdine, I., Shabana, A., Iskandar, Y., and Khayyat, S. (2025). Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Components. arXiv.
59. Wang, Z., Gao, C., Xiao, C., Huang, Y., Si, S., Luo, K., Bai, Y., Li, W., Duan, T., and Lv, C. (August, January 27). Document Segmentation Matters for Retrieval-Augmented Generation. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria.10.18653/v1/2025.findings-acl.422 ↗
60. Qu, R., Tu, R., and Bao, F.S. (May, January 29). Is Semantic Chunking Worth the Computational Cost?. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, NM, USA.10.32388/M7YKDZ ↗
61. Merola, C., and Singh, J. (2025). Reconstructing context: Evaluating advanced chunking strategies for retrieval-augmented generation. Proceedings of the International Workshop on Knowledge-Enhanced Information Retrieval, Springer.10.1007/978-3-032-02899-0_1 ↗
62. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V., Aswani, N., Roberts, I., Gorrell, G., Funk, A., Roberts, A., and Damljanovic, D. (2023). Developing Language Processing Components with GATE: Version 9 (User Guide), The University of Sheffield, Department of Computer Science.
63. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C.D. (2020, January 5–10). Stanza: A Python natural language processing toolkit for many human languages. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online.10.18653/v1/2020.acl-demos.14 ↗
64. Ollama, Inc. (2026, January 20). Ollama: Open-Source Software Tool for Large Language Model. Available online: https://ollama.com.
65. Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. (2024). Multilingual E5 Text Embeddings: A Technical Report. arXiv.
66. Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.10.18653/v1/D19-1410 ↗
67. Nacar, O., Koubaa, A., Sibaee, S., Al-Habashi, Y., Ammar, A., and Boulila, W. (2025). GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training. arXiv.
68. Nacar, O., and Koubaa, A. (2024). Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning. arXiv.10.1007/978-3-031-90573-5_6 ↗
69. The faiss library10.1109/TBDATA.2025.3618474 ↗
70. ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison
71. Bhat, S.R., Rudat, M., Spiekermann, J., and Flores-Herr, N. (2025). Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis. arXiv.