LLM-Enhanced Semantic Text Segmentation
Source abstract
Abstract
This study investigates semantic text segmentation enhanced by large language model (LLM) embeddings. We assess how effectively embeddings capture semantic coherence and topic closure by integrating them into both classical clustering algorithms and a modified graph-based methods. In addition, we propose a simple magnetic clustering algorithm as a lightweight baseline. Experiments are conducted across multiple datasets and embedding models, with segmentation quality evaluated using the boundary segmentation metric. Results demonstrate that LLM embeddings improve segmentation accuracy, highlight dataset-specific difficulties, and reveal how contextual window size and embedding choice affect performance. These findings clarify the strengths and limitations of embedding-based approaches to segmentation and provide insights relevant to retrieval-augmented generation (RAG).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Semantic Web and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratory for Analysis and Modeling of Information Processes, Institute of Information and Computational Technologies, Almaty 050010, Kazakhstan
AI Research Lab, Satbayev University, Almaty 050013, Kazakhstan
School of Digital Technologies, Almaty Management University, Almaty 050060, Kazakhstan
Funding
Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan
References (42)
Show source-provided references
1. Ustalov, D., Somasundaran, S., Panchenko, A., Malliaros, F.D., Hulpuș, I., Jansen, P., and Jana, A. (2020, January 13). A survey of embedding models of entities and relationships for knowledge graph completion. Proceedings of the Graph-Based Methods for Natural Language Processing (TextGraphs), Barcelona, Spain.
2. Liu, Q., Kusner, M.J., and Blunsom, P. (2020). A Survey on Contextual Embeddings. arXiv.
3. Al-Onaizan, Y., Bansal, M., and Chen, Y.N. (2024, January 12–16). Recent Trends in Linear Text Segmentation: A Survey. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, FL, USA.
4. Toleu, A., Tolegen, G., and Makazhanov, A. (2017, January 18–21). Character-based Deep Learning Models for Token and Sentence Segmentation. Proceedings of the 5th International Conference on Turkic Languages Processing (TurkLang 2017), Kazan, Russia.
5. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2018). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. arXiv.10.18653/v1/D17-1070 ↗
6. Fournier, C. (2013). Evaluating Text Segmentation. [Master’s Thesis, University of Ottawa].
7. Galley, M., McKeown, K.R., Fosler-Lussier, E., and Jing, H. (2003, January 7–12). Discourse Segmentation of Multi-Party Conversation. Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, Sapporo, Japan.10.3115/1075096.1075167 ↗
8. Choi, F.Y.Y. (May, January 29). Advances in domain independent linear text segmentation. Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, USA.
9. Solbiati, A., Heffernan, K., Damaskinos, G., Poddar, S., Modi, S., and Calì, J. (2021). Unsupervised Topic Segmentation of Meetings with BERT Embeddings. arXiv.
10. Walker, M., Ji, H., and Stent, A. (2018, January 1–6). Text Segmentation as a Supervised Learning Task. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LO, USA.
11. Gardent, C., Bernardi, R., and Titov, I. (2016, January 11–12). Unsupervised Text Segmentation Using Semantic Relatedness Graphs. Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, Berlin, Germany.
12. Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.S., and Li, Q. (2024, January 1–8). A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, New York, NY, USA.10.1145/3637528.3671470 ↗
13. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv.
14. Leng, Q., Portes, J., Havens, S., Zaharia, M., and Carbin, M. (2024, January 14). Long Context RAG Performance of Large Language Models. Proceedings of the Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, Vancouver, BC, Canada.
15. Tao, C., Shen, T., Gao, S., Zhang, J., Li, Z., Hua, K., Hu, W., Tao, Z., and Ma, S. (2025). LLMs are Also Effective Embedding Models: An In-depth Overview. arXiv.
16. Oro, E., Granata, F.M., and Ruffolo, M. (2025). A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian. Big Data Cogn. Comput., 9.10.20944/preprints202502.2143.v1 ↗
17. Ollama developers (2025, August 01). Ollama Project. Available online: https://ollama.com/search?c=embedding.
18. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1410 ↗
19. Nussbaum, Z., Morris, J.X., Duderstadt, B., and Mulyar, A. (2024). Nomic Embed: Training a Reproducible Long Context Text Embedder. arXiv.
20. Li, X., and Li, J. (2023). AnglE-optimized Text Embeddings. arXiv.
21. Lee, S., Shakir, A., Koenig, D., and Lipp, J. (2025, October 07). Open Source Strikes Bread-New Fluffy Embeddings Model. Available online: https://www.mixedbread.com/blog/mxbai-embed-large-v1.
22. Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv.10.18653/v1/2024.findings-acl.137 ↗
23. Scikit-learn: Machine Learning in Python
24. Yu, S.X., and Shi, J. (2003, January 14–17). Multiclass Spectral Clustering. Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV 2003), Nice, France.
25. Monath, N., Dubey, K.A., Guruganesh, G., Zaheer, M., Ahmed, A., McCallum, A., Mergen, G., Najork, M., Terzihan, M., and Tjanaka, B. (2021, January 14–18). Scalable Hierarchical Agglomerative Clustering. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’21, Singapore.10.1145/3447548.3467404 ↗
26. Scaling analysis of affinity propagation10.1103/PhysRevE.81.066102 ↗
27. A k-means clustering algorithm10.2307/2346830 ↗
28. How to Use K-means for Big Data Clustering?10.1016/j.patcog.2022.109269 ↗
29. Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019, January 4–8). Optuna: A Next-Generation Hyperparameter Optimization Framework. Proceedings of the The 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, AK, USA.10.1145/3292500.3330701 ↗
30. Schuetze, H., Fung, P., and Poesio, M. (2013, January 4–9). Evaluating Text Segmentation using Boundary Edit Distance. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria.
31. Statistical Models for Text Segmentation10.1023/A:1007506220214 ↗
32. A Critique and Improvement of an Evaluation Metric for Text Segmentation10.1162/089120102317341756 ↗
33. Lexical cohesion computed by thesaural relations as an indicator of the structure of text
34. Ostendorf, M., Collins, M., Narayanan, S., Oard, D.W., and Vanderwende, L. (2009). Global Models of Document Structure using Latent Permutations. Human Language Technologies: Proceedings of the 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Boulder, CO, USA, 31 May–5 June 2009, Association for Computational Linguistics.
35. Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages
36. Budge, I., Klingemann, H.D., Volkens, A., Bara, J., and Tanenbaum, E. (2001). Mapping Policy Preferences. Estimates for Parties, Electors, and Governments 1945–1998, Oxford University Press.10.1093/oso/9780199244003.001.0001 ↗
37. Lehmann, P., Franzmann, S., Al-Gaddooa, D., Burst, T., Ivanusch, C., Regel, S., Riethmüller, F., Volkens, A., Weßels, B., and Zehnter, L. (2025, October 07). The Manifesto Data Collection. Manifesto Project (MRG/CMP/MARPOR). Version 2024a. 2024. Available online: https://manifesto-project.wzb.eu/datasets/MPDS2024a.
38. Cheung, J.C.K., Hatori, J., Henriquez, C., and Irvine, A. (2012, January 8–14). TopicTiling: A Text Segmentation Algorithm based on LDA. Proceedings of the ACL 2012 Student Research Workshop, Jeju Island, Republic of Korea.
39. Beeferman, D., Berger, A., and Lafferty, J. (1997, January 1–2). Text Segmentation Using Exponential Models. Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, Providence, RI, USA.
40. Ostendorff, M. (2025, August 01). Dataset philpapers-2023-10-28. Available online: https://huggingface.co/datasets/malteos/philpapers-2023-10-28/tree/main/data.
41. Ostendorff, M., and Rehm, G. (2023). Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning. arXiv.
42. Krassovitskiy, A. (2025, October 07). Six Small Datasets for Text Segmentation, 2025. Available online: https://data.mendeley.com/datasets/cj22rpfdbb/1.