Distance-Based Compression Method for Large Language Models
Source abstract
Abstract
The computational cost of the Transformer architecture is highly dependent on the length of the input sequence, with a computational complexity of O(n2) due to the self-attention mechanism. As a result, Transformer-based models, such as Large Language Models, incur significant computational and storage overhead when processing tasks involving long input sequences. To mitigate these challenges, we propose a compression method that allows users to manually adjust the trade-off between compression efficiency and model performance. The method employs a trainable model to minimize information loss, ensuring that the impact on accuracy remains minimal. The method demonstrated an accuracy degradation within acceptable limits on LongBench v2.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Algorithms and Data Compression
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, Hangzhou Normal University, Hangzhou 311121, China
References (30)
Show source-provided references
1. Attention is all you need
2. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw., 5, 157–166.10.1109/72.279181 ↗
3. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning PMLR, Virtual.
4. Chain-of-thought prompting elicits reasoning in large language models
5. Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv.
6. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
7. Kitaev, N., Kaiser, Ł., and Levskaya, A. (2020). Reformer: The efficient transformer. arXiv.
8. Wang, S., Li, B.Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv.
9. Big bird: Transformers for longer sequences
10. Dai, Z. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. arXiv.10.18653/v1/P19-1285 ↗
11. Rae, J.W., Potapenko, A., Jayakumar, S.M., and Lillicrap, T.P. (2019). Compressive transformers for long-range sequence modelling. arXiv.
12. Munkhdalai, T., Faruqui, M., and Gopal, S. (2024). Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv.
13. Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., Han, S., and Sun, M. (2024). Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv.
14. Language models are few-shot learners
15. Exploring the limits of transfer learning with a unified text-to-text transformer
16. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
17. Rajpurkar, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
18. Zhang, Y. (2019). Dialogpt: Large-Scale generative pre-training for conversational response generation. arXiv.10.18653/v1/2020.acl-demos.30 ↗
19. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. (2020). Long range arena: A benchmark for efficient transformers. arXiv.
20. Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., and Dong, Y. (2024). Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv.10.18653/v1/2025.acl-long.183 ↗
21. Li, T., Zhang, G., Do, Q.D., Yue, X., and Chen, W. (2024). Long-context llms struggle with long in-context learning. arXiv.
22. Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M.K., Han, X., Thai, Z.L., Wang, S., and Liu, Z. (2024). ∞ Bench: Extending Long Context Evaluation Beyond 100K Tokens. arXiv.10.18653/v1/2024.acl-long.814 ↗
23. Wu, Y., Rabe, M.N., Hutchins, D., and Szegedy, C. (2022). Memorizing transformers. arXiv.
24. Chandar, S., Ahn, S., Larochelle, H., Vincent, P., Tesauro, G., and Bengio, Y. (2016). Hierarchical memory networks. arXiv.
25. Wu, Y., Zhao, Y., Hu, B., Minervini, P., Stenetorp, P., and Riedel, S. (2022). An efficient memory-augmented transformer for knowledge-intensive nlp tasks. arXiv.10.18653/v1/2022.emnlp-main.346 ↗
26. Roformer: Enhanced transformer with rotary position embedding10.1016/j.neucom.2023.127063 ↗
27. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K.A., and Oguz, B. (2023). Effective long-context scaling of foundation models. arXiv.10.18653/v1/2024.naacl-long.260 ↗
28. Chen, S., Wong, S., Chen, L., and Tian, Y. (2023). Extending context window of large language models via positional interpolation. arXiv.
29. touvron2023llama10.1016/0893-6080(89)90020-8 ↗
30. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., and Lv, C. (2025). Qwen3 technical report. arXiv.