A Green AI Methodology Based on Persistent Homology for Compressing BERT
Source abstract
Abstract
Large Language Models (LLMs) like BERT have gained significant prominence due to their remarkable performance in various natural language processing tasks. However, they come with substantial computational and memory costs. Additionally, they are essentially black-box models, being challenging to explain and interpret. In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a Green AI methodology to prune BERT models using persistent homology, aiming to measure the importance of each neuron by studying the topological characteristics of their outputs. As a result, PBCE can compress BERT significantly by reducing the number of parameters (47% of the original parameters for BERT Base, 42% for BERT Large). The proposed methodology has been evaluated on the standard GLUE Benchmark, comparing the results with state-of-the-art techniques achieving outstanding results. Consequently, PBCE can simplify the BERT model by providing explainability to its neurons and reducing the model’s size, making it more suitable for deployment on resource-constrained devices.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Graph Neural Networks · Topological and Geometric Data Analysis · Advanced Computing and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Artificial Intelligence, University of Granada, 18071 Granada, Spain
Distributed Computational Intelligence and Time Series Lab, University of Granada, 18071 Granada, Spain
Sport and Health University Research Institute, University of Granada, 18071 Granada, Spain
Andalusian Research Institute in Data Science and Computational Intelligence, University of Granada, 18071 Granada, Spain
Department of Software Engineering, University of Granada, 18071 Granada, Spain
Funding
MICIU/AEI/10.13039/501100011033 · Spanish Ministerio de Ciencia, Innovación y Universidades
References (53)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics.
2. A review of green artificial intelligence: Towards a more sustainable future10.1016/j.neucom.2024.128096 ↗
3. Schwartz, R., Dodge, J., Smith, N.A., and Etzioni, O. (2019). Green AI. arXiv.
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
5. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
6. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
7. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv.
8. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2020). TinyBERT: Distilling BERT for Natural Language Understanding. Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics.10.18653/v1/2020.findings-emnlp.372 ↗
9. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics.10.18653/v1/W18-5446 ↗
10. Probability measures on the space of persistence diagrams10.1088/0266-5611/27/12/124007 ↗
11. Chen, M., Wang, D., Feng, S., and Zhang, Y. (2023). Topological Regularization for Representation Learning via Persistent Homology. Mathematics, 11.10.3390/math11041008 ↗
12. Choe, S., and Ramanna, S. (2022). Cubical Homology-Based Machine Learning: An Application in Image Classification. Axioms, 11.10.3390/axioms11030112 ↗
13. Persistent-homology-based machine learning: A survey and a comparative study10.1007/s10462-022-10146-z ↗
14. Routray, M., Vipsita, S., Sundaray, A., and Kulkarni, S. (2022). DeepRHD: An efficient hybrid feature extraction technique for protein remote homology detection using deep learning strategies. Comput. Biol. Chem., 100.10.1016/j.compbiolchem.2022.107749 ↗
15. Beyond Homology Transfer: Deep Learning for Automated Annotation of Proteins10.1007/s10723-018-9450-6 ↗
16. TopP–S: Persistent homology-based multi-task deep neural networks for simultaneous predictions of partition coefficient and aqueous solubility10.1002/jcc.25213 ↗
17. TopoBERT: Exploring the topology of fine-tuned word representations10.1177/14738716231168671 ↗
18. Clark, K., Khandelwal, U., Levy, O., and Manning, C.D. (2019). What Does BERT Look at? An Analysis of BERT’s Attention. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics.10.18653/v1/W19-4828 ↗
19. (2023, September 03). google-bert/bert-base-cased · Hugging Face—huggingface.co. Available online: https://huggingface.co/bert-base-cased.
20. (2023, September 03). google-bert/bert-large-cased · Hugging Face—huggingface.co. Available online: https://huggingface.co/bert-large-cased.
21. Compression of Deep Learning Models for Text: A Survey10.1145/3487045 ↗
22. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT10.1162/tacl_a_00413 ↗
23. Lee, H.D., Lee, S., and Kang, U. (2021). AUBER: Automated BERT regularization. PLoS ONE, 16.10.1371/journal.pone.0253241 ↗
24. Zhang, X., Fan, J., and Hei, M. (2022). Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning. Appl. Sci., 12.10.3390/app122312055 ↗
25. Michel, P., Levy, O., and Neubig, G. (2019, January 8–14). Are sixteen heads really better than one?. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
26. Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics.10.18653/v1/P19-1580 ↗
27. Huang, S., Liu, N., Liang, Y., Peng, H., Li, H., Xu, D., Xie, M., and Ding, C. (2022). An Automatic and Efficient BERT Pruning for Edge AI Systems. arXiv.10.1109/ISQED54688.2022.9806197 ↗
28. Zheng, D., Li, J., Yang, Y., Wang, Y., and Pang, P.C.I. (2024). MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model. Appl. Sci., 14.10.3390/app14146171 ↗
29. DDK: Dynamic structure pruning based on differentiable search and recursive knowledge distillation for BERT10.1016/j.neunet.2024.106164 ↗
30. Uncertainty-Driven Knowledge Distillation for Language Model Compression10.1109/TASLP.2023.3289303 ↗
31. Lin, Y.J., Chen, K.Y., and Kao, H.Y. (2023). LAD: Layer-Wise Adaptive Distillation for BERT Model Compression. Sensors, 23.10.3390/s23031483 ↗
32. You only compress once: Towards effective and elastic BERT compression via exploit–explore stochastic nature gradient10.1016/j.neucom.2024.128140 ↗
33. The Lottery Ticket Hypothesis for Pre-trained BERT Networks
34. Guo, F.M., Liu, S., Mungall, F.S., Lin, X., and Wang, Y. (2019). Reweighted Proximal Pruning for Large-Scale Language Representation. arXiv.
35. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., and Keutzer, K. (2019). Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. arXiv.10.1609/aaai.v34i05.6409 ↗
36. Li, B., Kong, Z., Zhang, T., Li, J., Li, Z., Liu, H., and Ding, C. (2020). Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning. Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics.10.18653/v1/2020.findings-emnlp.286 ↗
37. Piao, T., Cho, I., and Kang, U. (2022). SensiMix: Sensitivity-Aware 8-bit index 1-bit value mixed precision quantization for BERT compression. PLoS ONE, 17.10.1371/journal.pone.0265621 ↗
38. (2024, September 08). legacy-datasets/wikipedia · Datasets at Hugging Face—huggingface.co. Available online: https://huggingface.co/datasets/wikipedia.
39. Williams, A., Nangia, N., and Bowman, S. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics.10.18653/v1/N18-1101 ↗
40. Wang, Z., Hamza, W., and Florian, R. (2017). Bilateral multi-perspective matching for natural language sentences. arXiv.10.24963/ijcai.2017/579 ↗
41. (2024, September 04). The Stanford Question Answering Dataset—rajpurkar.github.io. Available online: https://rajpurkar.github.io/SQuAD-explorer/.
42. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., and Potts, C. (2013, January 18–21). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.10.18653/v1/D13-1170 ↗
43. Neural Network Acceptability Judgments10.1162/tacl_a_00290 ↗
44. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Association for Computational Linguistics.10.18653/v1/S17-2001 ↗
45. (2023, September 04). Download Microsoft Research Paraphrase Corpus from Official Microsoft Download Center—microsoft.com. Available online: https://www.microsoft.com/en-us/download/details.aspx?id=52398.
46. Ide, N., and Pustejovsky, J. (2017). The Recognizing Textual Entailment Challenges: Datasets and Methodologies. Handbook of Linguistic Annotation, Springer.10.1007/978-94-024-0881-2 ↗
47. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2024). A Comprehensive Overview of Large Language Models. arXiv.
48. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., and Kumar, A. (2023). Holistic Evaluation of Language Models. arXiv.10.1111/nyas.15007 ↗
49. A Survey on Evaluation of Large Language Models10.1145/3641289 ↗
50. Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23.10.3390/e23010018 ↗
51. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy.10.1109/DSAA.2018.00018 ↗
52. Edelsbrunner, H., and Harer, J. (2010). Computational Topology: An Introduction, American Mathematical Society.10.1090/mbk/069 ↗
53. Hensel, F., Moor, M., and Rieck, B. (2021). A Survey of Topological Machine Learning Methods. Front. Artif. Intell., 4.10.3389/frai.2021.681108 ↗