Autocorrelation Matrix Knowledge Distillation: A Task-Specific Distillation Method for BERT Models
Source abstract
Abstract
Pre-trained language models perform well in various natural language processing tasks. However, their large number of parameters poses significant challenges for edge devices with limited resources, greatly limiting their application in practical deployment. This paper introduces a simple and efficient method called Autocorrelation Matrix Knowledge Distillation (AMKD), aimed at improving the performance of smaller BERT models for specific tasks and making them more applicable in practical deployment scenarios. The AMKD method effectively captures the relationships between features using the autocorrelation matrix, enabling the student model to learn not only the performance of individual features from the teacher model but also the correlations among these features. Additionally, it addresses the issue of dimensional mismatch between the hidden states of the student and teacher models. Even in cases where the dimensions are smaller, AMKD retains the essential features from the teacher model, thereby minimizing information loss. Experimental results demonstrate that BERTTINY-AMKD outperforms traditional distillation methods and baseline models, achieving an average score of 83.6% on GLUE tasks. This represents a 4.1% improvement over BERTTINY-KD and exceeds the performance of BERT4-PKD and DistilBERT4 by 2.6% and 3.9%, respectively. Moreover, despite having only 13.3% of the parameters of BERTBASE, the BERTTINY-AMKD model retains over 96.3% of the performance of the teacher model, BERTBASE.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Reinforcement Learning in Robotics · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic Information Engineering, School of Physics and Electronic Information, Henan Polytechnic University, Wenyuan Street, Jiaozuo 454099, China
Funding
Henan Provincial Science and Technology Research Project OF FUNDER
References (41)
Show source-provided references
1. Achiam, O.J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 Technical Report, OpenAI.
2. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
3. Lin, X., Xiong, G., Gou, G., Li, Z., Shi, J., and Yu, J. (2022, January 25–29). ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification. Proceedings of the ACM Web Conference 2022, Lyon, France.10.1145/3485447.3512217 ↗
4. Shi, Z., Luktarhan, N., Song, Y., and Tian, G. (2023). BFCN: A Novel Classification Method of Encrypted Traffic Based on BERT and CNN. Electronics, 12.10.3390/electronics12030516 ↗
5. PAL-BERT: An Improved Question Answering Model
6. Wu, T., Hou, C., Zhao, Z., Lao, S., Li, J., Wong, N., and Yang, Y. (2023). Weight-Inherited Distillation for Task-Agnostic BERT Compression. arXiv.10.18653/v1/2024.findings-naacl.2 ↗
7. Piao, T., Cho, I., and Kang, U. (2022). SensiMix: Sensitivity-Aware 8-bit index & 1-bit value mixed precision quantization for BERT compression. PLoS ONE, 17.10.1371/journal.pone.0265621 ↗
8. Liu, Y., Lin, Z., and Yuan, F. (2021). ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques. arXiv.10.1609/aaai.v35i10.17056 ↗
9. Lin, Y.J., Chen, K.Y., and Kao, H.Y. (2023). LAD: Layer-Wise Adaptive Distillation for BERT Model Compression. Sensors, 23.10.3390/s23031483 ↗
10. Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. (2021). Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. arXiv.
11. Post-training Quantization for Neural Networks with Provable Guarantees10.1137/22M1511709 ↗
12. Muksimova, S., Umirzakova, S., Mardieva, S., and Cho, Y.I. (2023). Enhancing Medical Image Denoising with Innovative Teacher–Student Model-Based Approaches for Precision Diagnostics. Sensors, 23.10.3390/s23239502 ↗
13. Kaushal, A., Vaidhya, T., and Rish, I. (2023). LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression. arXiv.
14. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
15. FedBKD: Heterogenous Federated Learning via Bidirectional Knowledge Distillation for Modulation Classification in IoT-Edge System10.1109/JSTSP.2022.3224597 ↗
16. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2021). LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation. arXiv.
17. Jiang, M., Lin, J., and Wang, Z.J. (2021, January 19–22). ShuffleCount: Task-Specific Knowledge Distillation for Crowd Counting. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.10.1109/ICIP42928.2021.9506698 ↗
18. Wu, Y., Chanda, S., Hosseinzadeh, M., Liu, Z., and Wang, Y. (2022, January 3–7). Few-Shot Learning of Compact Models via Task-Specific Meta Distillation. Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV56688.2023.00620 ↗
19. Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. (2019). Distilling task-specific knowledge from bert into simple neural networks. arXiv.
20. Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Patient knowledge distillation for bert model compression. arXiv.10.18653/v1/D19-1441 ↗
21. Aguilar, G., Ling, Y., Zhang, Y., Yao, B., Fan, X., and Guo, E. (2019). Knowledge Distillation from Internal Representations. arXiv.
22. Dynabert: Dynamic bert with adaptive width and depth
23. Xu, C., Zhou, W., Ge, T., Wei, F., and Zhou, M. (2020). Bert-of-theseus: Compressing bert by progressive module replacing. arXiv.10.18653/v1/2020.emnlp-main.633 ↗
24. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). Tinybert: Distilling bert for natural language understanding. arXiv.10.18653/v1/2020.findings-emnlp.372 ↗
25. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
27. Music Source Separation With Band-Split RNN10.1109/TASLP.2023.3271145 ↗
28. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
29. Sonkar, S., and Baraniuk, R. (2023). Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design. arXiv.
30. Biçici, E., Kanburoglu, A.B., and Türksoy, R.T. (2023, January 21–22). Residual Connections Improve Prediction Performance. Proceedings of the 2023 4th International Informatics and Software Engineering Conference (IISEC), Ankara, Turkiye.10.1109/IISEC59749.2023.10391034 ↗
31. Layer Normalization for TSK Fuzzy System Optimization in Regression Problems10.1109/TFUZZ.2022.3185464 ↗
32. Understanding stance classification of BERT models: An attention-based framework
33. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv.10.18653/v1/W18-5446 ↗
34. Williams, A., Nangia, N., and Bowman, S.R. (2017). A broad-coverage challenge corpus for sentence understanding through inference. arXiv.10.18653/v1/N18-1101 ↗
35. Chen, Z., Zhang, H., Zhang, X., and Zhao, L. (2024, October 08). Quora Question Pairs. 2018. Online Resource. Available online: https://api.semanticscholar.org/CorpusID:233225749.
36. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., and Potts, C. (2013, January 18–21). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.10.18653/v1/D13-1170 ↗
37. Dolan, B., and Brockett, C. (2005, January 14). Automatically constructing a corpus of sentential paraphrases. Proceedings of the Third international workshop on paraphrasing (IWP2005), Jeju Island, Republic of Korea.
38. The Fifth PASCAL Recognizing Textual Entailment Challenge
39. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
40. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for SQuAD. arXiv.10.18653/v1/P18-2124 ↗
41. Turc, I., Chang, M.W., Lee, K., and Toutanova, K. (2019). Well-read students learn better: On the importance of pre-training compact models. arXiv.