A Multi-Layer Attention Knowledge Tracking Method with Self-Supervised Noise Tolerance
Source abstract
Abstract
The knowledge tracing method based on deep learning is used to assess learners’ cognitive states, laying the foundation for personalized education. However, deep learning methods are inefficient when processing long-term series data and are prone to overfitting. To improve the accuracy of cognitive state prediction, we design a Multi-layer Attention Self-supervised Knowledge Tracing Method (MASKT) using self-supervised learning and the Transformer method. In the pre-training stage, MASKT uses a random forest method to filter out positive and negative correlation feature embeddings; then, it reuses noise-processed restoration tasks to extract more learnable features and enhance the learning ability of the model. The Transformer in MASKT not only solves the problem of long-term dependencies between input and output using an attention mechanism, but also has parallel computing capabilities that can effectively improve the learning efficiency of the prediction model. Finally, a multidimensional attention mechanism is integrated into cross-attention to further optimize prediction performance. The experimental results show that, compared with various knowledge tracing models on multiple datasets, MASKT’s prediction performance remains 2 percentage points higher. Compared with the multidimensional attention mechanism of graph neural networks, MASKT’s time efficiency is shortened by nearly 30%. Due to the improvement in prediction accuracy and performance, this method has broad application prospects in the field of cognitive diagnosis in intelligent education.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Intelligent Tutoring Systems and Adaptive Learning · Online Learning and Analytics · Machine Learning in Healthcare
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Engineering, Linyi University, Linyi 276000, China
Funding
Shandong Provincial Natural Science Foundation · Innovation Capacity Improvement Project for Technology-based SMEs in Shandong Province · Youth Innovation Team Development Plan for Universities in Shandong Province · Undergraduate Teaching Reform Research Project of Shandong Province
References (24)
Show source-provided references
1. Yudelson, M.V., Koedinger, K.R., and Gordon, G.J. (2013, January 9–13). Individualized bayesian knowledge tracing models. Proceedings of the Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA.
2. Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L.J., and Sohl-Dickstein, J. (2015, January 7–12). Deep knowledge tracing. Proceedings of the 29th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
3. Pandey, S., and Karypis, G. (2019). A self-attentive model for knowledge tracing. arXiv.
4. Ghosh, A., Heffernan, N., and Lan, A.S. (2020, January 6–10). Context-aware attentive knowledge tracing. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual, CA USA.10.1145/3394486.3403282 ↗
5. Hierarchical Attention Network Based Interpretable Knowledge Tracing
6. Tian, Z., Zheng, G., and Flanagan, B. (2021, January 22–26). BEKT: Deep knowledge tracing with bidirectional encoder representations from transformers. Proceedings of the International Conference on Computers in Education, Virtual.
7. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
8. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., and Le, Q.V. (2019, January 8–14). Xlnet: Generalized autoregressive pretraining for language understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
9. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.W. (2019, January 8–14). Unified language model pre-training for natural language understanding and generation. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
10. Mnih, A., and Hinton, G.E. (2008, January 8–10). A scalable hierarchical distributed language model. Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
11. Peters, M.E., Neumann, M., Zettlemoyer, L., and Yih, W.T. (2018). Dissecting contextual word embeddings: Architecture and representation. arXiv.10.18653/v1/D18-1179 ↗
12. Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P. (2019, January 3–7). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.10.1145/3357384.3357895 ↗
13. Lee, W., Chun, J., Lee, Y., Park, K., and Park, S. (2022, January 25–29). Contrastive learning for knowledge tracing. Proceedings of the ACM Web Conference 2022, Lyon, France.10.1145/3485447.3512105 ↗
14. Lin, S., and Tian, H. (2020, January 12–14). Short-term metro passenger flow prediction based on random forest and LSTM. Proceedings of the 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chongqing, China.10.1109/ITNEC48623.2020.9084974 ↗
15. Image classification method of cashmere and wool based on the multi-feature selection and random forest method10.1177/00405175211046060 ↗
16. Liu, Z., Liu, Q., Chen, J., Huang, S., Gao, B., Luo, W., and Weng, J. (May, January 30). Enhancing deep knowledge tracing with auxiliary tasks. Proceedings of the ACM Web Conference 2023, Austin, TX, USA.10.1145/3543507.3583866 ↗
17. Shen, S., Liu, Q., Chen, E., Huang, Z., Huang, W., Yin, Y., Su, Y., and Wang, S. (2021, January 14–18). Learning process-consistent knowledge tracing. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore.10.1145/3447548.3467237 ↗
18. Yeung, C.K., and Yeung, D.Y. (2018, January 26–28). Addressing two problems in deep knowledge tracing via prediction-consistent regularization. Proceedings of the Fifth Annual ACM Conference on Learning at Scale, London, UK.10.1145/3231644.3231647 ↗
19. The 2010 KDD Cup Competition Dataset: Engaging the machine learning community in predictive learning analytics10.18608/jla.2016.32.16 ↗
20. Addressing the assessment challenge with an online system that tutors as it assesses10.1007/s11257-009-9063-7 ↗
21. A data repository for the EDM community: The PSLC DataShop
22. King, D.R. (2020, January 19–24). Production implementation of recurrent neural networks in adaptive instructional systems. Proceedings of the Adaptive Instructional Systems: Second International Conference, AIS 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark.10.1007/978-3-030-50788-6_25 ↗
23. Choi, Y., Lee, Y., Shin, D., Cho, J., Park, S., Lee, S., Baek, J., Bae, C., Kim, B., and Heo, J. (2020, January 6–10). Ednet: A large-scale hierarchical dataset in education. Proceedings of the Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco.10.1007/978-3-030-52240-7_13 ↗
24. Wang, T., Ma, F., and Gao, J. (2019, January 2–5). Deep hierarchical knowledge tracing. Proceedings of the 12th International Conference on Educational Data Mining, Montréal, QC, Canada.