Uncertainty-Aware Active Meta-Learning for Few-Shot Text Classification
Source abstract
Abstract
Low-resource natural language understanding is one of the challenges in the field of language understanding. As natural language processing and natural language understanding take center stage in machine learning, these challenges need solutions more than ever. This paper introduces the technique of Uncertainty-Aware Active Meta-Learning (UA-AML), a methodology designed to enhance the efficiency of models in low-resource natural language understanding tasks. This methodology is particularly significant in the context of limited data availability, a common challenge in the field of natural language processing. Uncertainty-Aware Active Meta-Learning enables the selection of high-quality tasks from a diverse range of task data available during the learning process. By quantifying the prediction uncertainty of the model for the input data, we provide a loss function and learning strategy that can adjust the influence of the input data on the model’s learning. This approach ensures that the most relevant and informative data are utilized during the learning process, optimizing learning efficiency and model performance. We have applied this meta-learning technique to tasks in low-resource natural language understanding, such as few-shot relation classification and few-shot sentiment classification. Our experimental results, which are carried out on the Amazon Review Sentiment Classification (ARSC) and the FewRel dataset, demonstrate that this technique can construct low-resource natural language understanding models with improved performance, providing a robust solution for tasks with limited data availability. This research contributes to the expansion of meta-learning techniques beyond their traditional application in computer vision, demonstrating their potential in natural language processing. Our findings suggest that this methodology can be effectively utilized in a wider range of application areas, opening new avenues for future research in low-resource natural language understanding.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Dongguk University, Seoul 04620, Republic of Korea
Funding
MSIT (Ministry of Science and ICT) · Artificial Intelligence Convergence Innovation Human Resources Development · IITP (Institute for Information and Communications Technology Planning and Evaluation) · National Research Foundation of Korea (NRF)
References (41)
Show source-provided references
1. Liu, Z. (2022). Effective Transfer Learning for Low-Resource Natural Language Understanding. arXiv.
2. Low Resource Summarization Using Pre-Trained Language Models10.1145/3675780 ↗
3. Dou, Z.-Y., Yu, K., and Anastasopoulos, A. (2019). Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks. arXiv.10.18653/v1/D19-1112 ↗
4. Advances and Challenges in Meta-Learning: A Technical Review10.1109/TPAMI.2024.3357847 ↗
5. Conklin, H., Wang, B., Smith, K., and Titov, I. (2021). Meta-Learning to Compositionally Generalize. arXiv.10.18653/v1/2021.acl-long.258 ↗
6. Tian, H., Liu, B., Yuan, X.-T., and Liu, Q. (2020, January 23–28). Meta-Learning with Network Pruning. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58529-7_40 ↗
7. Hui, S., Zhou, S., Deng, Y., and Wang, J. (2023). Understanding the Overfitting of the Episodic Meta-Training. arXiv.
8. Rajendran, J., Irpan, A., and Jang, E. (2020). Meta-Learning Requires Meta-Augmentation. arXiv.
9. Quantifying Uncertainties in Natural Language Processing Tasks
10. Ulmer, D. (2024). On Uncertainty In Natural Language Processing. arXiv.
11. Hu, M., Zhang, Z., Zhao, S., Huang, M., and Wu, B. (2023). Uncertainty in Natural Language Processing: Sources, Quantification, and Applications. arXiv.
12. Hou, B., Liu, Y., Qian, K., Andreas, J., Chang, S., and Zhang, Y. (2024). Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. arXiv.
13. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models Are Few-Shot Learners. arXiv.
14. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–17). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
16. Bahdanau, D., Cho, K., and Bengio, Y. (2016). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
17. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. (2019). Unified Language Model Pre-Training for Natural Language Understanding and Generation. arXiv.
18. Zong, C., Xia, F., Li, W., and Navigli, R. (2021, January 1–6). A Survey of Data Augmentation Approaches for NLP. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP, Bangkok, Thailand.
19. A Survey of Transfer Learning10.1186/s40537-016-0043-6 ↗
20. Deep Learning of Representations for Unsupervised and Transfer Learning
21. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., and Liu, C. (2018). A Survey on Deep Transfer Learning. arXiv.10.1007/978-3-030-01424-7_27 ↗
22. HMTL: Heterogeneous Modality Transfer Learning for Audio-Visual Sentiment Analysis10.1109/ACCESS.2020.3006563 ↗
23. Finn, C., Abbeel, P., and Levine, S. (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv.
24. Snell, J., Swersky, K., and Zemel, R.S. (2017). Prototypical Networks for Few-Shot Learning. arXiv.
25. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., and Wierstra, D. (2017). Matching Networks for One Shot Learning. arXiv.
26. Siamese Neural Networks for One-Shot Image Recognition
27. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H.S., and Hospedales, T.M. (2018, January 18–23). Learning to Compare: Relation Network for Few-Shot Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00131 ↗
28. Geng, R., Li, B., Li, Y., Zhu, X., Jian, P., and Sun, J. (2019). Induction Networks for Few-Shot Text Classification. arXiv.10.18653/v1/2020.acl-main.102 ↗
29. Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M.W., Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N. (2016). Learning to Learn by Gradient Descent by Gradient Descent. arXiv.
30. Kaddour, J., Sæmundsson, S., and Deisenroth, M.P. (2020). Probabilistic Active Meta-Learning. arXiv.
31. Graves, A., Bellemare, M.G., Menick, J., Munos, R., and Kavukcuoglu, K. (2017, January 6–11). Automated Curriculum Learning for Neural Networks. Proceedings of the 34th International Conference on Machine Learning, PMLR, Sydney, Australia.
32. Yoo, D., and Kweon, I.S. (2019). Learning Loss for Active Learning. arXiv.10.1109/CVPR.2019.00018 ↗
33. Gal, Y., and Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv.
34. Farahani, A., Pourshojae, B., Rasheed, K., and Arabnia, H.R. (2021). A Concise Review of Transfer Learning. arXiv.10.1109/CSCI51800.2020.00065 ↗
35. Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. (2020). A Comprehensive Survey on Transfer Learning. arXiv.
36. A Survey on Transfer Learning10.1109/TKDE.2009.191 ↗
37. Graves, A. (2010, January 6–9). Practical Variational Inference for Neural Networks. Proceedings of the 24th International Conference on Neural Information Processing Systems, Vancouver, CA, Canada.
38. Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. (2019). Class-Balanced Loss Based on Effective Number of Samples. arXiv.10.1109/CVPR.2019.00949 ↗
39. Gao, T., Han, X., Zhu, H., Liu, Z., Li, P., Sun, M., and Zhou, J. (2019). FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. arXiv.10.18653/v1/D19-1649 ↗
40. Zaenen, A., and van den Bosch, A. (2007, January 23–30). Biographies, Bollywood, Boom-Boxes and Blenders: Domain Adaptation for Sentiment Classification. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, Prague, Czech Republic.
41. Gao, T., Han, X., Xie, R., Liu, Z., Lin, F., Lin, L., and Sun, M. (2019). Neural Snowball for Few-Shot Relation Learning. arXiv.10.1609/aaai.v34i05.6281 ↗