CFP-AL: Combining Model Features and Prediction for Active Learning in Sentence Classification
Source abstract
Abstract
Active learning has been a research area conducted across various domains for a long time, from traditional machine learning to the latest deep learning research. Particularly, obtaining high-quality labeled datasets for supervised learning requires human annotation, and an effective active learning strategy can greatly reduce annotation costs. In this study, we propose a new insight, CFP-AL (Combining model Features and Prediction for Active Learning), from the perspective of feature space by analyzing and diagnosing methods that have shown good performance in NLP (Natural Language Processing) sentence classification. According to our analysis, while previous active learning strategies that focus on finding data near the decision boundary to facilitate classifier tuning are effective, there are very few data points near the decision boundary. Therefore, a more detailed active learning strategy is needed beyond simply finding data near the decision boundary or data with high uncertainty. Based on this analysis, we propose CFP-AL, which considers the model’s feature space, and it demonstrated the best performance across six tasks and also outperformed others in three Out-Of-Domain (OOD) tasks. While suggesting that data sampling through CFP-AL is the most differential classification standard, it showed novelty in suggesting a method to overcome the anisotropy phenomenon of supervised models. Additionally, through various comparative experiments with basic methods, we analyzed which data are most beneficial or harmful for model training. Through our research, researchers will be able to expand into the area of considering features in active learning, which has been difficult so far.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Hanyang University, Seoul 04763, Republic of Korea
Funding
National Research Foundation of Korea (NRF) · the Institute of Information and Communications Technology Planning and 356 evaluation (IITP) · Korean Government
References (33)
Show source-provided references
1. Lewis, D.D., and Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. Machine Learning Proceedings 1994, Elsevier.10.1016/B978-1-55860-335-6.50026-X ↗
2. Active learning with statistical models10.1613/jair.295 ↗
3. Xia, Y., Liu, X., Yu, T., Kim, S., Rossi, R.A., Rao, A., Mai, T., and Li, S. (2024). Hallucination Diversity-Aware Active Learning for Text Summarization. arXiv.10.18653/v1/2024.naacl-long.479 ↗
4. Luo, Y., Yang, Z., Meng, F., Li, Y., Guo, F., Qi, Q., Zhou, J., and Zhang, Y. (2023). Xal: Explainable active learning makes classifiers better low-resource learners. arXiv.10.18653/v1/2024.naacl-long.372 ↗
5. Ru, D., Feng, J., Qiu, L., Zhou, H., Wang, M., Zhang, W., Yu, Y., and Li, L. (2020). Active sentence learning by adversarial uncertainty sampling in discrete space. arXiv.10.18653/v1/2020.findings-emnlp.441 ↗
6. Zhu, J., Wang, H., Yao, T., and Tsou, B.K. (2008, January 18–22). Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. Proceedings of the 22nd International Conference on Computational Linguistics, Coling 2008, Manchester, UK.10.3115/1599081.1599224 ↗
7. Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., and Agarwal, A. (2020, January 26–30). Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
8. Rajaee, S., and Pilehvar, M.T. (2021). An isotropy analysis in the multilingual BERT embedding space. arXiv.10.18653/v1/2022.findings-acl.103 ↗
9. Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv.10.18653/v1/D19-1006 ↗
10. Fuster Baggetto, A. (2022, January 7–11). Is anisotropy really the cause of BERT embeddings not being semantic?. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates.10.18653/v1/2022.findings-emnlp.314 ↗
11. Moens, M.F., Huang, X., Specia, L., and Yih, S.W.t. (2021, January 7–11). Active Learning by Acquiring Contrastive Examples. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online, Punta Cana, Dominican Republic.
12. Vaswani, A. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
13. Burstein, J., Doran, C., and Solorio, T. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
14. Howard, J., and Ruder, S. (2018, January 15–20). Fine-tuned Language Models for Text Classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.
15. Sun, C., Qiu, X., Xu, Y., and Huang, X. (2019, January 18–20). How to Fine-Tune BERT for Text Classification?. Proceedings of the Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China.10.1007/978-3-030-32381-3 ↗
16. Muresan, S., Nakov, P., and Villavicencio, A. (2022, January 22–27). A Closer Look at How Fine-tuning Changes BERT. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.
17. Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (2021, January 6–11). DirectProbe: Studying Representations without Classifiers. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.
18. Webber, B., Cohn, T., He, Y., and Liu, Y. (2020, January 16–20). On the Sentence Embeddings from Pre-trained Language Models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.
19. Su, J., Cao, J., Liu, W., and Ou, Y. (2021). Whitening sentence representations for better semantics and faster retrieval. arXiv.
20. Moens, M.F., Huang, X., Specia, L., and Yih, S.W.t. (2021, January 7–11). SimCSE: Simple Contrastive Learning of Sentence Embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online, Punta Cana, Dominican Republic.
21. Zheng, J., Qiu, S., and Ma, Q. (2023). Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models. arXiv.10.18653/v1/2024.acl-long.794 ↗
22. Multi-class active learning by uncertainty sampling with diversity maximization10.1007/s11263-014-0781-x ↗
23. How to measure uncertainty in uncertainty sampling for active learning10.1007/s10994-021-06003-9 ↗
24. Webber, B., Cohn, T., He, Y., and Liu, Y. (2020, January 16–20). Cold-start Active Learning through Self-supervised Language Modeling. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.
25. Lin, D., Matsumoto, Y., and Mihalcea, R. (2011, January 19–24). Learning Word Vectors for Sentiment Analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
26. Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Bethard, S. (2013, January 18–21). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WT, USA.
27. Kondrak, G., and Watanabe, T. (December, January 27). PubMed 200k RCT: A Dataset for Sequential Sentence Classification in Medical Abstracts. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Taipei, Taiwan.
28. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015, January 7–12). Character-level Convolutional Networks for Text Classification. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
29. Linzen, T., Chrupała, G., and Alishahi, A. (2018, January 1). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium.
30. Palmer, M., Hwa, R., and Riedel, S. (2017, January 9–11). A Continuously Growing Dataset of Sentential Paraphrases. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.
31. Desai, S., and Durrett, G. (2020, January 16–20). Calibration of Pre-trained Transformers. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Online.10.18653/v1/2020.emnlp-main.21 ↗
32. Liu, Q., and Schlangen, D. (2020, January 16–20). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online.
33. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. (2020). Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv.