A Sequential and Intensive Weighted Language Modeling Scheme for Multi-Task Learning-Based Natural Language Understanding
Source abstract
Abstract
Multi-task learning (MTL) approaches are actively used for various natural language processing (NLP) tasks. The Multi-Task Deep Neural Network (MT-DNN) has contributed significantly to improving the performance of natural language understanding (NLU) tasks. However, one drawback is that confusion about the language representation of various tasks arises during the training of the MT-DNN model. Inspired by the internal-transfer weighting of MTL in medical imaging, we introduce a Sequential and Intensive Weighted Language Modeling (SIWLM) scheme. The SIWLM consists of two stages: (1) Sequential weighted learning (SWL), which trains a model to learn entire tasks sequentially and concentrically, and (2) Intensive weighted learning (IWL), which enables the model to focus on the central task. We apply this scheme to the MT-DNN model and call this model the MTDNN-SIWLM. Our model achieves higher performance than the existing reference algorithms on six out of the eight GLUE benchmark tasks. Moreover, our model outperforms MT-DNN by 0.77 on average on the overall task. Finally, we conducted a thorough empirical investigation to determine the optimal weight for each GLUE task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Engineering, College of Engineering, Ewha Womans University, Seoul 03760, Korea
Welfare & Medical ICT Research Department, Electronics and Telecommunications Research Institute, Daejeon 34129, Korea
Division of Mechanical and Biomedical Engineering, Graduate Program in System Health Science and Engineering, College of Engineering, Ewha Womans University, Seoul 03760, Korea
Funding
National Research Foundation of Korea · Korea Medical Device Development Fund
References (38)
Show source-provided references
1. Jang, B., Kim, M., Harerimana, G., Kang, S.u., and Kim, J.W. (2020). Bi-LSTM model to increase accuracy in text classification: Combining Word2vec CNN and attention mechanism. Appl. Sci., 10.10.3390/app10175841 ↗
2. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv.10.18653/v1/W18-5446 ↗
3. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
4. Liu, X., He, P., Chen, W., and Gao, J. (2019). Multi-task deep neural networks for natural language understanding. arXiv.10.18653/v1/P19-1441 ↗
5. Clark, K., Luong, M.T., Khandelwal, U., Manning, C.D., and Le, Q.V. (2019). Bam! born-again multi-task networks for natural language understanding. arXiv.10.18653/v1/P19-1595 ↗
6. Internal-transfer weighting of multi-task learning for lung cancer detection
7. Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Zhao, T. (2019). Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv.10.18653/v1/2020.acl-main.197 ↗
8. Lee, C., Yang, K., Whang, T., Park, C., Matteson, A., and Lim, H. (2021). Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models. Appl. Sci., 11.10.3390/app11051974 ↗
9. Jwa, H., Oh, D., Park, K., Kang, J.M., and Lim, H. (2019). exBAKE: Automatic fake news detection model based on bidirectional encoder representations from transformers (bert). Appl. Sci., 9.10.3390/app9194062 ↗
10. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv.
11. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
12. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q.V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. arXiv.
13. Wang, W., Bi, B., Yan, M., Wu, C., Bao, Z., Xia, J., Peng, L., and Si, L. (2019). Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv.
14. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
15. He, P., Liu, X., Gao, J., and Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv.
16. Zhang, Y., and Yang, Q. (2017). A survey on multi-task learning. arXiv.10.1093/nsr/nwx105 ↗
17. Guo, H., Pasunuru, R., and Bansal, M. (2018). Soft layer-specific multi-task summarization with entailment and question generation. arXiv.10.18653/v1/P18-1064 ↗
18. Ruder, S., Bingel, J., Augenstein, I., and Søgaard, A. (February, January 27). Latent multi-task architecture learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
19. Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv.
20. Sun, Y., Wang, S., Li, Y.K., Feng, S., Tian, H., Wu, H., and Wang, H. (2020, January 7–12). ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6428 ↗
21. Bingel, J., and Søgaard, A. (2017). Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv.10.18653/v1/E17-2026 ↗
22. Sanh, V., Wolf, T., and Ruder, S. (February, January 27). A hierarchical multi-task approach for learning embeddings from semantic tasks. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
23. Wu, S., Zhang, H.R., and Ré, C. (2020). Understanding and Improving Information Transfer in Multi-Task Learning. arXiv.
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
25. Neural network acceptability judgments10.1162/tacl_a_00290 ↗
26. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., and Potts, C. (2013, January 18–21). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.
27. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv.10.18653/v1/S17-2001 ↗
28. Williams, A., Nangia, N., and Bowman, S.R. (2017). A broad-coverage challenge corpus for sentence understanding through inference. arXiv.10.18653/v1/N18-1101 ↗
29. Dagan, I., Glickman, O., and Magnini, B. (2005). The PASCAL recognising textual entailment challenge. Machine Learning Challenges Workshop, Springer.10.1007/11736790_9 ↗
30. Haim, R.B., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. (2006, January 10). The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venezia, Italy.
31. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. (2007, January 28–29). The third pascal recognizing textual entailment challenge. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic.10.3115/1654536.1654538 ↗
32. Bentivogli, L., Clark, P., Dagan, I., and Giampiccolo, D. (2009, January 16–17). The Fifth PASCAL Recognizing Textual Entailment Challenge. Proceedings of the Text Analysis Conference, Gaithersburg, MD, USA.
33. Dolan, W.B., and Brockett, C. (2005, January 14). Automatically constructing a corpus of sentential paraphrases. Proceedings of the Third International Workshop on Paraphrasing (IWP2005), Jeju Island, Korea.
34. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
35. Kochkina, E., Liakata, M., and Zubiaga, A. (2018). All-in-one: Multi-task learning for rumour verification. arXiv.
36. Sentiment and sarcasm classification with multitask learning10.1109/MIS.2019.2904691 ↗
37. Crichton, G., Pyysalo, S., Chiu, B., and Korhonen, A. (2017). A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinform., 18.10.1186/s12859-017-1776-8 ↗
38. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv.