VPN: Variation on Prompt Tuning for Named-Entity Recognition
Source abstract
Abstract
Recently, prompt-based methods have achieved a promising performance in many natural language processing benchmarks. Despite success in sentence-level classification tasks, prompt-based methods work poorly in token-level tasks, such as named entity recognition (NER), due to the sophisticated design of entity-related templates. Note that the nature of prompt tuning makes full use of the parameters of the mask language model (MLM) head, while previous methods solely utilized the last hidden layer of language models (LMs) and the power of the MLM head is overlooked. In this work, we discovered the characteristics of semantic feature changes in samples after being processed using MLMs. Based on this characteristic, we designed a prompt-tuning variant for NER tasks. We let the pre-trained model predict the label words derived from the training dataset at each position and fed the generated logits (non-normalized probability) to the CRF layer. We evaluated our method on three popular datasets, and the experiments showed that our proposed method outperforms the state-of-the-art model in all three Chinese datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
PAII Inc., Palo Alto, CA 94306, USA
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen 518063, China
Pengcheng Laboratory, Shenzhen 518000, China
Funding
National Natural Science Foundation of China · Research Center for Computer Network (Shenzhen) Ministry of Education · Basic Research Fund of Shenzhen City · Major Key Project of PCL for Experiments and Applications · Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School
References (33)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
2. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
3. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
4. Ptr: Prompt tuning with rules for text classification10.1016/j.aiopen.2022.11.003 ↗
5. Schick, T., and Schütze, H. (2021, January 19–23). Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online.10.18653/v1/2021.eacl-main.20 ↗
6. Tam, D., Menon, R.R., Bansal, M., Srivastava, S., and Raffel, C. (2021, January 7–11). Improving and Simplifying Pattern Exploiting Training. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.407 ↗
7. True few-shot learning with language models
8. Lester, B., Al-Rfou, R., and Constant, N. (2021, January 7–11). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.243 ↗
9. Qin, G., and Eisner, J. (2021, January 6–11). Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Online.10.18653/v1/2021.naacl-main.410 ↗
10. Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. (2021). GPT Understands, Too. arXiv.
11. Zhong, Z., Friedman, D., and Chen, D. (2021). Factual Probing Is [MASK]: Learning vs. Learning to Recall. arXiv.10.18653/v1/2021.naacl-main.398 ↗
12. Cui, L., Wu, Y., Liu, J., Yang, S., and Zhang, Y. (2021, January 1–6). Template-Based Named Entity Recognition Using BART. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online Event.10.18653/v1/2021.findings-acl.161 ↗
13. Ma, R., Zhou, X., Gui, T., Tan, Y., Li, L., Zhang, Q., and Huang, X.J. (2022, January 10–15). Template-free Prompt Tuning for Few-shot NER. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online and Seattle, WA, USA.10.18653/v1/2022.naacl-main.420 ↗
14. Language models are few-shot learners
15. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. (2019, January 3–7). Language Models as Knowledge Bases?. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1250 ↗
16. Davison, J., Feldman, J., and Rush, A.M. (2019, January 3–7). Commonsense Knowledge Mining from Pretrained Models. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1109 ↗
17. Trinh, T.H., and Le, Q.V. (2018). A Simple Method for Commonsense Reasoning. arXiv.
18. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking
19. Schick, T., Schmid, H., and Schütze, H. (2020, January 8–13). Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.488 ↗
20. Gao, T., Fisch, A., and Chen, D. (2021, January 1–6). Making Pre-trained Language Models Better Few-shot Learners. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.10.18653/v1/2021.acl-long.295 ↗
21. Li, X.L., and Liang, P. (2021, January 1–6). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.10.18653/v1/2021.acl-long.353 ↗
22. Weischedel, R., Pradhan, S., Ramshaw, L., Palmer, M., Xue, N., Marcus, M., Taylor, A., Greenberg, C., Hovy, E., and Belvin, R. (2011). Ontonotes Release 4.0, Linguistic Data Consortium. LDC2011T03.
23. Che, W., Wang, M., Manning, C.D., and Liu, T. (2013, January 9–15). Named entity recognition with bilingual constraints. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia.
24. Zhang, S., Qin, Y., Hou, W.J., and Wang, X. (2006, January 22–23). Word segmentation and named entity recognition for sighan bakeoff3. Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, Sydney, Australia.
25. Peng, N., and Dredze, M. (2015, January 17–21). Named entity recognition for chinese social media with jointly trained embeddings. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1064 ↗
26. Lafferty, J., McCallum, A., and Pereira, F.C. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proceedings of the Eighteenth International Conference on Machine Learning.
27. Meng, Y., Wu, W., Wang, F., Li, X., Nie, P., Yin, F., Li, M., Han, Q., Sun, X., and Li, J. (2019, January 8–14). Glyce: Glyph-vectors for chinese character representations. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
28. Li, X., Yan, H., Qiu, X., and Huang, X.J. (2020, January 5–10). FLAT: Chinese NER Using Flat-Lattice Transformer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.611 ↗
29. Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., and Li, J. (2020, January 5–10). A Unified MRC Framework for Named Entity Recognition. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.519 ↗
30. Zhang, X., Zhao, J., and LeCun, Y. (2015, January 7–12). Character-level convolutional networks for text classification. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
31. Sang, E.F., and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. arXiv.
32. Pradhan, S., Moschitti, A., Xue, N., Ng, H.T., Björkelund, A., Uryupina, O., Zhang, Y., and Zhong, Z. (2013, January 4–9). Towards robust linguistic analysis using ontonotes. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, Sofia, Bulgaria.
33. Scikit-learn: Machine Learning in Python