CGM: Copy Mechanism GPT with Mask for Ellipsis and Anaphora Resolution in Dialogue
Source abstract
Abstract
GPT (Generative Pre-trained Transformer) is a generative language model that demonstrates outstanding performance in the field of text generation. Generally, the attention mechanism of the transformer model behaves similarly to a copy distribution. However, due to the absence of a dedicated encoder, it is challenging to ensure that the input is retained for generation. We propose a model that emphasizes the copy mechanism in GPT. We generate masks for the input words to initialize the distribution and explicitly encourage copying through training. To demonstrate the effectiveness of our approach, we conducted experiments to restore ellipsis and anaphora in dialogue. In a single domain, we achieved 0.4319 (BLEU), 0.6408 (Rouge-L), 0.9040 (simCSE), and 0.9070 (BERTScore), while in multi-domain settings we obtained 0.4611 (BLEU), 0.6379 (Rouge-L), 0.8902 (simCSE), and 0.8999 (BERTScore). Additionally, we evaluated the operation of the copy mechanism on out-of-domain data, yielding excellent results. We anticipate that applying the copy mechanism to GPT will be useful for utilizing language models in constrained situations.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Changwon National University, Changwon 51140, Republic of Korea
Funding
Korea government (MSIT)
References (24)
Show source-provided references
1. Xu, S., Li, H., Yuan, P., Wu, Y., He, X., and Zhou, B. (2020, January 5–10). Self-Attention Guided Copy Mechanism for Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.125 ↗
2. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
3. González, J.L.V., and Rodríguez, A.F. (2000, January 3–6). Importance of Pronominal Anaphora Resolution in Question Answering Systems. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Hong Kong, China.
4. Aralikatte, R., Lamm, M., Hardt, D., and Søgaard, A. (2019). Ellipsis and Coreference Resolution as Question Answering. arXiv.
5. Aralikatte, R., Lamm, M., Hardt, D., and Søgaard, A. (2021, January 21–23). Ellipsis Resolution as Question Answering: An Evaluation. Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, Kiev, Ukraine.10.18653/v1/2021.eacl-main.68 ↗
6. Korean Coreference Resolution with Guided Mention Pair Model Using Deep Learning10.4218/etrij.16.0115.0896 ↗
7. Conversational question answering: A survey10.1007/s10115-022-01744-y ↗
8. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.T., Choi, Y., Liang, P., and Zettlemoyer, L. (November, January 31). QuAC: Question Answering in Context. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1241 ↗
9. Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W.B., and Iyyer, M. (2019, January 3–7). Attentive History Selection for Conversational Question Answering. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.10.1145/3357384.3357905 ↗
10. Qiu, M., Huang, X., Chen, C., Ji, F., Qu, C., Wei, W., Huang, J., and Zhang, Y. (2021, January 2–9). Reinforced History Backtracking for Conversational Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i15.17617 ↗
11. Mutal, J., Gerlach, J., Bouillon, P., and Spechbach, H. (2020, January 4–6). Ellipsis Translation for a Medical Speech to Speech Translation System. Proceedings of the European Association for Machine Translation Conferences/Workshops, Lisbon, Portugal.
12. Quan, J., Xiong, D., Webber, B.L., and Hu, C. (2019, January 3–7). GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1462 ↗
13. SpanBERT: Improving Pre-training by Representing and Predicting Spans10.1162/tacl_a_00300 ↗
14. A brief survey on recent advances in coreference resolution10.1007/s10462-023-10506-3 ↗
15. Gülçehre, C., Ahn, S., Nallapati, R., Zhou, B., and Bengio, Y. (2016). Pointing the Unknown Words. arXiv.10.18653/v1/P16-1014 ↗
16. Gu, J., Lu, Z., Li, H., and Li, V.O.K. (2016). Incorporating Copying Mechanism in Sequence-to-Sequence Learning. arXiv.10.18653/v1/P16-1154 ↗
17. Supervised Copy Mechanism for Grammatical Error Correction10.1109/ACCESS.2023.3294979 ↗
18. See, A., Liu, P.J., and Manning, C.D. (2017). Get To The Point: Summarization with Pointer-Generator Networks. arXiv.10.18653/v1/P17-1099 ↗
19. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. CoRR.
20. Language Models are Unsupervised Multitask Learners
21. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
22. Lin, C.Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, Association for Computational Linguistics.
23. Gao, T., Yao, X., and Chen, D. (2021, January 7–11). SimCSE: Simple Contrastive Learning of Sentence Embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.552 ↗
24. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). BERTScore: Evaluating Text Generation with BERT. arXiv.