Contrastive Learning Penalized Cross-Entropy with Diversity Contrastive Search Decoding for Diagnostic Report Generation of Reduced Token Repetition
Source abstract
Abstract
Medical imaging description and disease diagnosis are vitally important yet time-consuming. Automated diagnosis report generation (DRG) from medical imaging description can reduce clinicians’ workload and improve their routine efficiency. To address this natural language generation task, fine-tuning a pre-trained large language model (LLM) is cost-effective and indispensable, and its success has been witnessed in many downstream applications. However, semantic inconsistency of sentence embeddings has been massively observed from undesirable repetitions or unnaturalness in text generation. To address the underlying issue of anisotropic distribution of token representation, in this study, a contrastive learning penalized cross-entropy (CLpCE) objective function is implemented to enhance the semantic consistency and accuracy of token representation by guiding the fine-tuning procedure towards a specific task. Furthermore, to improve the diversity of token generation in text summarization and to prevent sampling from unreliable tail of token distributions, a diversity contrastive search (DCS) decoding method is designed for restricting the report generation derived from a probable candidate set with maintained semantic coherence. Furthermore, a novel metric named the maximum of token repetition ratio (maxTRR) is proposed to estimate the token diversity and to help determine the candidate output. Based on the LLM of a generative pre-trained Transformer 2 (GPT-2) of Chinese version, the proposed CLpCE with DCS (CLpCEwDCS) decoding framework is validated on 30,000 desensitized text samples from the “Medical Imaging Diagnosis Report Generation” track of 2023 Global Artificial Intelligence Technology Innovation Competition. Using four kinds of metrics evaluated from n-gram word matching, semantic relevance, and content similarity as well as the maxTRR metric extensive experiments reveal that the proposed framework effectively maintains semantic coherence and accuracy (BLEU-1, 0.4937; BLEU-2, 0.4107; BLEU-3, 0.3461; BLEU-4, 0.2933; METEOR, 0.2612; ROUGE, 0.5182; CIDER, 1.4339) and improves text generation diversity and naturalness (maxTRR, 0.12). The phenomenon of dull or repetitive text generation is common when fine-tuning pre-trained LLMs for natural language processing applications. This study might shed some light on relieving this issue by developing comprehensive strategies to enhance semantic coherence, accuracy and diversity of sentence embeddings.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Handwritten Text Recognition Techniques · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
Funding
National Key R&D Program of China · Fundamental Research Funds for the Central Universities
References (58)
Show source-provided references
1. Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C., and Socher, R. (2019). Neural text summarization: A critical evaluation. arXiv.10.18653/v1/D19-1051 ↗
2. Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E.D., Gutierrez, J.B., and Kochut, K. (2017). Text summarization techniques: A brief survey. arXiv.10.14569/IJACSA.2017.081052 ↗
3. Pang, T., Li, P., and Zhao, L. (2022). A survey on automatic generation of medical imaging reports based on deep learning. Biomed. Eng. Online, 22.10.1186/s12938-023-01113-y ↗
4. Chen, Z., Varma, M., Delbrouck, J., Paschali, M., Blankemeier, L., Van Veen, D., Valanarasu, J., Youssef, A., Cohen, J., and Reis, E. (2024). CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation. arXiv.
5. Automatic summarising: The state of the art10.1016/j.ipm.2007.03.009 ↗
6. Deep learning–based text classification: A comprehensive review10.1145/3439726 ↗
7. Bidirectional recurrent neural networks10.1109/78.650093 ↗
8. A review on the long short-term memory model10.1007/s10462-020-09838-1 ↗
9. Attention is all you need
10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv.
11. Paulus, R., Xiong, C., and Socher, R. (2017). A deep reinforced model for abstractive summarization. arXiv.
12. Chuang, Y., Tang, R., Jiang, X., and Hu, X. (2024). SPeC: A soft prompt-based calibration on performance variability of large language model in clinical notes summarization. J. Biomed. Inform., 151.10.1016/j.jbi.2024.104606 ↗
13. Tian, S., Jin, Q., Yeganova, L., Lai, P., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., and Comeau, D. (2024). Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings Bioinform., 25.10.1093/bib/bbad493 ↗
14. Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv.
15. Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E., and Seehofnerová, A. (2024). Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med.10.1038/s41591-024-02855-5 ↗
16. Dong, Y., Cordonnier, J.-B., and Loukas, A. (2021, January 18–24). Attention is not all you need: Pure attention loses rank doubly exponentially with depth. Proceedings of the 38th International Conference on Machine Learning, Virtual Event.
17. Ethayarajh, K. (2019). How contextual are contextualized word representations? comparing the geometry of BERT, ELMO, and GPT-2 embeddings. arXiv.10.18653/v1/D19-1006 ↗
18. Su, Y., Liu, F., Meng, Z., Lan, T., Shu, L., Shareghi, E., and Collier, N. (2021). Tacl: Improving bert pre-training with token-aware contrastive learning. arXiv.10.18653/v1/2022.findings-naacl.191 ↗
19. A contrastive framework for neural text generation
20. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. arXiv.10.18653/v1/2020.emnlp-main.733 ↗
21. Wang, Z., Zeng, J., Tao, H., and Zhong, L. (2023, January 18–23). RBPSum: An extractive summarization approach using Bi-stream attention and position residual connection. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.10.1109/IJCNN54540.2023.10191105 ↗
22. Abanoub, G.E., Fawzy, A.M., Waly, R.R., and Gomaa, W.H. (2023). Generate descriptions of medical dialogues through two-layers Transformer-based summarization. Intell. Method Syst. Appl., 32–37.10.1109/IMSA58542.2023.10217636 ↗
23. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
24. Exploring the limits of transfer learning with a unified text-to-text transformer
25. Debiased contrastive learning
26. Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. (2019). Neural text generation with unlikelihood training. arXiv.
27. Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., and Ma, H. (2020). CLEAR: Contrastive learning for sentence representation. arXiv.
28. Tan, C., and Sun, X. (2023, January 18–23). CoLRP: A contrastive learning abstractive text summarization method with ROUGE penalty. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.10.1109/IJCNN54540.2023.10191344 ↗
29. Mai, T.P., Nguyen, Q.A., Can, D.C., and Le, H.Q. (2023, January 18–20). Contrastive hierarchical discourse graph for vietnamese extractive multi-document summarization. Proceedings of the 2023 International Conference on Asian Language Processing (IALP), Singapore.10.1109/IALP61005.2023.10337087 ↗
30. OpenNMT: Open-Source Toolkit for Neural Machine Translation
31. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The curious case of neural text degeneration. arXiv.
32. A theoretical analysis of the repetition problem in text generation
33. Su, Y., and Xu, J. (2022). An empirical study on contrastive search and contrastive decoding for open-ended text generation. arXiv.
34. Chen, W.L., Wu, C.K., Chen, H.H., and Chen, C.C. (2023). Fidelity-enriched contrastive search: Reconciling the faithfulness-diversity trade-off in text generation. arXiv.10.18653/v1/2023.emnlp-main.54 ↗
35. Generic Mechanism for Reducing Repetitions in Encoder-Decoder Models10.5715/jnlp.30.401 ↗
36. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation
37. Dimensionality reduction by learning an invariant mapping
38. A simple framework for contrastive learning of visual representations
39. Du, Z. (2019). GPT2-Chinese: Tools for Training GPT2 Model in Chinese Language, GitHub Repository.
40. Shao, Y., Geng, Z., Liu, Y., Dai, J., Yan, H., Yang, F., Zhe, L., Bao, H., and Qiu, X. (2021). CPT: A pre-trained unbalanced transformer for both chinese language understanding and generation. arXiv.
41. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 technical report. arXiv.
42. Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. arXiv.10.18653/v1/P18-1082 ↗
43. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 6–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
44. Banerjee, S., and Lavie, A. (2005). Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics.
45. Lin, C.-Y. (2024, March 19). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004; pp. 74–81. Available online: https://aclanthology.org/W04-1013.pdf.
46. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
47. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
48. R-drop: Regularized dropout for neural networks
49. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. (2018). Averaging weights leads to wider optima and better generalization. arXiv.
50. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
51. Wu, X., Gao, Y., Zhang, H., Yang, Y., Guo, W., and Lu, J. (2023). The Solution for the CVPR2023 NICE Image Captioning Challenge. arXiv.
52. Retrieval-augmented generation for knowledge-intensive nlp tasks
53. Language models are unsupervised multitask learners
54. Language models are few-shot learners
55. Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. (2022). GLM: General language model pretraining with autoregressive blank infilling. arXiv.10.18653/v1/2022.acl-long.26 ↗
56. Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. (2022, January 17–23). Data2vec: A general framework for self-supervised learning in speech, vision and language. Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA.
57. Survey on reinforcement learning for language processing10.1007/s10462-022-10205-5 ↗
58. Training language models to follow instructions with human feedback