A Study of Contrastive Learning Algorithms for Sentence Representation Based on Simple Data Augmentation
Source abstract
Abstract
In the era of deep learning, representational text-matching algorithms based on BERT and its variant models have become mainstream and are limited by the sentence vectors generated by the BERT model, and the SimCSE algorithm proposed in 2021 has improved the sentence vector quality to a certain extent. In this paper, to address the problem that the SimCSE algorithm has—that the greater the difference in sentence length, the smaller the probability that the sentence pairs are similar—an EdaCSE algorithm is proposed to perturb the sentence length using a simple data enhancement method without affecting the semantics of the sentences. The perturbation is applied to the sentence length by adding meaningless English punctuation marks to the original sentence so that the model no longer tends to recognise sentences of similar length as similar sentences. Based on the BERT series of models, experiments were conducted on five different datasets, and the experiments proved that the EdaCSE method improves an average of 1.67, 0.84, and 1.08 on the five datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Domain Adaptation and Few-Shot Learning · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, China University of Geosciences (Wuhan), Wuhan 430079, China
Funding
research on adaptive integrated evolutionary algorithm for multi-root solution of complex nonlinear equations
References (21)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
2. Gao, T., Yao, X., and Chen, D. (2021, January 7–11). SimCSE: Simple Contrastive Learning of Sentence Embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Association for Computational Linguistics (ACL), Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.552 ↗
3. Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., and Xu, W. (2021, January 1–6). ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual Event.10.18653/v1/2021.acl-long.393 ↗
4. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
5. Wu, X., Gao, C., Zang, L., Han, J., Wang, Z., and Hu, S. (2022, January 12–17). ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea.
6. Zhang, Y., Zhu, H., Wang, Y., Xu, N., Li, X., and Zhao, B. (2022, January 22–27). A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.336 ↗
7. Jiang, T., Jiao, J., Huang, S., Zhang, Z., Wang, D., Zhuang, F., Wei, F., Huang, H., Deng, D., and Zhang, Q. (2022). Promptbert: Improving bert sentence embeddings with prompts. arXiv.10.18653/v1/2022.emnlp-main.603 ↗
8. Klein, T., and Nabi, M. (2022, January 22–27). SCD: Self-Contrastive Decorrelation of Sentence Embeddings. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland.10.18653/v1/2022.acl-short.44 ↗
9. Chuang, Y.-S., Dangovski, R., Luo, H., Zhang, Y., Chang, S., Soljacic, M., Li, S.-W., Yih, S., Kim, Y., and Glass, J. (2022, January 10–15). DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.311 ↗
10. Wang, H., Li, Y., Huang, Z., Dou, Y., Kong, L., and Shao, J. (2022). SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples. arXiv.10.1007/978-981-99-4752-2_35 ↗
11. Nishikawa, S., Ri, R., Yamada, I., Tsuruoka, Y., and Echizen, I. (2022, January 10–15). EASE: Entity-Aware Contrastive Learning of Sentence Embedding. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.284 ↗
12. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
13. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
14. Miyato, T., Dai, A.M., and Goodfellow, I. (2016). Adversarial training methods for semi-supervised text classification. arXiv.
15. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
16. Chen, J., Chen, Q., Liu, X., Yang, H., Lu, D., and Tang, B. (November, January 31). The BQ Corpus: A Large-scale Domain-specific Chinese Corpus for Sentence Semantic Equivalence Identification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP2018, Brussels, Belgium.10.18653/v1/D18-1536 ↗
17. Liu, X., Chen, Q., Deng, C., Zeng, H., Chen, J., Li, D., and Tang, B. (2018, January 20–26). Lcqmc: A large-scale chinese question matching corpus. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA.
18. Yang, Y., Zhang, Y., Tar, C., and Baldridge, J. (2019). PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. arXiv.10.18653/v1/D19-1382 ↗
19. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv.10.18653/v1/S17-2001 ↗
20. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
21. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗