Evaluation of the Coherence of Polish Texts Using Neural Network Models
Source abstract
Abstract
Coherence evaluation of texts falls into a category of natural language processing tasks. The evaluation of texts’ coherence implies the estimation of their semantic and logical integrity; such a feature of a text can be utilized during the solving of multidisciplinary tasks (SEO analysis, medicine area, detection of fake texts, etc.). In this paper, different state-of-the-art coherence evaluation methods based on machine learning models have been analyzed. The investigation of the effectiveness of different methods for the coherence estimation of Polish texts has been performed. The impact of text’s features on the output coherence value has been analyzed using different approaches of a semantic similarity graph. Two neural networks based on LSTM layers and a pre-trained BERT model correspondingly have been designed and trained for the coherence estimation of input texts. The results obtained may indicate that both lexical and semantic components should be taken into account during the coherence evaluation of Polish documents; moreover, it is advisable to analyze corresponding documents in a sentence-by-sentence manner taking into account word order. According to the retrieved accuracy of the proposed neural networks, it can be concluded that suggested models may be used in order to solve typical coherence estimation tasks for a Polish corpus.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Language and Culture · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Theoretical Electrical Engineering and Computer Science, Cracow University of Technology, Warszawska 24, 31-155 Cracow, Poland
Computer Engineering Department, Taras Shevchenko National University of Kyiv, 60 Volodymyrska Street, 01033 Kyiv, Ukraine
References (28)
Show source-provided references
1. Pishdad, L., Fancellu, F., Zhang, R., and Fazly, A. (2020, January 8–13). How coherent are neural models of coherence?. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.539 ↗
2. Xiong, H., He, Z., Wu, H., and Wang, H. (February, January 27). Modeling coherence for discourse neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
3. Pogorilyy, S., and Kramov, A. (2019, January 18–20). Coreference Resolution Method Using a Convolutional Neural Network. Proceedings of the 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT), Kyiv, Ukraine.10.1109/ATIT49449.2019.9030596 ↗
4. Evaluating text coherence based on the graph of the consistency of phrases to identify symptoms of schizophrenia
5. Modeling Local Coherence: An Entity-Based Approach10.1162/coli.2008.34.1.1 ↗
6. Walker, M.A., Joshi, A.K., and Prince, E.F. (1998). Centering Theory in Discourse, Claredon.10.1093/oso/9780198236870.001.0001 ↗
7. Guinaudeau, C., and Strube, M. (2013, January 4–9). Graph-based Local Coherence Modeling. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria.
8. Zhang, M., Feng, V.W., Qin, B., Hirst, G., Liu, T., and Huang, J. (June, January 31). Encoding World Knowledge in the Evaluation of Local Coherence. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, CO, USA.10.3115/v1/N15-1115 ↗
9. Feng, V.W., Lin, Z., and Hirst, G. (2014, January 23–29). The Impact of Deep Hierarchical Discourse Structures in the Evaluation of Text Coherence. Proceedings of the COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland.
10. Rhetorical Structure Theory: Toward a functional theory of text organization10.1515/text.1.1988.8.3.243 ↗
11. Le, Q., and Mikolov, T. (2014, January 22–24). Distributed Representations of Sentences and Documents. Proceedings of the 31st International Conference on International Conference on Machine Learning—Volume 32, JMLR.org, ICML’14, Bejing, China.
12. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
13. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
14. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
15. Li, J., and Hovy, E. (2014, January 25–29). A Model of Coherence Based on Distributed Sentence Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1218 ↗
16. Lai, A., and Tetreault, J. (2018, January 12–14). Discourse Coherence in the Wild: A Dataset, Evaluation and Methods. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia.10.18653/v1/W18-5023 ↗
17. Mesgar, M., and Strube, M. (November, January 31). A Neural Local Coherence Model for Text Quality Assessment. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1464 ↗
18. Moon, H.C., Mohiuddin, T., Joty, S., and Xu, C. (2019, January 3–7). A Unified Neural Coherence Model. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1231 ↗
19. Assessment of Text Coherence by Constructing the Graph of Semantic, Lexical, and Grammatical Consistancy of Phrases of Sentences10.1007/s10559-020-00309-7 ↗
20. Cui, B., Li, Y., Zhang, Y., and Zhang, Z. (2017, January 6–10). Text Coherence Analysis Based on Deep Neural Network. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore. CIKM ’17.10.1145/3132847.3133047 ↗
21. (2021, March 11). How to Make Your Copy More Readable: Make Sentences Shorter—PRsay. Available online: http://comprehension.prsa.org/?p=217.
22. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, U., and Polosukhin, I. (2017, January 4–9). Attention is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA. NIPS’17.
24. Pawlowski, A. (2017). ChronoPress—Chronological Corpus of Polish Press Texts (1945–1962), CLARIN ERIC Location. CLARIN2017 Book of Abstracts.
25. Fares, M., Kutuzov, A., Oepen, S., and Velldal, E. (2017, January 22–24). Word vectors, reuse, and replicability: Towards a community repository of large-text resources. Proceedings of the 21st Nordic Conference on Computational Linguistics, Gothenburg, Sweden.
26. Woliński, M. (2014, January 26–31). Morfeusz Reloaded. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
27. (2021, March 11). Polbert—Polish BERT. Available online: https://huggingface.co/dkleczek/bert-base-polish-uncased-v1.
28. (2021, March 11). Polish Coherence Models—Google Colab Notebook. Available online: https://colab.research.google.com/drive/1OMbYKmy9fYVtKMRTE-byQrivqXmooH8r?usp=sharing.