Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction
Source abstract
Abstract
Automated story writing has been a subject of study for over 60 years. Today, large language models can generate narratively consistent and linguistically coherent short fiction texts. Despite these advancements, rigorous assessment of such outputs in terms of literary merit—especially concerning aesthetic qualities—has received scant attention. In this paper, we address the challenge of evaluating AI-generated microfiction (MF) and argue that this task requires consideration of literary criteria across various aspects of the text, including thematic coherence, textual clarity, interpretive depth, and aesthetic quality. To facilitate this, we present GrAImes: an evaluation protocol grounded in literary theory; specifically, GrAImes draws from a literary perspective to offer an objective framework for assessing AI-generated microfiction. Furthermore, we report the results of our validation of the evaluation protocol as answered by both literature experts and literary enthusiasts. This protocol will serve as a foundation for evaluating automatically generated microfiction and assessing its literary value.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Digital Humanities and Scholarship · Comparative Literary Analysis and Criticism · Latin American Literature Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, Carr. Lago de Guadalupe Km.3.5, Col. Margarita M. de Juarez, Atizapan 52926, Mexico
Laboratoire d’Informatique de Paris Nord, Centre National de la Recherche Scientifique, Université Sorbonne Paris Nord, 99 av. Jean-Baptiste Clément, 93430 Villetaneuse, France
Funding
ECOS NORD
References (61)
Show source-provided references
1. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., and Bi, X. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv.
2. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., and Carney, A. (2024). Openai o1 system card. arXiv.
3. Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., and Wang, S. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv.
4. Porter, B., and Machery, E. (2024). AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Sci. Rep., 14.10.1038/s41598-024-76900-1 ↗
5. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
6. Leslie, D., Ashurst, C., González, N.M., Griffiths, F., Jayadeva, S., Jorgensen, M., Katell, M., Krishna, S., Kwiatkowski, D., and Martins, C.I. (2024). ‘Frontier AI,’Power, and the Public Interest: Who benefits, who decides?. Harv. Data Sci. Rev.10.1162/99608f92.4a42495c ↗
7. Clark, E., Ji, Y., and Smith, N.A. (2018). Neural text generation in stories using entity representations as context. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics.10.18653/v1/N18-1204 ↗
8. Human heuristics for AI-generated language are flawed10.1073/pnas.2208839120 ↗
9. Automatic story generation: A survey of approaches
10. Iser, W. (1979). The act of reading: A theory of aesthetic response. J. Aesthet. Art Crit., 38.10.2307/430052 ↗
11. Ingarden, R. (1993). Concretización y reconstrucción. En Busca del Texto: Teoría de la Recepción Literaria, ENALTTA.
12. Ryan, J. (2017). Grimes’ Fairy Tales: A 1960s Story Generator, Springer International Publishing.10.1007/978-3-319-71027-3_8 ↗
13. Propp, V.Y. (2012). The Russian Folktale by Vladimir Yakovlevich Propp, Wayne State University Press.
14. Ginna, P. (2017). What Editors Do: The Art, Craft, and Business of Book Editing, University of Chicago Press.10.7208/chicago/9780226300030.001.0001 ↗
15. Evaluation of automatic generation of basic stories10.1007/BF03037336 ↗
16. Boden, M.A. (2004). The Creative Mind: Myths and Mechanisms, Routledge.10.4324/9780203508527 ↗
17. La minificción como clase textual transgenérica
18. Medina, Y.d.J.G. (2017). Microrrelato o minificción: De la nomenclatura a la estructura de un género literario. Microtextualidades. Rev. Int. Microrrelato Minificción, 89–102.10.31921/microtextualidades.n2a9 ↗
19. La función narrativa
20. Barthes, R., and Alcalde, R. (1990). La Aventura Semiológica, Paidós.
21. Shua, A.M. (2023). Cómo Escribir un Microrrelato, Siglo XXI Editores.
22. Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. arXiv.10.18653/v1/P18-1082 ↗
23. Guan, J., Mao, X., Fan, C., Liu, Z., Ding, W., and Huang, M. (2021). Long text generation by modeling sentence-level and discourse-level coherence. arXiv.10.18653/v1/2021.acl-long.499 ↗
24. Deep Learning-Based Short Story Generation for an Image Using the Encoder-Decoder Structure10.1109/ACCESS.2021.3104276 ↗
25. Lo, K.L., Ariss, R., and Kurz, P. (2022). GPoeT-2: A GPT-2 Based Poem Generator. arXiv.
26. Character-based interactive storytelling10.1109/MIS.2002.1024747 ↗
27. Story plot generation based on CBR10.1016/j.knosys.2004.10.011 ↗
28. Mori, Y., Yamane, H., Mukuta, Y., and Harada, T. (November, January 29). Toward a Better Story End: Collecting Human Evaluation with Reasons. Proceedings of the 12th International Conference on Natural Language Generation, Tokyo, Japan.
29. Rishes, E., Lukin, S.M., Elson, D.K., and Walker, M.A. (2013, January 6–9). Generating different story tellings from semantic representations of narrative. Proceedings of the International Conference on Interactive Digital Storytelling, Istanbul, Turkey.10.1007/978-3-319-02756-2_24 ↗
30. Elson, D.K., and McKeown, K.R. A Tool for Deep Semantic Encoding of Narrative Texts. Proceedings of the ACL-IJCNLP 2009 Software Demonstrations.10.3115/1667872.1667875 ↗
31. Sutskever, I., Martens, J., and Hinton, G.E. (July, January 28). Generating Text with Recurrent Neural Networks. Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA.
32. Kiddon, C., Zettlemoyer, L., and Choi, Y. (2016, January 1–5). Globally coherent text generation with neural checklist models. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1032 ↗
33. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015, January 7–13). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.11 ↗
34. Walker, M.A., Grant, R., Sawyer, J., Lin, G.I., Wardrip-Fruin, N., and Buell, M. (December, January 28). Perceived or not perceived: Film character models for expressive nlg. Proceedings of the International Conference on Interactive Digital Storytelling, Vancouver, BC, Canada.
35. Goodwin, R., and Sharp, O. (2025, June 08). Sunspring. YouTube Website. Available online: https://apps.dtic.mil/sti/citations/ADA460392.
36. Lukin, S.M., Reed, L.I., and Walker, M.A. (2017). Generating sentence planning variations for story telling. arXiv.
37. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
38. Language models are unsupervised multitask learners
39. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
40. Genette, G. (1976). Points. Figures I, Editions de Seuil.
41. Bal, M., and Van Boheemen, C. (2009). Narratology: Introduction to the Theory of Narrative, University of Toronto Press.
42. A statistical, grammar-based approach to microplanning10.1162/COLI_a_00273 ↗
43. Dorr, B., and Gaasterland, T. (2025, June 08). Summarization-Inspired Temporal-Relation Extraction: Tense-Pair Templates and Treebank-3 Analysis. Available online: https://apps.dtic.mil/sti/citations/ADA460392.
44. Zhu, J. (2012, January 15–17). Towards a Mixed Evaluation Approach for Computational Narrative Systems. Proceedings of the ICCC’12: 2012 1st IEEE International Conference on Communications in China, Beijing, China.
45. Koziev, I. (2025). Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry. arXiv.
46. Franceschelli, G., and Musolesi, M. (2025). Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generation. arXiv.
47. Ludmer, J. (2015). Clases 1985: Algunos Problemas de Teoría Literaria, Paidós.
48. Austin, J. (1962). How to Do Things with Words, Harvard University Press.
49. Bertochi, D. (1995). La aproximación al texto literario en la enseñanza obligatoria. Textos de Didáctica de la Lengua y la Literatura, Grao.
50. Huamán, M.Á. (2003). Educación y Literatura, Mantaro.
51. Lizaur Guerra, M.B.d. (2003). La telenovela mexicana: Forma y contenido de un formato narrativo de ficcíon de alcance mayoritario. [Ph.D. Thesis, Universidad Nacional Autónoma de México].
52. Thompson, J.B. (2013). Merchants of Culture: The Publishing Business in the Twenty-First Century, John Wiley & Sons.
53. Calasso, R. (2014). La Marca del Editor, Anagrama.
54. The logic of narrative possibilities10.2307/468934 ↗
55. Shklovsky, V. (1917). Art as technique. Literary Theory: An Anthology, Wiley-Blackwell.
56. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 2017: 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
57. Oñate Latorre, A., and Ortiz Fuentes, J. (2025, June 08). GPT2-Spanish. Available online: https://huggingface.co/DeepESP/gpt2-spanish.
58. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
59. From novice to expert: Implications of language skills and writing-relevant knowledge for memory during the development of writing skill10.17239/jowr-2011.03.01.3 ↗
60. Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.10.18653/v1/D19-1410 ↗
61. Rahutomo, F., Kitasuka, T., and Aritsugi, M. (2012, January 29–30). Semantic cosine similarity. Proceedings of the 7th International Student Conference on Advanced Science and Technology ICAST, Seoul, Republic of Korea.