Toward a Model to Evaluate Machine-Processing Quality in Scientific Documentation and Its Impact on Information Retrieval
Source abstract
Abstract
The lack of quality in scientific documents affects how documents can be retrieved depending on a user query. Existing search tools for scientific documentation usually retrieve a vast number of documents, of which only a small fraction proves relevant to the user’s query. However, these documents do not always appear at the top of the retrieval process output. This is mainly due to the substantial volume of continuously generated information, which complicates the search and access not properly considering all metadata and content. Regarding document content, the way in which the author structures it and the way the user formulates the query can lead to linguistic differences, potentially resulting in issues of ambiguity between the vocabulary employed by authors and users. In this context, our research aims to address the challenge of evaluating the machine-processing quality of scientific documentation and measure its influence on the processes of indexing and information retrieval. To achieve this objective, we propose a set of indicators and metrics for the construction of the evaluation model. This set of quality indicators have been grouped into three main areas based on the principles of Open Science: accessibility, content, and reproducibility. In this sense, quality is defined as the value that determines whether a document meets the requirements to be retrieved successfully. To prioritize the different indicators, a hierarchical analysis process (AHP) has been carried out with the participation of three referees, obtaining as a result a set of nine weighted indicators. Furthermore, a method to implement the quality model has been designed to support the automatic evaluation of quality and perform the indexing and retrieval process. The impact of quality in the retrieval process has been validated through a case study comprising 120 scientific documents from the field of the computer science discipline and 25 queries, obtaining as a result 21% high, 39% low, and 40% moderate quality.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Topic Modeling · Scientific Computing and Data Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Engineering Department, Carlos III University of Madrid, 28911 Leganés, Spain
Engineering Departament, Libre University, Barranquilla 08002, Colombia
References (53)
Show source-provided references
1. Zhang, X., Li, X., Jiang, S., Li, X., and Xie, B. (2019, January 18–20). Evolution Analysis of Information Retrieval based on co-word network. Proceedings of the 2019 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE), Xiamen, China.10.1109/EITCE47263.2019.9094904 ↗
2. Fuzzy retrieval algorithm for film and television animation resource database based on deep neural network
3. Efficient and secure content-based image retrieval with deep neural networks in the mobile cloud computing10.1016/j.cose.2023.103163 ↗
4. Transformer based contextual text representation framework for intelligent information retrieval10.1016/j.eswa.2023.121629 ↗
5. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models 2021. arXiv.
6. Koga, S., Martin, N.B., and Dickson, D.W. (2023). Evaluating the performance of large language models: ChatGPT and Google Bard in generating differential diagnoses in clinicopathological conferences of neurodegenerative disorders. Brain Pathol., e13207.10.1111/bpa.13207 ↗
7. Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. (2023). LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv.
8. Touvron, H., Lavril, T., Izacard, G., and Martinet, X. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
9. Sánchez-Ruiz, L.M., Moll-López, S., Nuñez-Pérez, A., Moraño-Fernández, J.A., and Vega-Fleitas, E. (2023). ChatGPT Challenges Blended Learning Methodologies in Engineering Education: A Case Study in Mathematics. Appl. Sci., 13.10.3390/app13106039 ↗
10. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the Opportunities and Risks of Foundation Models. arXiv.
11. Optimizing Selection of Assessment Solutions for Completing Information Extraction Results
12. An Ontological Framework for Information Extraction from Diverse Scientific Sources10.1109/ACCESS.2021.3063181 ↗
13. Rocha, Á., Ferrás, C., Montenegro Marin, C.E., and Medina García, V.H. (2020, January 5–7). Quality in Documentation: Key Factor for the Retrieval Process. Proceedings of the Information Technology and Systems, Bogota, Colombia.10.1007/978-3-030-40690-5 ↗
14. Modelo computacional para el desarrollo de sistemas de recuperación de información
15. Tamrakar, A., and Vishwakarma, S.K. (2015, January 12–14). Analysis of Probabilistic Model for Document Retrieval in Information Retrieval. Proceedings of the 2015 International Conference on Computational Intelligence and Communication Networks (CICN), Jabalpur, India.10.1109/CICN.2015.155 ↗
16. Li, X., Li, K., Qiao, D., Ding, Y., and Wei, D. (2019, January 5–7). Application Research of Machine Learning Method Based on Distributed Cluster in Information Retrieval. Proceedings of the 2019 International Conference on Communications, Information System and Computer Engineering (CISCE), Haikou, China.10.1109/CISCE.2019.00097 ↗
17. Taylor, S.J.E., Anagnostou, A., Fabiyi, A., Currie, C., Monks, T., Barbera, R., and Becker, B. (2017, January 3–6). Open science: Approaches and benefits for modeling & simulation. Proceedings of the 2017 Winter Simulation Conference (WSC), Las Vegas, NV, USA.10.1109/WSC.2017.8247813 ↗
18. Sidi, M.L., and Gunal, S. (2023). A Purely Entity-Based Semantic Search Approach for Document Retrieval. Appl. Sci., 13.10.20944/preprints202308.1279.v1 ↗
19. Nagumothu, D., Eklund, P.W., Ofoghi, B., and Bouadjenek, M.R. (2021). Linked Data Triples Enhance Document Relevance Classification. Appl. Sci., 11.10.3390/app11146636 ↗
20. Frihat, S., Beckmann, C.L., Hartmann, E.M., and Fuhr, N. (2023). Document Difficulty Aspects for Medical Practitioners: Enhancing Information Retrieval in Personalized Search Engines. Appl. Sci., 13.10.3390/app131910612 ↗
21. Al Sibahee, M.A., Abdulsada, A.I., Abduljabbar, Z.A., Ma, J., Nyangaresi, V.O., and Umran, S.M. (2021). Lightweight, Secure, Similar-Document Retrieval over Encrypted Data. Appl. Sci., 11.10.3390/app112412040 ↗
22. Yeshambel, T., Mothe, J., and Assabie, Y. (2022). Amharic Adhoc Information Retrieval System Based on Morphological Features. Appl. Sci., 12.10.3390/app12031294 ↗
23. Why is a document relevant? Understanding the relevance scores in cross-lingual document retrieval10.1016/j.knosys.2022.108545 ↗
24. Information retrieval from scientific abstract and citation databases: A query-by-documents approach based on Monte-Carlo sampling10.1016/j.eswa.2022.116967 ↗
25. Ciencia Abierta: Cómo han evolucionado la denominación y el concepto
26. Hasselbring, W., Carr, L., Hettrick, S., Packer, H., and Tiropanis, T. (2019). FAIR and Open Computer Science Research Software. arXiv.10.1109/MC.2020.2998235 ↗
27. Bezjak, S., Clyburne-Sherin, A., Conzett, P., Fernandes, P., Görögh, E., Helbig, K., Kramer, B., Labastida, I., Niemeyer, K., and Psomopoulos, F. (2018). Open Science Training Handbook, Zenodo.
28. Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud
29. (2021, July 28). FAIR-Aware Online Assessment Tool. Available online: https://fairaware.dans.knaw.nl.
30. (2020). FAIRsFAIR Data Object Assessment Metrics: Request for Comments, FAIRsFAIR.
31. DG for Research and Innovation (2020). Reproducibility of Scientific Results in the EU Scoping Report, DG for Research and Innovation.
32. Echtler, F., and Häußler, M. (2018, January 21–26). Open Source, Open Science, and the Replication Crisis in HCI. Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3170427.3188395 ↗
33. From FAIR research data toward FAIR and open research software
34. A manifesto for reproducible science10.1038/s41562-016-0021 ↗
35. Reproducibility and replicability of systematic reviews10.13105/wjma.v7.i3.66 ↗
36. Sivagnanam, S., Nandigam, V., and Lin, K. (August, January 28). Introducing the Open Science Chain: Protecting Integrity and Provenance of Research Data. Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning), Chicago, IL, USA.
37. Reproducibility and replicability: Opportunities and challenges for geospatial research10.1080/13658816.2020.1802032 ↗
38. Sistemas de recuperación de información implementados a partir de CORD-19: Herramientas clave en la gestión de la información sobre COVID-1910.3989/redc.2020.4.1794 ↗
39. TREC-COVID: Rationale and structure of an information retrieval shared task for COVID-1910.1093/jamia/ocaa091 ↗
40. AHP in nonlinear scaling: From two-envelope problem to modeling by predictors10.1590/0103-6513.20210007 ↗
41. Metodología para la toma de decisiones de inversión en portafolio de acciones utilizando la técnica multicriterio AHP10.1016/S0186-1042(15)30004-8 ↗
42. Ciência Aberta na América Latina: Duas perspectivas em disputa10.1590/238180889201931e190028 ↗
43. Importancia de las revistas de acceso abierto: La indización como meta fundamental10.18041/2619-4244/dl.26.6184 ↗
44. Vainshtein, R., Katz, G., Shapira, B., and Rokach, L. (2019). Assessing the Quality of Scientific Papers. arXiv.
45. (2021, July 27). 5.22. File Location (Ubicación del Archivo) (MA)—Documentación de Directrices Para Repositorios Institucionales de Investigación de la Red Colombiana de Información Científica (RedCol) 2020—1.0. Available online: https://redcol.readthedocs.io/es/latest/field_filelocation.html#aire-file.
46. (2021, July 27). Metadata in Science Publishing. Available online: http://wwwis.win.tue.nl/infwet03/proceedings/8/.
47. Legibilidad del texto, métricas de complejidad y la importancia de las palabras
48. Legibilidad Y Variabilidad de los Textos
49. Implementation of an Online Software Tool for the Analytic Hierarchy Process (AHP-OS)
50. Application of the Analytical Hierarchy Process (AHP) for decision-making with expert judgment10.4067/S0718-33052019000300348 ↗
51. Shah, N., Willick, D., and Mago, V. (2022). A framework for social media data analytics using Elasticsearch and Kibana|SpringerLink. Red Ina., 1179–1187.10.1007/s11276-018-01896-2 ↗
52. (2023, May 21). Metadata 2020 Principles. Available online: https://metadata2020.org/resources/metadata-principles/.
53. Yang, A., Zhu, S., Li, X., Yu, J., Wei, M., and Li, C. (2018, January 26–28). The research of policy big data retrieval and analysis based on elastic search. Proceedings of the 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), Chengdu, China.10.1109/ICAIBD.2018.8396164 ↗