A FAIR Resource Recommender System for Smart Open Scientific Inquiries
Source abstract
Abstract
A vast proportion of scientific data remains locked behind dynamic web interfaces, often called the deep web—inaccessible to conventional search engines and standard crawlers. This gap between data availability and machine usability hampers the goals of open science and automation. While registries like FAIRsharing offer structured metadata describing data standards, repositories, and policies aligned with the FAIR (Findable, Accessible, Interoperable, and Reusable) principles, they do not enable seamless, programmatic access to the underlying datasets. We present FAIRFind, a system designed to bridge this accessibility gap. FAIRFind autonomously discovers, interprets, and operationalizes access paths to biological databases on the deep web, regardless of their FAIR compliance. Central to our approach is the Deep Web Communication Protocol (DWCP), a resource description language that represents web forms, HyperText Markup Language (HTML) tables, and file-based data interfaces in a machine-actionable format. Leveraging large language models (LLMs), FAIRFind combines a specialized deep web crawler and web-form comprehension engine to transform passive web metadata into executable workflows. By indexing and embedding these workflows, FAIRFind enables natural language querying over diverse biological data sources and returns structured, source-resolved results. Evaluation across multiple open-source LLMs and database types demonstrates over 90% success in structured data extraction and high semantic retrieval accuracy. FAIRFind advances existing registries by turning linked resources from static references into actionable endpoints, laying a foundation for intelligent, autonomous data discovery across scientific domains.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Scientific Computing and Data Management · Research Data Management Practices · Genomics and Phylogenetic Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, University of Idaho, Moscow, ID 83844, USA
Funding
National Institutes of Health IDeA · National Science Foundation CSSI · US Department of Energy
References (48)
Show source-provided references
1. The FAIR Guiding Principles for scientific data management and stewardship10.1038/sdata.2016.18 ↗
2. FAIRsharing as a community approach to standards, repositories and policies10.1038/s41587-019-0080-8 ↗
3. The compliance to FAIR principles of shared data in addiction research10.1007/s11192-024-05227-5 ↗
4. Sofi-Mahmudi, A., Raittio, E., Khazaei, Y., Ashraf, J., Schwendicke, F., Uribe, S.E., and Moher, D. (2024). COVID-19-related research data availability and quality according to the FAIR principles: A meta-research study. PLoS ONE, 19.10.1371/journal.pone.0313991 ↗
5. Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A.U., Wu, L., Read, E., Manoff, M., and Frame, M. (2011). Data sharing by scientists: Practices and perceptions. PLoS ONE, 6.10.1371/journal.pone.0021101 ↗
6. A review of the heterogeneous landscape of biodiversity databases: Opportunities and challenges for a synthesized biodiversity knowledge base10.1111/geb.13497 ↗
7. UniProt: A worldwide hub of protein knowledge10.1093/nar/gky1049 ↗
8. The human plasma proteome: History, character, and diagnostic prospects10.1074/mcp.R200007-MCP200 ↗
9. Human protein reference database—2009 update10.1093/nar/gkn892 ↗
10. DisGeNET: A comprehensive platform integrating information on human disease-associated genes and variants
11. Comparative toxicogenomics database (CTD): Update 202310.1093/nar/gkac833 ↗
12. The reactome pathway knowledgebase10.1093/nar/gkx1132 ↗
13. The 27th annual Nucleic Acids Research database issue and molecular biology database collection10.1093/nar/gkz1161 ↗
14. Pampel, H., Vierkant, P., Scholze, F., Bertelmann, R., Kindling, M., Klump, J., Goebelbecker, H.J., Gundlach, J., Schirmbacher, P., and Dierolf, U. (2013). Making research data repositories visible: The re3data. org registry. PLoS ONE, 8.10.7287/peerj.preprints.21v1 ↗
15. DataCite as a novel bibliometric source: Coverage, strengths and limitations
16. Zenodo (2025, April 05). Zenodo—Research. Shared. Available online: https://zenodo.org/.
17. Dryad (2025, April 05). Dryad—Publish and Preserve Your Data. Available online: https://datadryad.org/stash.
18. Figshare (2025, April 05). Figshare—Share Your Research. Available online: https://figshare.com/.
19. OpenAIRE-Building a collaborative Open Access infrastructure for European researchers10.18352/lq.8110 ↗
20. Raghavan, S., and Garcia-Molina, H. (2001, January 11–14). Crawling the Hidden Web. Proceedings of the 27th International Conference on Very Large Data Bases, Roma, Italy.
21. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
22. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
23. Stureborg, R., Alikaniotis, D., and Suhara, Y. (2024). Large language models are inconsistent and biased evaluators. arXiv.
24. Sharma, D.K., and Sharma, A. (2009, January 22–24). Query intensive interface information extraction protocol for deep web. Proceedings of the 2009 International Conference on Intelligent Agent & Multi-Agent Systems, Chennai, India.10.1109/IAMA.2009.5228052 ↗
25. Open metadata for research data discovery in Canada10.1080/19386389.2018.1443698 ↗
26. Genetics and genomics for the prevention and treatment of cardiovascular disease: Update: A scientific statement from the American Heart Association10.1161/01.cir.0000437913.98912.1d ↗
27. Barbosa, L., and Freire, J. (2005, January 16–17). Searching for Hidden-Web Databases. Proceedings of the Eight International Workshop on the Web & Databases (WebDB 2005), Baltimore, ML, USA.
28. Barbosa, L., and Freire, J. (2007, January 8–12). An adaptive crawler for locating hidden-web entry points. Proceedings of the 16th international conference on World Wide Web, Banff, AB, Canada.10.1145/1242572.1242632 ↗
29. Google’s Deep Web crawl10.14778/1454159.1454163 ↗
30. Behfarshad, Z., and Mesbah, A. (2013, January 8–12). Hidden-web induced by client-side scripting: An empirical study. Proceedings of the Web Engineering: 13th International Conference, ICWE 2013, Aalborg, Denmark.10.1007/978-3-642-39200-9_7 ↗
31. Mirtaheri, S.M., Dinçktürk, M.E., Hooshmand, S., Bochmann, G.V., Jourdan, G.V., and Onut, I.V. (2014). A brief history of web crawlers. arXiv.
32. Crawling Ajax-based web applications through dynamic analysis of user interface state changes10.1145/2109205.2109208 ↗
33. Lai, H., Liu, X., Iong, I.L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., and Dong, Y. (2024, January 25–29). AutoWebGLM: A Large Language Model-based Web Navigating Agent. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain.10.1145/3637528.3671620 ↗
34. Tang, B., and Shin, K.G. (2024). Steward: Natural language web automation. arXiv.
35. Skyvern-AI (2025, July 04). Skyvern: Automate browser-based workflows with LLMs and Computer Vision. Available online: https://github.com/Skyvern-AI/skyvern.
36. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023, January 1–5). React: Synergizing reasoning and acting in language models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
37. Bohra, A., Saroyan, M., Melkozerov, D., Karufanyan, V., Maher, G., Weinberger, P., Harutyunyan, A., and Campagna, G. (2025). WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents. arXiv.
38. Jia, F., Wu, T., Qin, X., and Squicciarini, A. (2024). The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. arXiv.
39. Chen, F.K., Liu, C.H., and You, S.D. (2025). Using Large Language Model to Fill in Web Forms to Support Automated Web Application Testing. Information, 16.10.3390/info16020102 ↗
40. Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., and Faust, A. (2022). Understanding html with large language models. arXiv.10.18653/v1/2023.findings-emnlp.185 ↗
41. Stafeev, A., Recktenwald, T., De Stefano, G., Khodayari, S., and Pellegrino, G. (March, January 26). YURASCANNER: Leveraging LLMs for Task-driven Web App Scanning. Proceedings of the Network and Distributed System Security (NDSS) Symposium, CISPA, San Diego, CA, USA.
42. Resource harvesting within the OAI-PMH framework
43. Brickley, D., Burgess, M., and Noy, N. (2019, January 13–17). Google Dataset Search: Building a search engine for datasets in an open Web ecosystem. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313685 ↗
44. Chen, Q., Pitawela, D., Zhao, C., Zhou, G., Chen, H.T., and Wu, Q. (2024, January 20–27). Webvln: Vision-and-language navigation on websites. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i2.27878 ↗
45. Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemiński, D., and Winata, G.I. (2025). MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv.
46. Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., and Le, Q.V. (2021). Finetuned language models are zero-shot learners. arXiv.
47. Large language models are zero-shot reasoners
48. Sun, Y., Zhuang, Z., and Giles, C.L. (2007, January 8–12). A large-scale study of robots.txt. Proceedings of the 16th International Conference on World Wide Web, Banff, AB, Canada.10.1145/1242572.1242726 ↗