A Novel Approach to Data Extraction on Hyperlinked Webpages
Source abstract
Abstract
The World Wide Web has an enormous amount of useful data presented as HTML tables. These tables are often linked to other web pages, providing further detailed information to certain attribute values. Extracting schema of such relational tables is a challenge due to the non-existence of a standard format and a lack of published algorithms. We downloaded 15,000 web pages using our in-house developed web-crawler, from various web sites. Tables from the HTML code were extracted and table rows were labeled with appropriate class labels. Conditional random fields (CRF) were used for the classification of table rows, and a nondeterministic finite automaton (NFA) algorithm was designed to identify simple, complex, hyperlinked, or non-linked tables. A simple schema for non-linked tables was extracted and for the linked-tables, relational schema in the form of primary and foreign-keys (PK and FK) were developed. Child tables were concatenated with the parent table’s attribute value (PK), serving as foreign keys (FKs). Resultantly, these tables could assist with performing better and stronger queries using the join operation. A manual checking of the linked web table results revealed a 99% precision and 68% recall values. Our 15,000-strong downloadable corpus and a novel algorithm will provide the basis for further research in this field.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Web Data Mining and Analysis · Advanced Database Systems and Queries · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering and Computer Science, University of Stavanger, 4036 Stavanger, Norway
School of Electrical Engineering and Computing, The University of Newcastle, Callaghan 2308, Australia
Punjab University College of Information Technology, University of the Punjab, Lahore 54590, Pakistan
Department of Computer Science, Capital University of Science and Technology, Islamabad 45750, Pakistan
School of Computer Science, The University of Sydney, Sydney 2006, Australia
Funding
Universitetet i Stavanger
References (39)
Show source-provided references
1. Annotating and searching web tables using entities, types and relationships10.14778/1920841.1921005 ↗
2. Wang, Y., and Hu, J. (2002, January 19–21). Detecting tables in html documents. Proceedings of the International Workshop on Document Analysis Systems, Princeton, NJ, USA.
3. A survey of table recognition
4. Webtables: Exploring the power of tables on the web10.14778/1453856.1453916 ↗
5. Cafarella, M.J., Halevy, A., Zhang, Y., Wang, D.Z., and Wu, E. (2008, January 13). Uncovering the Relational Web. Proceedings of the 11th International Workshop on Web and Databases (WebDB 2008), Vancouver, BC, Canada.
6. Ten years of webtables10.14778/3229863.3240492 ↗
7. Embley, D.W., Krishnamoorthy, M., Nagy, G., and Seth, S. (July, January 28). Factoring web tables. Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Syracuse, NY, USA.
8. Chen, H.-H., Tsai, S.-C., and Tsai, J.-H. (August, January 31). Mining tables from large scale HTML texts. Proceedings of the 18th Conference on Computational Linguistics, Saarbrücken, Germany.
9. Chen, Z., and Cafarella, M. (2013, January 30). Automatic web spreadsheet data extraction. Proceedings of the 3rd International Workshop on Semantic Search over the Web, Riva del Garda, Italy.10.1145/2509908.2509909 ↗
10. Fang, J., Mitra, P., Tang, Z., and Giles, C.L. (2012, January 22–26). Table header detection and classification. Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, ON, Canada.
11. Caragea, C., Wu, J., Ciobanu, A., Williams, K., Fernández-Ramírez, J., and Chen, H.H. (2014, January 13–16). Citeseer x: A scholarly big dataset. Proceedings of the European Conference on Information Retrieval, Amsterdam, The Netherlands.10.1007/978-3-319-06028-6_26 ↗
12. Penn, G., Hu, J., Luo, H., and McDonald, R. (2001, January 13). Flexible web document analysis for delivery to narrow-bandwidth devices. Proceedings of the Sixth International Conference on Document Analysis and Recognition, Seattle, WA, USA.
13. Han, J., Pei, J., and Kamber, M. (2011). Data Mining: Concepts and Techniques, Elsevier.
14. Web table taxonomy and formalization10.1145/2536669.2536674 ↗
15. Nagy, G. (2012, January 11–15). Learning the characteristics of critical cells from web tables. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
16. Development of a data entry auditing protocol and quality assurance for a tissue bank database10.1007/s10561-011-9240-x ↗
17. Hurst, M. (2019, October 31). Layout and language: Challenges for table understanding on the web. Available online: http://wda2001.csc.liv.ac.uk//Papers/12_hurst_wda2001.
18. Nagy, G., Padmanabhan, R., Jandhyala, R., Silversmith, W., and Krishnamoorthy, M. (2019, October 31). Table metadata: Headers, augmentations and aggregates. Available online: https://www.ecse.rpi.edu/~nagy/PDF_chrono/2010_Padmanabhan_Nagy_etal_DAS2010.
19. Yakout, M., Ganjam, K., Chakrabarti, K., and Chaudhuri, S. (2012, January 20–24). Infogather: Entity augmentation and attribute discovery by holistic matching with web tables. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, New York, NY, USA.10.1145/2213836.2213848 ↗
20. Di Mauro, N., Basile, T.M., Ferilli, S., and Esposito, F. (2011, January 28–30). Optimizing probabilistic models for relational sequence learning. Proceedings of the International Symposium on Methodologies for Intelligent Systems, Warsaw, Poland.10.1007/978-3-642-21916-0_27 ↗
21. Multi-dimensional relational sequence mining
22. Koller, D., Friedam, N, Džeroski, S, Sutton, C., McCallum, A., Pfeffer, A., and Neville, J. (2007). Introduction to Statistical Relational Learning, MIT Press.
23. Shaukat, K., Masood, N., and Mehreen, S. (2017). Population of Data in Extracted Web Table Schema, LAP Lambert Academic Publishing.10.1109/INMIC.2016.7840093 ↗
24. Shaukat, K., Masood, N., Mehreen, S., Haider, F., Bakar, A., and Shaukat, U. (2016, January 5–6). Population of data in web-tables schema. Proceedings of the 2016 19th International Multi-Topic Conference (INMIC), Islamabad, Pakistan.10.1109/INMIC.2016.7840093 ↗
25. Schema extraction for tabular data on the web10.14778/2536336.2536343 ↗
26. Babu, S., Motwani, R., Munagala, K., Nishizawa, I., and Widom, J. (2004, January 13–18). Adaptive ordering of pipelined stream filters. Proceedings of the 2004 ACM SIGMOD international conference on Management of data, Paris, France.10.1145/1007568.1007615 ↗
27. Lafferty, J., McCallum, A., and Pereira, F.C. (2019, October 31). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Available online: https://dl.acm.org/citation.cfm?id=655813.
28. Condon, A., Deshpande, A., Hellerstein, L., and Wu, N. (2006, January 26–28). Flow algorithms for two pipelined filter ordering problems. Proceedings of the Twenty-Fifth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, Chicago, IL, USA.10.1145/1142351.1142379 ↗
29. Kodialam, M.S. (2001, January 13–15). The throughput of sequential testing. Proceedings of the International Conference on Integer Programming and Combinatorial Optimization, Utrecht, The Netherlands.10.1007/3-540-45535-3_22 ↗
30. Srivastava, U., Munagala, K., Widom, J., and Motwani, R. (2006, January 12–15). Query optimization over web services. Proceedings of the 32nd international conference on Very large data bases, Seoul, Korea.
31. Sekhavat, Y.A., di Paolo, F., Barbosa, D., and Merialdo, P. (2014, January 8). Knowledge Base Augmentation using Tabular Data. Proceedings of the LDOW, Seoul, Korea.
32. DiFranzo, D., Ding, L., Graves, A., Michaelis, J.R., Li, X., McGuinness, D.L., and Hendler, J. (2010, January 22–24). Data-gov wiki: Towards linking government data. Proceedings of the 2010 AAAI Spring Symposium Series, Palo Alto, CA, USA.
33. Recovering semantics of tables on the web10.14778/2002938.2002939 ↗
34. Embley, D.W., Seth, S., and Nagy, G. (2014, January 24–28). Transforming web tables to a RELATIONAL database. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.479 ↗
35. Random forests10.1023/A:1010933404324 ↗
36. Electronic biorepository application system: Web-based software to manage receipt, peer review, and approval of researcher applications to a biobank10.1089/bio.2011.0038 ↗
37. Hassan, M.U., Shaukat, K., Niu, D., Mahreen, S., Ma, Y., Haider, F., and Zhao, X. (2018, January 25–27). An Overview of Schema Extraction and Matching Techniques. Proceedings of the 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Xi’an, China.10.1109/IMCEC.2018.8469502 ↗
38. Data integration for the relational web10.14778/1687627.1687750 ↗
39. Benchmarking Database Performance for Genomic Data