Hierarchical Indexing and Compression Method with AI-Enhanced Restoration for Scientific Data Service
Source abstract
Abstract
In the process of data services, compressing and indexing data can reduce storage costs, improve query efficiency, and thus enhance the quality of data services. However, different service requirements have diverse demands for data precision. Traditional lossy compression techniques fail to meet the precision requirements of different data due to their fixed compression parameters and schemes. Additionally, error-bounded lossy compression techniques, due to their tightly coupled design, cannot achieve high compression ratios under high precision requirements. To address these issues, this paper proposes a lossy compression technique based on error control. Instead of imposing precision constraints during compression, this method first uses the JPEG compression algorithm for multi-level compression and then manages data through a tree-based index structure to achieve error control. This approach satisfies error control requirements while effectively avoiding tight coupling. Additionally, this paper enhances data restoration effects using a deep learning network and provides a range query processing algorithm for the tree-based index to improve query efficiency. We evaluated our solution using ocean data. Experimental results show that, while maintaining data precision requirements (PSNR of at least 39 dB), our compression ratio can reach 64, which is twice that of the SZ compression algorithm.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Scientific Computing and Data Management · Advanced Computational Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software, Nanjing University of Information Science and Technology, Nanjing 210044, China
School of Computer Science, Nanjing University of Information Science and Technology, Nanjing 210044, China
School of Teacher Education, Nanjing University of Information Science and Technology, Nanjing 210044, China
School of International Education, Nanjing Institute of Technology, Nanjing 211167, China
Funding
the national natural science foundation of China
References (30)
Show source-provided references
1. Liu, J., Di, S., Zhao, K., Jin, S., Tao, D., Liang, X., Chen, Z., and Cappello, F. (2021, January 7–10). Exploring autoencoder-based error-bounded compression for scientific data. Proceedings of the 2021 IEEE International Conference on Cluster Computing (CLUSTER), Portland, OR, USA.10.1109/Cluster48925.2021.00034 ↗
2. Guan, R., Wang, Z., Pan, X., Zhu, R., Song, B., and Zhang, X. (2023). SbMBR Tree—A Spatiotemporal Data Indexing and Compression Algorithm for Data Analysis and Mining. Appl. Sci., 13.10.3390/app131910562 ↗
3. A survey on data compression techniques: From the perspective of data quality, coding schemes, data type and applications
4. Ballé, J., Laparra, V., and Simoncelli, E.P. (2016). End-to-end optimized image compression. arXiv.
5. End-to-End Deep ROI Image Compression10.1587/transinf.2019EDP7264 ↗
6. Theis, L., Shi, W., Cunningham, A., and Huszár, F. (2022, January 25–29). Lossy image compression with compressive autoencoders. Proceedings of International Conference on Learning Representations, Virtual Event.
7. Zhai, J., Zhang, S., Chen, J., and He, Q. (2018, January 7–10). Autoencoder and its various variants. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.10.1109/SMC.2018.00080 ↗
8. Deep learning for in situ data compression of large turbulent flow simulations10.1103/PhysRevFluids.5.114602 ↗
9. Deepcomp: A hybrid framework for data compression using attention coupled autoencoder10.1007/s13369-022-06587-x ↗
10. An introduction to arithmetic coding10.1147/rd.282.0135 ↗
11. A method for the construction of minimum-redundancy codes10.1109/JRPROC.1952.273898 ↗
12. The JPEG still picture compression standard10.1145/103085.103089 ↗
13. A universal algorithm for sequential data compression10.1109/TIT.1977.1055714 ↗
14. Tao, D., Di, S., Chen, Z., and Cappello, F. (June, January 29). Significantly improving lossy compression for scientific data sets based on multidimensional prediction and error-controlled quantization. Proceedings of the 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Orlando, FL, USA.10.1109/IPDPS.2017.115 ↗
15. Fixed-rate compressed floating-point arrays10.1109/TVCG.2014.2346458 ↗
16. High-ratio lossy compression: Exploring the autoencoder to compress scientific data10.1109/TBDATA.2021.3066151 ↗
17. Azri, S., Ujang, U., Anton, F., Mioc, D., and Rahman, A.A. (2013, January 24–25). Review of spatial indexing techniques for large urban data management. Proceedings of the International Symposium & Exhibition on Geoinformation (ISG), Kuala Lumpur, Malaysia.
18. Guttman, A. (1984, January 18–21). R-trees: A dynamic index structure for spatial searching. Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, Boston, MA, USA.10.1145/602264.602266 ↗
19. Sellis, T., Roussopoulos, N., and Faloutsos, C. (1987, January 1–4). The R+-Tree: A Dynamic Index for Multi-Dimensional Objects. Proceedings of the 13th International Conference on Very Large Data Bases, Brighton, UK.
20. Beckmann, N., Kriegel, H.P., Schneider, R., and Seeger, B. (1990, January 23–26). The R*-tree: An efficient and robust access method for points and rectangles. Proceedings of the 1990 ACM SIGMOD International Conference on Management of Data, Atlantic, NJ, USA.10.1145/93597.98741 ↗
21. Parallel R-trees10.1145/141484.130315 ↗
22. Kamel, I., and Faloutsos, C. (1994, January 12–15). Hilbert r-tree: An improved rtree using fractals. Proceedings of the 20th International Conference on Very Large Data Bases, Santiago de, Chile, Chile.
23. The priority R-tree: A practically efficient and worst-case optimal R-tree10.1145/1328911.1328920 ↗
24. Quad trees a data structure for retrieval on composite keys10.1007/BF00288933 ↗
25. Robinson, J.T. (May, January 29). The KDB-tree: A search structure for large multidimensional dynamic indexes. Proceedings of the 1981 ACM SIGMOD International Conference on Management of Data, Ann Arbor, MI, USA.10.1145/582318.582321 ↗
26. A hybrid spatio-temporal data indexing method for trajectory databases10.3390/s140712990 ↗
27. Tang, X., Han, B., and Chen, H. (2016, January 17–19). A hybrid index for multi-dimensional query in HBase. Proceedings of the 2016 4th International Conference on Cloud Computing and Intelligence Systems (CCIS), Beijing, China.10.1109/CCIS.2016.7790279 ↗
28. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
29. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
30. Angular momentum eddy detection and tracking algorithm (AMEDA) and its application to coastal eddy formation10.1175/JTECH-D-17-0010.1 ↗