A Clustering Visualization Method for Density Partitioning of Trajectory Big Data Based on Multi-Level Time Encoding
Source abstract
Abstract
The proliferation of the Internet and the widespread adoption of mobile devices have given rise to an immense volume of real-time trajectory big data. However, a single computer and conventional databases with limited scalability struggle to manage this data effectively. During the process of visual rendering, issues such as page stuttering and subpar visual outcomes often arise. This paper, founded on a distributed architecture, introduces a multi-level time encoding method using “minutes”, “hours”, and “days” as fundamental units, achieving a storage model for trajectory data at multi-scale time. Furthermore, building upon an improved DBSCAN clustering algorithm and integrating it with the K-means clustering algorithm, a novel density-based partitioning clustering algorithm has been introduced, which incorporates road coefficients to circumvent architectural obstacles, successfully resolving page stuttering issues and significantly enhancing the quality of visualization. The results indicate the following: (1) when data is extracted using the units of “minutes”, “hours”, and “days”, the retrieval efficiency of this model is 6.206 times, 12.475 times, and 18.634 times higher, respectively, compared to the retrieval efficiency of the original storage model. As the volume of retrieved data increases, the retrieval efficiency of the proposed storage model becomes increasingly superior to that of the original storage model. Under identical experimental conditions, this model’s retrieval efficiency also outperforms the space–time-coded storage model; (2) Under a consistent rendering level, the clustered trajectory data, when compared to the unclustered raw data, has shown a 40% improvement in the loading speed of generating heat maps. There is an absence of page stuttering. Furthermore, the heat kernel phenomenon in the heat map was also resolved while enhancing the visualization rendering speed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Management and Algorithms · Human Mobility and Location-Based Analysis · Traffic Prediction and Management Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Geomatics and Urban Spatial Informatics, Beijing University of Civil Engineering and Architecture, Beijing 102612, China
Funding
the National Natural Science Foundation of China · the Beijing University of Civil Engineering and Architecture 2023 Doctoral Postgraduate Research Ability Improvement Program
References (59)
Show source-provided references
1. Analysis of urban residents’ commuting activities based on mobile trajectory data support
2. Research on the Method and Application of MapReduce in Mobile Track Big Data Mining
3. Understanding transportation modes based on GPS data for web applications10.1145/1658373.1658374 ↗
4. Zhang, H., Zhang, J., Guo, X., Lu, J., and Lu, H. (2021). Cloud storage and heatmap generation method for trajectory big data. Bull. Surv. Mapp., 146–149.
5. Bala, P. (2021). Introduction of Big Data with Analytics of Big Data, IGI Global.10.4018/978-1-7998-2791-7.ch006 ↗
6. Big data in smart city
7. Gupta, P., Mittal, P.K., and Gopal, G. (2022, October 18). Big Data: Problems, Challenges and Techniques. Available online: https://www.researchgate.net/publication/321134019_Big_Data_Problems_Challenges_and_Techniques.
8. LatentMap: Effective auto-encoding of density maps for spatiotemporal data visualizations10.1016/j.gvc.2021.200019 ↗
9. Zhang, H. (2021). Research on Trajectory Big Data Model and Visualization Method Based on Hadoop. [Master’s Thesis, Beijing Architecture University].
10. Jeyaraj, R., Pugalendhi, G., and Paul, A. (2020). Big Data with Hadoop MapReduce, Apple Academic Press.10.1201/9780429321733 ↗
11. Research on mass monitoring data Retrieval Technology based on HBase10.1088/1742-6596/1871/1/012133 ↗
12. Hughes, J.N., Annex, A., Eichelberger, C.N., Fox, A., Hulbert, A., and Ronquest, M. (2015, January 20–24). GeoMesa: A distributed architecture for spatio-temporal fusion. Proceedings of the SPIE Defense + Security, Baltimore, MD, USA.10.1117/12.2177233 ↗
13. A demonstration of st-hadoop: A mapreduce framework for big spatio-temporal data10.14778/3137765.3137819 ↗
14. Optimizing segmented trajectory data storage with HBase for improved spatio-temporal query efficiency10.1080/17538947.2023.2192979 ↗
15. Building an efficient storage model of spatial-temporal information based on HBase10.1080/14498596.2018.1440648 ↗
16. MR-DBSCAN: A scalable MapReduce-based DBSCAN algorithm for heavily skewed data10.1007/s11704-013-3158-3 ↗
17. Massive data storage and sharing algorithm in distributed heterogeneous environment10.3233/JIFS-169724 ↗
18. -HBase: Design and implementation of an elastic data infrastructure for cloud-scale location services10.1007/s10619-012-7109-z ↗
19. Yao, Z., Zhang, J., Li, T., and Ding, Y. (2022). A trajectory big data storage model incorporating partitioning and spatio-temporal multidimensional hierarchical organization. ISPRS Int. J. Geo-Inf., 11.10.3390/ijgi11120621 ↗
20. V-SOINN: A Topology Preserving Visualization Method for Multidimensional Data10.1016/j.neucom.2021.03.113 ↗
21. Transcriptome network data in larval zebrafish (Danio rerio) following exposure to the phenylpyrazole fipronil10.1016/j.dib.2020.106413 ↗
22. Deep Time Series Models for Scarce Data10.1016/j.neucom.2020.12.132 ↗
23. AppAware: A policy visualization model for mobile applications10.1108/ICS-04-2019-0049 ↗
24. Big-Data Visualization10.1109/MCG.2013.54 ↗
25. Uncovering urban human mobility from large scale taxi GPS data10.1016/j.physa.2015.06.032 ↗
26. A rapid density method for taxi passengers hot spot recognition and visualization based on DBSCAN+10.1038/s41598-021-88822-3 ↗
27. A review of spatial clustering algorithms based on obstacle constraints
28. Voronoi diagram-based clustering algorithm for uncertain data in obstacle space
29. Tung, A.K.H., Hou, J., and Han, J. (2001, January 2–6). Spatial clustering in the presence of obstacles. Proceedings of the 17th International Conference on Data Engineering, Heidelberg, Germany.
30. Ng, R.T. (1994, January 12–15). Efficient and Effective Clustering Methods for Spatial Data Mining. Proceedings of the 20th VLDB Conference, Santiago de Chile, Chile.
31. Estivill-Castro, V., and Lee, I. (2000). TSDM 2000: Temporal, Spatial, and Spatio-Temporal Data Mining, Proceedings of the International Workshop on Temporal, Spatial, and Spatio-Temporal Data Mining, Lyon, France, 12 September 2000, Springer.
32. Zaiane, O.R., and Lee, C.H. (2002, January 9–12). Clustering spatial data when facing physical constraints. Proceedings of the 2002 IEEE International Conference on Data Mining, Maebashi City, Japan.
33. Ester, M. (1996, January 2–4). A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Proceedings of the KDD’96: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Portland, OR, USA.
34. Zhang, X., Wang, J., Wu, F., Fan, Z., and Li, X. (2006, January 16–18). A Novel Spatial Clustering with Obstacles Constraints Based on Genetic Algorithms and K-Medoids. Proceedings of the Sixth International Conference on Intelligent Systems Design and Applications, Jian, China.10.1109/ISDA.2006.75 ↗
35. Zhang, X., Wu, J., Si, H., Yang, T., and Liu, Y. (2007). PAKDD 2007: Emerging Technologies in Knowledge Discovery and Data Mining, Proceedings of the International Conference on Emerging Technologies in Knowledge Discovery & Data Mining, Nanjing, China, 22–25 May 2007, Springer.
36. A new algorithm for spatial clustering with obstacles by combining QPSO and K-Medoids
37. Research on the preprocessing technology of massive cab trajectory data under the support of Hadoop
38. MapReduce-based parallel partitioning algorithm for spatial data
39. An analysis of Ext4 for digital forensics10.1016/j.diin.2012.05.010 ↗
40. Gilmore, W.J. (2008). Beginning PHP and MySQL: From Novice to Professional, Apress.
41. An effective multi-scale time period dissection method with integer coding calculation
42. Cache optimization for compressed databases in multiple storage environments
43. Research on storage method of spatio-temporal trajectory data
44. Lei, Y. (2017). Vehicle Trajectory Data Management and Analysis Based on HBase. [Master’s Thesis, Southwest Jiaotong University].
45. A MapReduce-based method for storing and querying vector spatial data
46. A review of clustering algorithms
47. DBSCAN multi-density clustering algorithm based on region partitioning
48. Based on K-Means and DBSCAN clustering algorithm according to the background of student behavior analysis and research based on comprehensive university data
49. Improved adaptive parametric DBSCAN clustering algorithm
50. Research on nondestructive detection of sprouted potato based on Euclidean distance
51. Optimization algorithm for black holes based on Euclidean distance
52. Analysis of bus operating hours based on K-means clustering algorithm
53. K-means clustering algorithm for optimizing initial clustering centers
54. Spark-based heatmap visualization method for big data
55. A real-time mapping method of thematic heat maps for mobile terminals
56. A thermal map generation method with structural constraints for indoor spaces
57. Heatmap-based calculation method for spatial distribution of geographic objects
58. A research on visual representation of geotagged data based on Heat Map
59. A heat map generation algorithm for millions of data