A Gaussian Process Decoder with Spectral Mixtures and a Locally Estimated Manifold for Data Visualization
Source abstract
Abstract
Dimensionality reduction plays an important role in interpreting and visualizing high-dimensional data. Previous methods for data visualization overestimate the local structure and lack the consideration of global preservation. In this study, we develop a Gaussian process latent variable model (GP-LVM) for data visualization. GP-LVMs are one of the frameworks of principal component analysis and preserve the global structure effectively. The drawbacks of GP-LVMs are the absence of local structure preservation and the use of low-expressive kernel functions. Therefore, we introduce regularization for local preservation and an expressive kernel function into GP-LVMs to overcome these limitations. As a result, we reflect the global and local structures in low-dimensional representations, improving the reliability and visibility of embeddings. We conduct qualitative and quantitative experiments comparing baselines and state-of-the-art methods on image and text datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaussian Processes and Bayesian Inference · Time Series Analysis and Forecasting · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Hokkaido, Japan
Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Hokkaido, Japan
Funding
JSPS KAKENHI · AMED
References (61)
Show source-provided references
1. Dimensionality reduction: A comparative
2. Analysis of a complex of statistical variables into principal components10.1037/h0071325 ↗
3. Principal component analysis10.1016/0169-7439(87)80084-9 ↗
4. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
5. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
6. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016, January 2–4). β-vae: Learning basic visual concepts with a constrained variational framework. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
7. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015, January 6–11). Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the International Conference on Machine Learning (ICML), Lille, France.
8. Denoising diffusion probabilistic models
9. Deep learning10.1038/nature14539 ↗
10. Visualizing data using t-SNE
11. Accelerating t-SNE using tree-based algorithms
12. McInnes, L., Healy, J., and Melville, J. (2018). UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv.10.21105/joss.00861 ↗
13. The art of using t-SNE for single-cell transcriptomics10.1038/s41467-019-13056-x ↗
14. Dimensionality reduction for visualizing single-cell data using UMAP10.1038/nbt.4314 ↗
15. Probabilistic non-linear principal component analysis with Gaussian process latent variable models
16. Titsias, M., and Lawrence, N.D. (2010, January 13–15). Bayesian Gaussian process latent variable model. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, Italy.
17. Neal, R.M. (2012). Bayesian Learning for Neural Networks, Springer Science & Business Media.
18. Lee, J., Bahri, Y., Novak, R., Schoenholz, S.S., Pennington, J., and Sohl-Dickstein, J. (2017). Deep neural networks as Gaussian processes. arXiv.
19. Märtens, K., Campbell, K., and Yau, C. (2019, January 9–15). Decomposing feature-level variation with covariate Gaussian process latent variable models. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
20. Manifold GPLVMs for discovering non-Euclidean latent structure in neural data
21. Liu, Z. (2020). Visualizing single-cell RNA-seq data with semisupervised principal component analysis. Int. J. Mol. Sci., 21.10.3390/ijms21165797 ↗
22. Jørgensen, M., and Hauberg, S. (2021, January 18–24). Isometric Gaussian process latent variable model for dissimilarity data. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia.
23. Lalchand, V., Ravuri, A., and Lawrence, N.D. (2022, January 28–30). Generalised GPLVM with stochastic variational inference. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Virtual Event.
24. Gaussian process dynamical models for human motion10.1109/TPAMI.2007.1167 ↗
25. Variational Inference for Latent Variables and Uncertain Inputs in Gaussian Processes
26. Ferris, B., Fox, D., and Lawrence, N. (2007, January 6–12). WiFi-SLAM using Gaussian process latent variable models. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India.
27. Zhang, G., Wang, P., Chen, H., and Zhang, L. (2019). Wireless indoor localization using convolutional neural network and Gaussian process regression. Sensors, 19.10.3390/s19112508 ↗
28. Lu, C., and Tang, X. (2015, January 25–30). Surpassing human-level face verification performance on LFW with GaussianFace. Proceedings of the AAAI conference on Artificial Intelligence (AAAI), Austin, TX, USA.10.1609/aaai.v29i1.9797 ↗
29. Cho, Y., and Saul, L. (2009, January 7–10). Kernel methods for deep learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
30. Wilson, A., and Adams, R. (2013, January 16–21). Gaussian process kernels for pattern discovery and extrapolation. Proceedings of the International Conference on Machine Learning (ICML), Atlanta, GA, USA.
31. Lloyd, J., Duvenaud, D., Grosse, R., Tenenbaum, J., and Ghahramani, Z. (2014, January 27–31). Automatic construction and natural-language description of nonparametric regression models. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Quebec City, QC, Canada.10.1609/aaai.v28i1.8904 ↗
32. Urtasun, R., and Darrell, T. (2007, January 19–22). Discriminative Gaussian process latent variable model for classification. Proceedings of the International Conference on Machine Learning (ICML), Hong Kong, China.10.1145/1273496.1273613 ↗
33. Zhong, G., Li, W.J., Yeung, D.Y., Hou, X., and Liu, C.L. (2010, January 11–15). Gaussian process latent random field. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Atlanta, GA, USA.10.1609/aaai.v24i1.7697 ↗
34. Laplacian eigenmaps for dimensionality reduction and data representation10.1162/089976603321780317 ↗
35. Belkin, M., and Niyogi, P. (2006, January 4–7). Convergence of Laplacian eigenmaps. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
36. Carreira-Perpinán, M.A. (2010, January 21–24). The Elastic Embedding Algorithm for Dimensionality Reduction. Proceedings of the International Conference on Machine Learning (ICML), Haifa, Israel.
37. Fu, C., Zhang, Y., Cai, D., and Ren, X. (2019, January 4–8). AtSNE: Efficient and robust visualization on gpu through hierarchical optimization. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AL, USA.10.1145/3292500.3330834 ↗
38. Attraction-repulsion spectrum in neighbor embeddings
39. Amid, E., and Warmuth, M.K. (2019). TriMap: Large-scale dimensionality reduction using triplets. arXiv.
40. Understanding how dimension reduction tools work: An empirical approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data visualization
41. Visualizing structure and transitions in high-dimensional biological data10.1038/s41587-019-0336-3 ↗
42. Probabilistic principal component analysis10.1111/1467-9868.00196 ↗
43. Kernel methods in machine learning10.1214/009053607000000677 ↗
44. Rasmussen, C.E., and Williams, C.K. (2006). Gaussian Processes for Machine Learning, MIT Press.10.7551/mitpress/3206.001.0001 ↗
45. Lawrence, N.D., and Moore, A.J. (2007, January 20–24). Hierarchical Gaussian process latent variable models. Proceedings of the International Conference on Machine Learning (ICML), Corvallis, OR, USA.10.1145/1273496.1273557 ↗
46. A unifying view of sparse approximate Gaussian process regression
47. Titsias, M. (2009, January 16–18). Variational learning of inducing variables in sparse Gaussian processes. Proceedings of the International Conference on Artificial intelligence and statistics (AISTATS), Clearwater Beach, FL, USA.
48. Lawrence, N.D. (2007, January 21–24). Learning for larger datasets with the Gaussian process latent variable model. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), San Juan, Puerto Rico.10.1145/1273496.1273557 ↗
49. Damianou, A., and Lawrence, N.D. (May, January 29). Deep Gaussian processes. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR, Scottsdale, AZ, USA.
50. Dai, Z., Damianou, A., González, J., and Lawrence, N. (2016, January 2–4). Variational auto-encoded deep Gaussian processes. Proceedings of the International Conference on Learning Representation (ICLR), San Juan, Puerto Rico.
51. Shewchuk, J.R. (1994). An Introduction to the Conjugate Gradient Method without the Agonizing Pain, Carnegie Mellon University.
52. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization10.1145/279232.279236 ↗
53. GPy (2020, June 26). GPy: A Gaussian Process Framework in Python. Available online: http://github.com/SheffieldML/GPy.
54. Nene, S.A., Nayar, S.K., and Murase, H. (1996). Columbia Object Image Library (COIL-20), Columbia University. Technical Report CUCS-006-96.
55. Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv.
56. Diffusion maps10.1016/j.acha.2006.04.006 ↗
57. Venna, J., and Kaski, S. (2001, January 21–25). Neighborhood preservation in nonlinear projection methods: An experimental study. Proceedings of the International Conference on Artificial Neural Networks (ICANN), Vienna, Austria.10.1007/3-540-44668-0_68 ↗
58. Toward a quantitative survey of dimension reduction techniques10.1109/TVCG.2019.2944182 ↗
59. Zu, X., and Tao, Q. (2022, January 17–23). SpaceMAP: Visualizing High-dimensional Data by Space Expansion. Proceedings of the International Conference on Machine Learning (ICML), Baltimore, MD, USA.
60. Local affine multidimensional projection10.1109/TVCG.2011.220 ↗
61. SciPy 1.0: Fundamental algorithms for scientific computing in Python10.1038/s41592-019-0686-2 ↗