Multi-Angle Fast Neural Tangent Kernel Classifier
Source abstract
Abstract
Multi-kernel learning methods are essential kernel learning methods. Still, the base kernel functions in most multi-kernel learning methods only with select kernel functions with shallow structures, which are weak for large-scale uneven data. We propose two types of acceleration models from a multidimensional perspective of the data: the neural tangent kernel (NTK)-based multi-kernel learning method is proposed, where the NTK kernel regressor is shown to be equivalent to an infinitely wide neural network predictor, and the NTK with deep structure is used as the base kernel function to enhance the learning ability of multi-kernel models; and a parallel computing kernel model based on data partitioning techniques. An RBF, POLY-based multi-kernel model is also proposed. All models use historical memory-based PSO (HMPSO) for efficient search of parameters within the model. Since NTK has a multi-layer structure and thus has a significant computational complexity, the use of a Monotone Disjunctive Kernel (MDK) to store and train Boolean features in binary achieves a 15–60% training time compression of NTK models in different datasets while obtaining a 1–25% accuracy improvement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and ELM · Neural Networks and Applications · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mathematics and Physics, Lanzhou Jiaotong University, Lanzhou 730070, China
School of Statistics, Southwestern University of Finance and Economics, Chengdu 611130, China
Funding
Department of Education Research Project of Gansu Province
References (46)
Show source-provided references
1. Federated machine learning: Concept and applications10.1145/3339474 ↗
2. Communication-efficient learning of deep networks from decentralized data
3. Wei, K., Li, J., Ma, C., Ding, M., Wei, S., Wu, F., and Ranbaduge, T. (2022). Vertical Federated Learning: Challenges, Methodologies and Experiments. arXiv.
4. Distributed learning of deep neural network over multiple agents10.1016/j.jnca.2018.05.003 ↗
5. Thapa, C., Arachchige, P.C.M., Camtepe, S., and Sun, L. (2022, January 24–28). Splitfed: When federated learning meets split learning. Proceedings of the AAAI Conference on Artificial Intelligence, Pomona, CA, USA.10.1609/aaai.v36i8.20825 ↗
6. Iñaki, E.A., and Abdellatif, Z. (2018, January 21–23). Distributed information bottleneck method for discrete and Gaussian sources. Proceedings of the International Zurich Seminar on Information and Communication, Zurich, Switzerland.
7. Moldoveanu, M., and Abdellatif, Z. (2021, January 27–30). On in-network learning. A comparative study with federated and split learning. Proceedings of the IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications, Lucca, Italy.10.1109/SPAWC51858.2021.9593182 ↗
8. Learning the kernel matrix with semidefinite programming
9. Sefidgaran, M., Chor, R., and Zaidi, A. (2022, January 20). Rate-distortion bounds on the generalization error of distributed learning. Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems, New Orleans, LA, USA.
10. The effect of anisotropic scattering on the critical slab problem in neutron transport theory using a synthetic kernel10.1088/0022-3727/10/15/007 ↗
11. Support vector machine learning from heterogeneous data: An empirical analysis using protein sequence and structure10.1093/bioinformatics/btl475 ↗
12. Multiscale kernel based residual convolutional neural network for motor fault diagnosis under nonstationary conditions10.1109/TII.2019.2941868 ↗
13. Multiscale kernels
14. Multiscale kernel pls-based exponentially weighted-glrt and its application to fault detection10.1109/TETCI.2017.2769111 ↗
15. Infinite kernel learning via infinite and semi-infinite programming10.1080/10556780903483349 ↗
16. On numerical optimization theory of infinite kernel learning10.1007/s10898-009-9488-x ↗
17. Liu, Y., Liao, S., Lin, H., Yue, Y., and Wang, W. (2017, January 4–9). Infinite kernel learning: Generalization bounds and algorithms. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10771 ↗
18. Gastroesophageal reflux disease diagnosis using hierarchical heterogeneous descriptor fusion support vector machine10.1109/TBME.2015.2466460 ↗
19. Improved support vector machine algorithm for heterogeneous data10.1016/j.patcog.2014.12.015 ↗
20. Shi, Y., and Eberhart, R.C. (1998, January 4–9). A modified particle swarms optimizer. Proceedings of the IEEE Congress on Evolutionary Computation, Anchorage, AK, USA.
21. Dey, A., Sharma, K.D., Sanyal, T., Bhattacharjee, P., and Bhattacharjee, P. (IEEE Trans. Nanobiosci., 2022). Identification of Biomarkers for Arsenicosis Employing Multiple Kernel Learning Embedded Multi-objective Swarm Intelligence, IEEE Trans. Nanobiosci., online ahead of print.10.1109/TNB.2022.3194091 ↗
22. Prediction of high and low disease activity in early MS patients using multiple kernel learning identifies importance of lateral ventricle intensity
23. A high-order norm-product regularized multiple kernel learning framework for kernel optimization10.1016/j.ins.2022.05.044 ↗
24. Price, S.R., Anderson, D.T., Havens, T.C., and Price, S.R. (2022). Kernel Matrix-Based Heuristic Multiple Kernel Learning. Mathematics, 10.10.3390/math10122026 ↗
25. A bi-level formulation for multiple kernel learning via self-paced training10.1016/j.patcog.2022.108770 ↗
26. Gravitation balanced multiple kernel learning for imbalanced classification10.1007/s00521-022-07187-4 ↗
27. Kernel learning backward SDE filter for data assimilation10.1016/j.jcp.2022.111009 ↗
28. Incomplete-view oriented kernel learning method with generalization error bound10.1016/j.ins.2021.10.011 ↗
29. Quantum semi-supervised kernel learning10.1007/s42484-021-00053-x ↗
30. Semi-supervised multiple empirical kernel learning with pseudo empirical loss and similarity regularization10.1002/int.22690 ↗
31. Deep Multiple Kernel Learning for Prediction of MicroRNA Precursors
32. Multiple birth support vector machine based on dynamic quantum particle swarm optimization algorithm10.1016/j.neucom.2022.01.012 ↗
33. Han, B., Ji, K., Singh, B.P.M., Qiu, J., and Zhang, P. (2022). An Optimization Method for Mix Proportion of Wet-Mix Shotcrete: Combining Artificial Neural Network with Particle Swarm Optimization. Appl. Sci., 12.10.3390/app12031698 ↗
34. Model Identification and Human-robot Coupling Control of Lower Limb Exoskeleton with Biogeography-based Learning Particle Swarm Optimization10.1007/s12555-020-0632-1 ↗
35. Polato, M., Lauriola, I., and Aiolli, F. (2018). A Novel Boolean Kernels Family for Categorical Data. Entropy, 20.10.3390/e20060444 ↗
36. Boolean kernels for rule based interpretation of support vector machines10.1016/j.neucom.2018.11.094 ↗
37. Alfaro, C., Gomez, J., Moguerza, J.M., Castillo, J., and Martinez, J.I. (2021). Toward Accelerated Training of Parallel Support Vector Machines Based on Voronoi Diagrams. Entropy, 23.10.3390/e23121605 ↗
38. An accelerator for support vector machines based on the local geometrical information and data partition10.1007/s13042-018-0877-7 ↗
39. Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN’95-International Conference on Neural Networks, Perth, Australia.
40. Composite Particle Swarm Optimizer with Historical Memory for Function Optimization10.1109/TCYB.2015.2424836 ↗
41. Jacot, A., Gabriel, F., and Clément, H. (2018, January 3–8). Neural Tangent Kernel: Convergence and Generalization in Neural Networks. Proceedings of the Neural Information Processing Systems, Montréal, QC, Canada.
42. Neal, R.M. (1996). Priors for infinite networks. Bayesian Learning for Neural Networks, Springer.10.1007/978-1-4612-0745-0 ↗
43. Matthews, A.G.D.G., Rowland, M., Hron, J., Turner, R.E., and Ghahramani, Z. (2019, January 6–9). Gaussian Process Behaviour in Wide Deep Neural Networks. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
44. Arora, S., Du, S.S., Li, Z., Salakhutdinov, R., Wang, R., and Yu, D. (2020, January 26–30). Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia.
45. Du, S.S., Hou, K., Salakhutdinov, R.R., Poczos, B., Wang, R., and Xu, K. (2019, January 8–14). Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels. Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada.
46. Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A.A., Sohl-Dickstein, J., and Schoenholz, S.S. (2021, January 3–7). Neural Tangents: Fast and Easy Infinite Neural Networks in Python. Proceedings of the 9th International Conference on Learning Representations, Virtual Event, Austria.