Local Back-Propagation for Forward-Forward Networks: Independent Unsupervised Layer-Wise Training
Source abstract
Abstract
Recent deep learning models, including GPT-4, have achieved remarkable performance using the back-propagation (BP) algorithm. However, the mechanism of BP is fundamentally different from how the human brain processes learning. To address this discrepancy, the Forward-Forward (FF) algorithm was introduced. Although FF enables deep learning without backward passes, it suffers from instability, dependence on artificial input construction, and limited generalizability. To overcome these challenges, we propose Local Back-Propagation (LBP), a method that integrates layer-wise unsupervised learning with standard inputs and conventional loss functions. Specifically, LBP demonstrates high training stability and competitive accuracy, significantly outperforming FF-based training methods. Moreover, LBP reduces memory usage by up to 48% compared to convolutional neural networks trained with back-propagation, making it particularly suitable for resource-constrained environments such as federated learning. These results suggest that LBP is a promising biologically inspired training method for decentralized deep learning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Indoor and Outdoor Localization Technologies · Microwave Imaging and Scattering Analysis · Energy Efficient Wireless Sensor Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Engineering, Chungnam National University, Daejeon 34134, Republic of Korea
Eureka AI, Daejeon 34134, Republic of Korea
Funding
Institute of Information and Communications Technology Planning and Evaluation (IITP) · Artificial Intelligence Convergence Innovation Human Resources Development (Chungnam National University) · National Research Foundation of Korea (NRF) · research fund from Chung National University
References (34)
Show source-provided references
1. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
2. Gradient theory of optimal flight paths10.2514/8.5282 ↗
3. Linnainmaa, S. (1970). The Representation of the Cumulative Rounding Error of an Algorithm as a Taylor Expansion of the Local Rounding Errors. [Master’s Thesis, University of Helsinki]. (In Finnish).
4. Competitive learning: From interactive activation to adaptive resonance10.1111/j.1551-6708.1987.tb00862.x ↗
5. The recent excitement about neural networks10.1038/337129a0 ↗
6. Shepherd, G.M. (1990). The significance of real neuron architectures for neural network simulations. Computational Neuroscience, ACM Digital Library.
7. Marblestone, A.H., Wayne, G., and Kording, K.P. (2016). Toward an integration of deep learning and neuroscience. Front. Comput. Neurosci., 10.10.3389/fncom.2016.00094 ↗
8. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects10.1038/4580 ↗
9. Lillicrap, T.P., Cownden, D., Tweed, D.B., and Akerman, C.J. (2014). Random feedback weights support learning in deep neural networks. arXiv.
10. Hinton, G. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv.
11. Self-Contrastive Forward-Forward Algorithm10.1038/s41467-025-61037-0 ↗
12. Tang, D.Y. (2023). The integrated forward-forward algorithm: Integrating forward-forward and shallow backpropagation with local losses. arXiv.
13. Pau, D., Pisani, A., and Candelieri, A. (2024). Towards Full Forward On-Tiny-Device Learning: A Guided Search for a Randomly Initialized Neural Network. Algorithms, 17.10.3390/a17010022 ↗
14. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex10.1113/jphysiol.1962.sp006837 ↗
15. Topographic organization of the reciprocal connections between the monkey entorhinal cortex and the perirhinal and parahippocampal cortices10.1523/JNEUROSCI.14-03-01856.1994 ↗
16. An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity10.1162/NECO_a_00949 ↗
17. Predictive coding approximates backprop along arbitrary computation graphs10.1162/neco_a_01497 ↗
18. Rumelhart, D.E., McClelland, J.L., and PDP Research Group (1988). Parallel Distributed Processing; Explorations in the Microstructure of Cognition: Foundations, MIT Press.10.7551/mitpress/5237.001.0001 ↗
19. Backpropagation applied to handwritten zip code recognition10.1162/neco.1989.1.4.541 ↗
20. Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1985). Learning Internal Representations by Error Propagation, Stanford University.10.21236/ADA164453 ↗
21. Ororbia, A., and Mali, A.A. (2023, January 26–29). The Predictive Forward-Forward Algorithm. Proceedings of the Annual Meeting of the Cognitive Science Society 2023, Sydney, Australia.
22. Konečný, J., McMahan, H.B., Yu, F.X., Richtarik, P., Suresh, A.T., and Bacon, D. (2016, January 9). Federated Learning: Strategies for Improving Communication Efficiency. Proceedings of the NIPS Workshop on Private Multi-Party Machine Learning 2016, Barcelona, Spain.
23. Park, S., Shin, D., Chung, J., and Lee, N. (2023, January 23–29). FedFwd: Federated Learning without Backpropagation. Proceedings of the Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities 2023, Honolulu, HI, USA.
24. Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. (2006). Greedy Layer-Wise Training of Deep Networks. Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference, MIT Press.10.7551/mitpress/7503.003.0024 ↗
25. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
26. Coates, A., and Ng, A.Y. (2012). Learning Feature Representations with k-means. Neural Networks: Tricks of the Trade: Second Edition, Springer.10.1007/978-3-642-35289-8_30 ↗
27. Ha, D., Kim, M., Moon, K., and Jeong, C.Y. (2021). Accelerating on-device learning with layer-wise processor selection method on unified memory. Sensors, 21.10.3390/s21072364 ↗
28. Loshchilov, I., and Hutter, F. (2019, January 6–9). Decoupled Weight Decay Regularization. Proceedings of the International Conference on Learning Representations 2019, New Orleans, LA, USA.
29. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
30. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
31. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
32. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto.
33. Biewald, L. (2025, June 08). Experiment Tracking with Weights and Biases. Available online: https://www.wandb.com.
34. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the Artificial Intelligence and Statistics PMLR, Ft. Lauderdale, FL, USA.