Modular Dynamic Neural Network: A Continual Learning Architecture
Source abstract
Abstract
Learning to recognize a new object after having learned to recognize other objects may be a simple task for a human, but not for machines. The present go-to approaches for teaching a machine to recognize a set of objects are based on the use of deep neural networks (DNN). So, intuitively, the solution for teaching new objects on the fly to a machine should be DNN. The problem is that the trained DNN weights used to classify the initial set of objects are extremely fragile, meaning that any change to those weights can severely damage the capacity to perform the initial recognitions; this phenomenon is known as catastrophic forgetting (CF). This paper presents a new (DNN) continual learning (CL) architecture that can deal with CF, the modular dynamic neural network (MDNN). The presented architecture consists of two main components: (a) the ResNet50-based feature extraction component as the backbone; and (b) the modular dynamic classification component, which consists of multiple sub-networks and progressively builds itself up in a tree-like structure that rearranges itself as it learns over time in such a way that each sub-network can function independently. The main contribution of the paper is a new architecture that is strongly based on its modular dynamic training feature. This modular structure allows for new classes to be added while only altering specific sub-networks in such a way that previously known classes are not forgotten. Tests on the CORe50 dataset showed results above the state of the art for CL architectures.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · COVID-19 diagnosis using AI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
LARSYS & ISE, Universidade do Algarve, 8005-139 Faro, Portugal
Funding
Fundação para a Ciência e Tecnologia
References (36)
Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
2. She, Q., Feng, F., Hao, X., Yang, Q., Lan, C., Lomonaco, V., Shi, X., Wang, Z., Guo, Y., and Zhang, Y. (2019). OpenLORIS-Object: A dataset and benchmark towards lifelong object recognition. arXiv.10.1109/ICRA40945.2020.9196887 ↗
3. Continual lifelong learning with neural networks: A review10.1016/j.neunet.2019.01.012 ↗
4. Ebrahimi, S., Meier, F., Calandra, R., Darrell, T., and Rohrbach, M. (2020, January 23–28). Adversarial continual learning. Proceedings of the 16th European Conference Computer Vision, Glasgow, UK.10.1007/978-3-030-58621-8_23 ↗
5. Aljundi, R., Rohrbach, M., and Tuytelaars, T. (2018). Selfless sequential learning. arXiv.
6. Broad learning system: An effective and efficient incremental learning system without the need for deep architecture10.1109/TNNLS.2017.2716952 ↗
7. Chen, Z., and Liu, B. (2018). Lifelong Machine Learning, Morgan & Claypool Publishers. [2nd ed.]. Synthesis Lectures on Artificial Intelligence and Machine Learning.10.1007/978-3-031-01581-6 ↗
8. Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. (2021). A continual learning survey: Defying forgetting in classification tasks. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2021.3057446 ↗
9. Lomonaco, V., and Maltoni, D. (2017, January 13–15). CORe50: A new dataset and benchmark for continuous object recognition. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA.
10. Turner, D., Cardoso, P.J., and Rodrigues, J.M. (2021). Continual Learning for Object Classification: Consolidation and reconsolidation. Perception, in press.
11. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges10.1016/j.inffus.2019.12.004 ↗
12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
13. Krizhevsky, A., Nair, V., and Hinton, G. (2021, October 15). CIFAR-10 and CIFAR-100 Datasets. Available online: https://www.cs.toronto.edu/~kriz/cifar.html.
14. Continuous learning in single-incremental-task scenarios10.1016/j.neunet.2019.03.010 ↗
15. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
16. Learning without Forgetting10.1109/TPAMI.2017.2773081 ↗
17. Zenke, F., Poole, B., and Ganguli, S. (2017, January 6–11). Continual learning through synaptic intelligence. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
18. Rebuffi, S.A., Kolesnikov, A., Sperl, G., and Lampert, C.H. (2017, January 21–26). iCaRL: Incremental Classifier and Representation Learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.587 ↗
19. Mallya, A., and Lazebnik, S. (2018, January 18–22). PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00810 ↗
20. Fast and flexible multi-task classification using conditional neural adaptive processes
21. van de Ven, G.M., and Tolias, A.S. (2019, January 6–9). Three continual learning scenarios and a case for generative replay. Proceedings of the ICLR 2019, New Orleans, LA, USA.
22. Pellegrini, L., Graffieti, G., Lomonaco, V., and Maltoni, D. (2020, January 24–29). Latent Replay for Real-Time Continual Learning. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341460 ↗
23. Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H., and Sanner, S. (2021). Online continual learning in image classification: An empirical survey. arXiv.10.1016/j.neucom.2021.10.021 ↗
24. An analysis of convolutional neural networks for image classification10.1016/j.procs.2018.05.198 ↗
25. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
26. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
27. Tan, M., and Le, Q. (2019, January 9–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
28. Fang, W., Wang, C., Chen, X., Wan, W., Li, H., Zhu, S., Fang, Y., Liu, B., and Hong, Y. (2018, January 10–14). Recognizing global reservoirs from Landsat 8 imageries: A Deep Learning Approach. Proceedings of the AGU Fall Meeting, Washington, DC, USA.
29. Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Jin, H., Petryk, S., Bargal, S.A., and Gonzalez, J.E. (2020). NBDT: Neural-backed decision trees. arXiv.
30. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
31. Hensman, P., and Masko, D. (2015). The Impact of Imbalanced Training Data for Convolutional Neural Networks, KTH Royal Institute of Technology. Degree Project in Computer Science.
32. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
33. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3–7). Caffe: Convolutional Architecture for Fast Feature Embedding. Proceedings of the 22nd ACM international conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654889 ↗
34. Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. arXiv.10.5244/C.28.6 ↗
35. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
36. Antona, M., and Stephanidis, C. (2021). Continual Learning for Object Classification: A Modular Approach. Universal Access in Human-Computer Interaction. Access to Media, Learning and Assistive Environments, Springer. LNCS 12769.