Mitigating the Stability–Plasticity Trade-Off in Neural Networks via Shared Extractors in Class-Incremental Learning
Source abstract
Abstract
Humans learn new tasks without forgetting, but neural networks suffer from catastrophic forgetting when trained sequentially. Dynamic expandable networks attempt to address this by assigning each task its own feature extractor and freezing previous ones to preserve past knowledge. While effective for retaining old tasks, this design leads to rapid parameter growth, and frozen extractors never adapt to future data, often producing irrelevant features that degrade later performance. To overcome these limitations, we propose Task-Sharing Distillation (TSD), which reduces the number of extractors by allowing multiple tasks to share one extractor and consolidating them through distillation. We study two strategies: (1) grouped rolling consolidation, which groups consecutive tasks and consolidates them into a shared extractor, and (2) a fixed-size pooling with similarity-based consolidation, where new tasks are merged into the most compatible extractor in a fixed pool according to prototype similarity. Experiments on the CIFAR-100 and ImageNet-100 datasets show that TSD maintains strong performance across tasks, demonstrating that careful feature sharing is more effective than simply adding more extractors. On ImageNet-100, our method achieves 2.5% higher average accuracy than DER while using fewer feature extractors.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Machine Learning and ELM · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Lab of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China
School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
China Mobile (HangZhou) Information Technology Co., Ltd., Hangzhou 311121, China
School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China
References (41)
Show source-provided references
1. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
2. Rebuffi, S.A., Kolesnikov, A., Sperl, G., and Lampert, C.H. (2017, January 21–26). iCaRL: Incremental Classifier and Representation Learning. Proceedings of the 27th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.587 ↗
3. Three types of incremental learning10.1038/s42256-022-00568-3 ↗
4. Class-Incremental Learning: A Survey10.1109/TPAMI.2024.3429383 ↗
5. Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. (2023, January 18–22). CODA-Prompt: Continual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning. Proceedings of the 36th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01146 ↗
6. Zhou, D.W., Cai, Z.W., Ye, H.J., Zhang, L., and Zhan, D.C. (2025). Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning. arXiv.10.1109/CVPR52734.2025.01913 ↗
7. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. Proceedings of the 14th European Conference on Computer Vision (ECCV), Springer.
8. Lee, K., Lee, K., Shin, J., and Lee, H. (November, January 27). Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild. Proceedings of the 27th IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
9. Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning. Proceedings of the 16th European Conference on Computer Vision (ECCV), Springer.
10. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). Lifelong Learning via Progressive Distillation and Retrospection. Proceedings of the Computer Vision—ECCV 2018, Springer.10.1007/978-3-030-01249-6 ↗
11. Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. (2020, January 6–12). Dark Experience for General Continual Learning: A Strong, Simple Baseline. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Online.
12. Class-Incremental Continual Learning into the Extended Der-Verse10.1109/TPAMI.2022.3206549 ↗
13. Dhar, P., Singh, R.V., Peng, K.C., Wu, Z., and Chellappa, R. (2019). Learning without Memorizing. arXiv.10.1109/CVPR.2019.00528 ↗
14. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
15. Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., and Fu, Y. (2019, January 15–20). Large Scale Incremental Learning. Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00046 ↗
16. Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S.T. (2020, January 14–19). Maintaining Discrimination and Fairness in Class Incremental Learning. Proceedings of the 33rd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01322 ↗
17. Liu, Y., Schiele, B., and Sun, Q. (2021, January 20–25). Adaptive Aggregation Networks for Class-Incremental Learning. Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00257 ↗
18. Pham, Q., Liu, C., and Hoi, S.C. (2021, January 6–14). DualNet: Continual learning, fast and slow. Proceedings of the 35th International Conference on Neural Information Processing Systems, Red Hook, NY, USA. NIPS ’21.
19. Why There are Complementary Learning Systems in the Hippocampus and Neocortex: Insights from the Successes and Failures of Connectionist Models of Learning and Memory10.1037/0033-295X.102.3.419 ↗
20. Shin, H., Lee, J.K., Kim, J., and Kim, J. (2017, January 4–9). Continual Learning with Deep Generative Replay. Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA. NIPS’17.
21. Xiang, Y., Fu, Y., Ji, P., and Huang, H. (November, January 27). Incremental Learning Using Conditional Adversarial Networks. Proceedings of the 32nd IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
22. Chen, H., Wang, Y., Fan, Y., Jiang, G., and Hu, Q. (2025). Reducing Class-wise Confusion for Incremental Learning with Disentangled Manifolds. arXiv.10.1109/CVPR52734.2025.00946 ↗
23. Yan, S., Xie, J., and He, X. (2021, January 19–25). 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Online.
24. Douillard, A., Ramé, A., Couairon, G., and Cord, M. (2022, January 18–24). DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion. Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00907 ↗
25. Zheng, B., Zhou, D.W., Ye, H.J., and Zhan, D.C. (2025). Task-Agnostic Guided Feature Expansion for Class-Incremental Learning. arXiv.10.1109/CVPR52734.2025.00944 ↗
26. Wang, F.Y., Zhou, D.W., Ye, H.J., and Zhan, D.C. (2022, January 23–27). FOSTER: Feature Boosting and Compression for Class-Incremental Learning. Proceedings of the 17th European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19806-9_23 ↗
27. Zhou, D.W., Wang, Q.W., Ye, H.J., and Zhan, D.C. (2023, January 1–5). A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. Proceedings of the 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda.
28. Rypeść, G., Cygert, S., Khan, V., Trzcinski, T., Zieliński, B., and Twardowski, B. (2024, January 7–11). Divide and Not Forget: Ensemble of Selectively Trained Experts in Continual Learning. Proceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria.
29. Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., and Dy, J. (2022, January 23–27). DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning. Proceedings of the 17th European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19809-0_36 ↗
30. Wang, Y., Huang, Z., and Hong, X. (December, January 28). S-Prompts Learning with Pre-Trained Transformers: An Occam’s Razor for Domain Incremental Learning. Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
31. Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. (2022, January 18–24). Learning to Prompt for Continual Learning. Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00024 ↗
32. Wang, L., Xie, J., Zhang, X., Huang, M., Su, H., and Zhu, J. (2023, January 10–16). Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA. NIPS ’23.
33. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020, January 6–12). Supervised Contrastive Learning. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), Online.
34. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images. In Technical Report. University of Toronto.
35. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
36. Welling, M. (2009, January 14–18). Herding Dynamical Weights to Learn. Proceedings of the 26th International Conference on Machine Learning (ICML), Montreal, QC, Canada.10.1145/1553374.1553517 ↗
37. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
38. Rajasegaran, J., Hayat, M., Khan, S., Khan, F.S., and Shao, L. (2019, January 8–14). Random Path Selection for Incremental Learning. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada.
39. Wang, F.Y., Zhou, D.W., Liu, L., Ye, H.J., Bian, Y., Zhan, D.C., and Zhao, P. (2023, January 1–5). BEEF: Bi-Compatible Class-Incremental Learning via Energy-Based Expansion and Fusion. Proceedings of the The 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda.
40. Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). Topology-Preserving Class-Incremental Learning. Proceedings of the 16th European Conference on Computer Vision (ECCV), Springer.10.1007/978-3-030-58589-1 ↗
41. Zhou, D.W., Ye, H.J., and Zhan, D.C. (2021, January 20–24). Co-Transport for Class-Incremental Learning. Proceedings of the 29th ACM International Conference on Multimedia, New York, NY, USA. MM’21.10.1145/3474085.3475306 ↗