Adaptive Dynamic Search for Multi-Task Learning
Source abstract
Abstract
Multi-task learning (MTL) is a learning strategy for solving multiple tasks simultaneously while exploiting commonalities and differences between tasks for improved learning efficiency and prediction performance. Despite its potential, there remain several major challenges to be addressed. First of all, the task performance degrades when the number of tasks to solve increases or the tasks are less related. In addition, finding the prediction model for each task is typically laborious and can be suboptimal. This nature of manually designing the architecture further aggravates the problem when it comes to solving multiple tasks under different computational budgets. In this work, we propose a novel MTL approach to address these issues. The proposed method learns to search in a finely modularized base network dynamically and to discover an optimal prediction model for each instance of a task on the fly while taking the computational costs of the discovered models into account. We evaluate our learning framework on a diverse set of MTL scenarios comprising standard benchmark datasets. We achieve significant improvements in performance for all tested cases compared with existing MTL alternatives.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Chung-Ang University, Seoul 06974, Republic of Korea
Funding
Ministry of Education of Korea · Chung-Ang University Research
References (47)
Show source-provided references
1. Deep learning10.1038/nature14539 ↗
2. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
3. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
4. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
5. Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., and Torr, P.H. (2016, January 11–14). Fully-convolutional siamese networks for object tracking. Proceedings of the ECCV, Amsterdam, The Netherlands.10.1007/978-3-319-48881-3_56 ↗
6. Huang, G., Liu, Z., Weinberger, K.Q., and van der Maaten, L. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
7. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
8. Kim, E., Ahn, C., and Oh, S. (2018, January 18–23). NestedNet: Learning Nested Sparse Structures in Deep Neural Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00904 ↗
9. Multitask learning10.1023/A:1007379606734 ↗
10. Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv.
11. Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016, January 27–30). Cross-stitch networks for multi-task learning. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.433 ↗
12. Kendall, A., Gal, Y., and Cipolla, R. (2018, January 18–23). Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
13. Mallya, A., and Lazebnik, S. (2018, January 18–23). PackNet: Adding multiple tasks to a single network by iterative pruning. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00810 ↗
14. Adashare: Learning what to share for efficient deep multi-task learning
15. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning
16. Raychaudhuri, D.S., Suh, Y., Schulter, S., Yu, X., Faraki, M., Roy-Chowdhury, A.K., and Chandraker, M. (2022, January 18–24). Controllable Dynamic Multi-Task Architectures. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01068 ↗
17. Rosenbaum, C., Klinger, T., and Riemer, M. (2017, January 24–26). Routing networks: Adaptive selection of non-linear functions for multi-task learning. Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), Toulon, France.
18. Zhao, X., Li, H., Shen, X., Liang, X., and Wu, Y. (2018, January 8–14). A Modulation Module for Multi-task Learning with Applications in Image Retrieval. Proceedings of the ECCV, 15th European Conference, Munich, Germany.10.1007/978-3-030-01246-5_25 ↗
19. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014, January 21–26). Decaf: A deep convolutional activation feature for generic visual recognition. Proceedings of the ICML, 31st International Conference on International Conference on Machine Learning, Beijing, China.
20. Yang, Y., and Hospedales, T.M. (2016). Trace norm regularised deep multi-task learning. arXiv.
21. Jou, B., and Chang, S.F. (2016, January 15–19). Deep cross residual learning for multitask visual recognition. Proceedings of the 24th ACM International Conference on Multimedia, ACM, Amsterdam, The Netherlands.10.1145/2964284.2964309 ↗
22. Zamir, A.R., Wu, T.L., Sun, L., Shen, W.B., Shi, B.E., Malik, J., and Savarese, S. (2017, January 21–26). Feedback networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.196 ↗
23. Mallya, A., Davis, D., and Lazebnik, S. (2018, January 8–14). Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights. Proceedings of the ECCV, 15th European Conference, Munich, Germany.10.1007/978-3-030-01225-0_5 ↗
24. Kokkinos, I. (2017, January 21–26). Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.579 ↗
25. Kim, E., Ahn, C., Torr, P.H., and Oh, S. (2019, January 15–20). Deep Virtual Networks for Memory Efficient Inference of Multiple Tasks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00282 ↗
26. Zoph, B., and Le, Q.V. (2017, January 24–26). Neural architecture search with reinforcement learning. Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), Toulon, France.
27. Lin, J., Rao, Y., Lu, J., and Zhou, J. (2017, January 4–9). Runtime neural pruning. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
28. Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L.S., Grauman, K., and Feris, R. (2018, January 18–23). Blockdrop: Dynamic inference paths in residual networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00919 ↗
29. Liu, H., Simonyan, K., and Yang, Y. (May, January 30). Darts: Differentiable architecture search. Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada.
30. Wang, X., Yu, F., Dou, Z.Y., Darrell, T., and Gonzalez, J.E. (2018, January 8–14). Skipnet: Learning dynamic routing in convolutional networks. Proceedings of the ECCV, 15th European Conference, Munich, Germany.10.1007/978-3-030-01261-8_25 ↗
31. Ahn, C., Kim, E., and Oh, S. (November, January 27). Deep Elastic Networks with Model Selection for Multi-Task Learning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00663 ↗
32. Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., and Dean, J. (2018, January 10–15). Efficient neural architecture search via parameter sharing. Proceedings of the 35th International Conference on Machine Learning (ICML 2018), Stockholm, Sweden.
33. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv.
34. Strezoski, G., Noord, N.V., and Worring, M. (November, January 27). Many task learning with task routing. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00146 ↗
35. Bragman, F.J., Tanno, R., Ourselin, S., Alexander, D.C., and Cardoso, J. (November, January 27). Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
36. Zagoruyko, S., and Komodakis, N. (2016, January 19–22). Wide residual networks. Proceedings of the British Machine Vision Conference (BMVC 2016), New York, NY, USA.10.5244/C.30.87 ↗
37. Larsson, G., Maire, M., and Shakhnarovich, G. (2017, January 24–26). FractalNet: Ultra-deep neural networks without residuals. Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), Toulon, France.
38. Sutton, R.S., McAllester, D.A., Singh, S.P., and Mansour, Y. (2000, January 1). Policy gradient methods for reinforcement learning with function approximation. Proceedings of the 13th International Conference on Neural Information Processing Systems, Cambridge, MA, USA.
39. The monte carlo method10.1080/01621459.1949.10483310 ↗
40. Sutton, R.S., and Barto, A.G. (1998). Introduction to Reinforcement Learning, MIT Press Cambridge.
41. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
42. Coates, A., Lee, H., and Ng, A.Y. (2011, January 11–13). An Analysis of Single Layer Networks in Unsupervised Feature Learning. Proceedings of the AISTATS, 14th International Conference on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, USA.
43. Vinyals, O., Blundell, C., Lillicrap, T., and Wierstra, D. (2016, January 9). Matching networks for one shot learning. Proceedings of the NIPS, Workshop on Interpretable Machine Learning for Complex Systems, Barcelona, Spain.
44. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K.Q. (2016., January 11–14). Deep networks with stochastic depth. Proceedings of the ECCV, 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_39 ↗
45. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010), Sardinia, Italy.
46. A method for unconstrained convex minimization problem with the rate of convergence O (1/k 2)
47. Kingma, D.P., and Ba, J. (2014, January 14–16). Adam: A method for stochastic optimization. Proceedings of the 2nd International Conference on Learning Representations, Banff, AB, Canada.