Gradient Agreement Hinders the Memorization of Noisy Labels
Source abstract
Abstract
The performance of deep neural networks (DNNs) critically relies on high-quality annotations, while training DNNs with noisy labels remains challenging owing to their incredible capacity to memorize the entire training set. In this work, we use two synchronously trained networks to reveal that noisy labels may result in more divergent gradients when updating the parameters. To overcome this, we propose a novel co-training framework named gradient agreement learning (GAL). By dynamically evaluating the gradient agreement coefficient of every pair of parameters from two identical DNNs to determine whether to update them in the training process. GAL can effectively hinder the memorization of noisy labels. Furthermore, we utilize the pseudo labels produced by the two DNNs as the supervision for the training of another network, thereby gaining further improvement by correcting some noisy labels while overcoming the confirmation bias. Extensive experiments on various benchmark datasets demonstrate the superiority of the proposed GAL.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Data Classification · Infrastructure Maintenance and Monitoring · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310007, China
Zhejiang Provincial Key Laboratory for Network Multimedia Technologies, Hangzhou 310007, China
Zhejiang University Embedded System Engineering Research Center, Institute of Advanced Digital Technologies and Instrumentation, Ministry of Education of China, Hangzhou 310007, China
Funding
Fundamental Research Funds for the Central Universities
References (37)
Show source-provided references
1. Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., and Bengio, Y. (2017, January 7–9). A closer look at memorization in deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
2. Toneva, M., Sordoni, A., des Combes, R.T., Trischler, A., Bengio, Y., and Gordon, G. (2019, January 6–9). An empirical study of example forgetting during deep neural network learning. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
3. Zhang, C., Recht, B., Bengio, S., Hardt, M., and Vinyals, O. (2017, January 24–26). Understanding deep learning requires rethinking generalization. Proceedings of the International Conference on Learning Representations, Toulon, France.
4. Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.G. (2022). Learning from noisy labels with deep neural networks: A survey. IEEE Trans. NNLS.10.1109/TNNLS.2022.3152527 ↗
5. Active label cleaning for improved dataset quality under resource constraints10.1038/s41467-022-28818-3 ↗
6. Jiang, L., Zhou, Z., Leung, T., Li, L., and Fei-Fei, L. (2018, January 10–15). MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
7. Malach, E., and Shalev-Shwartz, S. (2017, January 4–7). Decoupling “when to update” from “how to update”. Proceedings of the Conference on Neural Information Processing Systems, Long Beach, CA, USA.
8. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. (2018, January 2–8). Co-teaching: Robust training of deep neural networks with extremely noisy labels. Proceedings of the Conference on Neural Information Processing Systems, Montreal, Canada.
9. Yu, X., Han, B., Yao, J., Niu, G., Tsang, I.W., and Sugiyama, M. (2019, January 9–15). How does disagreement help generalization against label corruption?. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
10. Wei, H., Feng, L., Chen, X., and An, B. (2020, January 14–19). Combating noisy labels by agreement: A joint training method with co-regularization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR42600.2020.01374 ↗
11. Tan, C., Xia, J., Wu, L., and Li, S.Z. (2021, January 20–24). Co-learning: Learning from noisy labels with self-supervision. Proceedings of the ACM International Conference on Multimedia, Chengdu, China.10.1145/3474085.3475622 ↗
12. Jenni, S., and Favaro, P. (2018, January 8–14). Deep bilevel learning. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01249-6_38 ↗
13. Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., and Silberman, N. (2019, January 16–20). Learning from noisy labels by regularized estimation of annotator confusion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01150 ↗
14. Hendrycks, D., Lee, K., and Mazeika, M. (2019, January 9–15). Using pre-training can improve model robustness and uncertainty. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
15. Menon, A.K., Rawat, A.S., Reddi, S.J., and Kumar, S. (2020, January 26–30). Can gradient clipping mitigate label noise?. Proceedings of the International Conference on Learning Representations, Virtual.
16. Xia, X., Liu, T., Han, B., Gong, C., Wang, N., Ge, Z., and Chang, Y. (2021, January 3–5). Robust early-learning: Hindering the memorization of noisy labels. Proceedings of the International Conference on Learning Representations, Virtual.
17. Liu, S., Niles-Weed, J., Razavian, N., and Fernandez-Granda, C. (2020, January 6–12). Early-learning regularization prevents memorization of noisy labels. Proceedings of the Conference on Neural Information Processing Systems, Virtual.
18. Lukasik, M., Bhojanapalli, S., Menon, A., and Kumar, S. (2020, January 26–30). Does label smoothing mitigate label noise?. Proceedings of the International Conference on Learning Representations, Virtual.
19. On Information and Sufficiency10.1214/aoms/1177729694 ↗
20. Yang, X., Song, Z., King, I., and Xu, Z. (2021). A Survey on Deep Semi-supervised Learning. arXiv.10.1109/TKDE.2022.3220219 ↗
21. Self-supervised Learning: Generative or Contrastive
22. Nguyen, D.T., Mummadi, C.K., Ngo, T.P.N., Nguyen, T.H.P., Beggel, L., and Brox, T. (2020, January 26–30). SELF: Learning to filter noisy labels with self-ensembling. Proceedings of the International Conference on Learning Representations, Virtual.
23. Zhou, T., Wang, S., and Bilmes, J. (2021, January 4). Robust curriculum learning: From clean label detection to noisy label self-correction. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
24. Li, J., Socher, R., and Hoi, S. (2020, January 26–30). Dividemix: Learning with noisy labels as semi-supervised learning. Proceedings of the International Conference on Learning Representations, Virtual.
25. Ortego, D., Arazo, E., Albert, P., and O’Connor, N.E. (2021, January 20–25). Multi-objective interpolation training for robustness to label noise. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00654 ↗
26. Li, S., Xia, X., Ge, S., and Liu, T. (2022, January 21–24). Selective-Supervised Contrastive Learning with Noisy Labels. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00041 ↗
27. A study of gaussian mixture models of color and texture features for image classification and segmentation10.1016/j.patcog.2005.10.028 ↗
28. Krizhevsky, A., Nair, V., and Hinton, G. (2021). CIFAR-10, Canadian Institute for Advanced Research.
29. Krizhevsky, A., Nair, V., and Hinton, G. (2021). CIFAR-100, Canadian Institute for Advanced Research.
30. Song, H., Kim, M., and Lee, J. (2019, January 9–15). SELFIE: Refurbishing Unclean Samples for Robust Deep Learning. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
31. Xiao, T., Xia, T., Yang, Y., Huang, C., and Wang, X. (2015, January 8–10). Learning from massive noisy labeled data for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
32. Song, H., Kim, M., Park, D., and Lee, J. (2019, January 9–15). Prestopping: How does early stopping help generalization against label noise?. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
33. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 8–16). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_38 ↗
34. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 26–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
35. TC-Net: Detecting Noisy Labels via Transform Consistency10.1109/TMM.2021.3115635 ↗
36. Kim, T., Ko, J., Choi, J., and Yun, S.Y. (2021, January 7–10). Fine samples for learning with noisy labels. Proceedings of the Conference on Neural Information Processing Systems, Virtual.
37. Wei, J., and Liu, Y. (2021, January 3–5). When Optimizing f-divergence is Robust with Label Noise. Proceedings of the International Conference on Learning Representations, Virtual.