Review of Image Augmentation Used in Deep Learning-Based Material Microscopic Image Segmentation
Source abstract
Abstract
The deep learning-based image segmentation approach has evolved into the mainstream of target detection and shape characterization in microscopic image analysis. However, the accuracy and generalizability of deep learning approaches are still hindered by the insufficient data problem that results from the high expense of human and material resources for microscopic image acquisition and annotation. Generally, image augmentation can increase the amount of data in a short time by means of mathematical simulation, and has become a necessary module for deep learning-based material microscopic image analysis. In this work, we first review the commonly used image augmentation methods and divide more than 60 basic image augmentation methods into eleven categories based on different implementation strategies. Secondly, we conduct experiments to verify the effectiveness of various basic image augmentation methods for the image segmentation task of two classical material microscopic images using evaluation metrics with different applicabilities. The U-Net model was selected as a representative benchmark model for image segmentation tasks, as it is the classic and most widely used model in this field. We utilize this model to verify the improvement of segmentation performance by various augmentation methods. Then, we discuss the advantages and applicability of various image augmentation methods in the material microscopic image segmentation task. The evaluation experiments and conclusions in this work can serve as a guide for the creation of intelligent modeling frameworks in the materials industry.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Processing Techniques and Applications · Cell Image Analysis Techniques · Machine Learning in Materials Science
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Beijing Advanced Innovation Center for Materials Genome Engineering, University of Science and Technology Beijing, Beijing 100083, China
Collaborative Innovation Center of Steel Generic Technology, University of Science and Technology Beijing, Beijing 100083, China
School of Intelligence Science and Technology, University of Science and Technology Beijing, Beijing 100083, China
Hebei Iron and Steel Group Co., Ltd., Shijiazhuang 050000, China
Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China
Shunde Graduate School, University of Science and Technology Beijing, Foshan 528399, China
Handon Iron and Steel Company, Hebei Iron and Steel Group Co., Ltd., Shijiazhuang 056015, China
Institute for Advanced Materials and Technology, University of Science and Technology Beijing, Beijing 100083, China
Funding
National Key Research and Development Program of China · National Foreign Expert Program · Scientific and Technological Innovation Foundation of Shunde Graduate School, USTB · Fundamental Research Funds for the Central Universities of China · Interdisciplinary Research Project for Young Teachers of USTB (Fundamental Research Funds for the Central Universities)
References (66)
Show source-provided references
1. Recent developments in advanced aircraft aluminium alloys10.1016/j.matdes.2013.12.002 ↗
2. Grain boundary stability governs hardening and softening in extremely fine nanograined metals10.1126/science.aal5166 ↗
3. Deep learning10.1038/nature14539 ↗
4. Sonka, M., Hlavac, V., and Boyle, R. (2014). Image Processing, Analysis, and Machine Vision, Cengage Learning.
5. Sesf-fuse: An unsupervised deep model for multi-focus image fusion10.1007/s00521-020-05358-9 ↗
6. End-to-end learning for simultaneously generating decision map and multi-focus image fusion result10.1016/j.neucom.2021.10.115 ↗
7. Ma, B., Ban, X., Huang, H., Chen, Y., Liu, W., and Zhi, Y. (2018). Deep learning-based image segmentation for al-la alloy microscopic images. Symmetry, 10.10.3390/sym10040107 ↗
8. Deep learning-based automatic inpainting for material microscopic images10.1111/jmi.12960 ↗
9. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
10. Boundary learning by using weighted propagation in convolution network10.1016/j.jocs.2022.101709 ↗
11. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
12. Boyuan, M. (2021). Research and Application of Few-Shot Image Segmentation Method for Complex 3D Material Microstructure. [Ph.D. Thesis, University of Science and Technology Beijing].
13. Data augmentation in microscopic images for material data mining10.1038/s41524-020-00392-6 ↗
14. Improving fine-tuning of self-supervised models with Contrastive Initialization10.1016/j.neunet.2022.12.012 ↗
15. Molchanov, D., Ashukha, A., and Vetrov, D. (2017, January 19–25). Variational dropout sparsifies deep neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Melbourne, Australia.
16. Bjorck, N., Gomes, C.P., Selman, B., and Weinberger, K.Q. (2018, January 3–8). Understanding batch normalization. Proceedings of the Advances in Neural Information Processing Systems Conference, Montreal, QC, Canada.
17. A comprehensive survey on transfer learning10.1109/JPROC.2020.3004555 ↗
18. Review of data augmentation for image in deep learning10.11834/jig.200089 ↗
19. A comprehensive survey of image augmentation techniques for deep learning10.1016/j.patcog.2023.109347 ↗
20. A review of medical image data augmentation techniques for deep learning applications10.1111/1754-9485.13261 ↗
21. An adaptive threshold image enhancement algorithm based on histogram equalization
22. Contrast limited adaptive histogram equalization10.1016/B978-0-12-336156-1.50061-6 ↗
23. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, PMLR, Online.
24. Sural, S., Qian, G., and Pramanik, S. (2002, January 22–25). Segmentation and histogram generation using the HSV color space for image retrieval. Proceedings of the IEEE International Conference on Image Processing, Rochester, NY, USA.
25. Taylor, L., and Nitschke, G. (2018, January 18–21). Improving deep learning with generic data augmentation. Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence (SSCI), Bengaluru, India.10.1109/SSCI.2018.8628742 ↗
26. Moreno-Barea, F.J., Strazzera, F., Jerez, J.M., Urda, D., and Franco, L. (2018, January 18–21). Forward noise adjustment scheme for data augmentation. Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence (SSCI), Bengaluru, India.10.1109/SSCI.2018.8628917 ↗
27. Shijie, J., Ping, W., Peiyi, J., and Siping, H. (2017, January 20–22). Research on data augmentation for image classification based on convolution neural networks. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.10.1109/CAC.2017.8243510 ↗
28. A survey on face data augmentation for the training of deep neural networks10.1007/s00521-020-04748-3 ↗
29. Dropout: A simple way to prevent neural networks from overfitting
30. DeVries, T., and Taylor, G.W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv.
31. The effectiveness of image augmentation in deep learning networks for detecting COVID-19: A geometric transformation perspective10.3389/fmed.2021.629134 ↗
32. Farahani, A., Voghoei, S., Rasheed, K., and Arabnia, H.R. (2021). Advances in Data Science and Information Engineering, Springer.
33. Yang, Y., and Soatto, S. (2020, January 13–19). Fda: Fourier domain adaptation for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00414 ↗
34. Yaras, C., Huang, B., Bradbury, K., and Malof, J.M. (2021). Randomized Histogram Matching: A Simple Augmentation for Unsupervised Domain Adaptation in Overhead Imagery. arXiv.
35. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A.A. (2020). Albumentations: Fast and flexible image augmentations. Information, 11.10.3390/info11020125 ↗
36. Principal component analysis10.1016/0169-7439(87)80084-9 ↗
37. Raju, V.G., Lakshmi, K.P., Jain, V.M., Kalidindi, A., and Padma, V. (2020, January 20–22). Study the influence of normalization/transformation process on the accuracy of supervised classification. Proceedings of the 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India.10.1109/ICSSIT48917.2020.9214160 ↗
38. Shaheen, H., Agarwal, S., and Ranjan, P. (2022). Soft Computing: Theories and Applications, Springer.
39. Cubuk, E., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. (2019). Autoaugment: Learning augmentation policies from data. arXiv.10.1109/CVPR.2019.00020 ↗
40. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2018). Do cifar-10 classifiers generalize to cifar-10?. arXiv.
41. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
42. Lim, S., Kim, I., Kim, T., Kim, C., and Kim, S. (2019). Advances in Neural Information Processing Systems, Springer.
43. Hataya, R., Zdenek, J., Yoshizoe, K., and Nakayama, H. (2020, January 23–28). Faster autoaugment: Learning augmentation strategies using backpropagation. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58595-2_1 ↗
44. Cubuk, E.D., Zoph, B., Shlens, J., and Le, Q.V. (2020, January 13–19). Randaugment: Practical automated data augmentation with a reduced search space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00359 ↗
45. Ho, D., Liang, E., Chen, X., Stoica, I., and Abbeel, P. (2019, January 9–15). Population based augmentation: Efficient learning of augmentation policy schedules. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
46. Greedy autoaugment10.1016/j.patrec.2020.08.024 ↗
47. LingChen, T.C., Khonsari, A., Lashkari, A., Nazari, M.R., Sambee, J.S., and Nascimento, M.A. (2020). Uniformaugment: A search-free probabilistic data augmentation approach. arXiv.
48. Gong, C., Wang, D., Li, M., Chandra, V., and Liu, Q. (2021, January 20–25). Keepaugment: A simple information-preserving data augmentation approach. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00111 ↗
49. Zheng, Y., Zhang, Z., Yan, S., and Zhang, M. (2022). Deep autoaugment. arXiv.
50. Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., and Shen, F. (2022). Image Data Augmentation for Deep Learning: A Survey. arXiv.
51. Generative adversarial networks10.1145/3422622 ↗
52. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv.
53. Olaniyi, E., Chen, D., Lu, Y., and Huang, Y. (2022). Generative adversarial networks for image augmentation in agriculture: A systematic review. arXiv.
54. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
55. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
56. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
57. Liu, S., Zhang, J., Chen, Y., Liu, Y., Qin, Z., and Wan, T. (2019, January 12–17). Pixel level data augmentation for semantic image segmentation using generative adversarial networks. Proceedings of the ICASSP 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683590 ↗
58. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B. (2018, January 18–22). High-resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00917 ↗
59. An image augmentation approach using two-stage generative adversarial network for nuclei image segmentation10.1016/j.bspc.2019.101782 ↗
60. Li, R., Bastiani, M., Auer, D., Wagner, C., and Chen, X. (2021, January 12–14). Image Augmentation Using a Task Guided Generative Adversarial Network for Age Estimation on Brain MRI. Proceedings of the Annual Conference on Medical Image Understanding and Analysis, Cambridge, UK.10.1007/978-3-030-80432-9_27 ↗
61. He, X., Wandt, B., and Rhodin, H. (2022, January 18–24). GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00129 ↗
62. Zhao, A., Balakrishnan, G., Durand, F., Guttag, J.V., and Dalca, A.V. (2019, January 15–20). Data augmentation using learned transformations for one-shot medical image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00874 ↗
63. Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. (2017). One-shot learning for semantic segmentation. arXiv.10.5244/C.31.167 ↗
64. Diederik, P., and Jimmy, B. (2014). Adam: A method for stochastic optimization. arXiv.
65. Comparing clusterings—An information based distance10.1016/j.jmva.2006.11.013 ↗
66. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗