Deep Learning Methods for Microstructural Image Analysis: The State-of-the-Art and Future Perspectives

1. McCauley JW (2016) An introduction to materials by design including a dynamic stress environment. Eng Ceram Curr Stat Future Prospect 1–2810.1002/9781119100430.ch1 ↗

2. Ueno T, Rhone TD, Hou Z, Mizoguchi T, Tsuda K (2016) Combo: an efficient Bayesian optimization library for materials science. Mater Discov 4:18–2110.1016/j.md.2016.04.001 ↗

3. Alberi K, Nardelli MB, Zakutayev A, Mitas L, Curtarolo S, Jain A, Fornari M, Marzari N, Takeuchi I, Green ML et al (2018) The 2019 materials by design roadmap. J Phys D Appl Phys 52(1):01300110.1088/1361-6463/aad926 ↗

4. Kordijazi A, Zhao T, Zhang J, Alrfou K, Rohatgi P (2021) A review of application of machine learning in design, synthesis, and characterization of metal matrix composites: current status and emerging applications. JOM 73(7):2060–207410.1007/s11837-021-04701-2 ↗

5. Kalidindi SR (2013) Microstructure informatics. In: Informatics for materials science and engineering. Elsevier, 443–46610.1016/B978-0-12-394399-6.00018-7 ↗

6. Liu H, Yucel B, Wheeler D, Ganapathysubramanian B, Kalidindi SR, Wodo O (2022) How important is microstructural feature selection for data-driven structure-property mapping? MRS Commun 12(1):95–10310.1557/s43579-021-00147-4 ↗

7. Holm EA, Cohn R, Gao N, Kitahara AR, Matson TP, Lei B, Yarasi SR (2020) Overview: computer vision and machine learning for microstructural characterization and analysis. Metall Mater Trans A 51:5985–599910.1007/s11661-020-06008-4 ↗

8. Yang Z, Li X, Catherine Brinson L, Choudhary AN, Chen W, Agrawal A (2018) Microstructural materials design via deep adversarial learning methodology. J Mech Des 140(11):11141610.1115/1.4041371 ↗

9. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–232410.1109/5.726791 ↗

10. Heo B, Yun S, Han D, Chun S, Choe J, Oh SJ (2021) Rethinking spatial dimensions of vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, 11936–1194510.1109/ICCV48922.2021.01172 ↗

11. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

12. Cecen A, Dai H, Yabansu YC, Kalidindi SR, Song L (2018) Material structure-property linkages using three-dimensional convolutional neural networks. Acta Mater 146:76–8410.1016/j.actamat.2017.11.053 ↗

13. Yang Z, Yabansu YC, Al-Bahrani R, Liao W-K, Choudhary AN, Kalidindi SR, Agrawal A (2018) Deep learning approaches for mining structure-property linkages in high contrast composites from simulation datasets. Comput Mater Sci 151:278–28710.1016/j.commatsci.2018.05.014 ↗

14. Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Van Esesn BC, Awwal AAS, Asari VK (2018) The history began from alexnet: a comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164

15. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–11710.1016/j.neunet.2014.09.003 ↗

16. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25

17. Kelleher JD, Mac Namee B, D’Arcy A (2020) Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies. MIT Press

18. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 580–58710.1109/CVPR.2014.81 ↗

19. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks in Computer Vision-ECCV, 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I 13. 818–83310.1007/978-3-319-10590-1_53 ↗

20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 770–77810.1109/CVPR.2016.90 ↗

21. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

22. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 1–910.1109/CVPR.2015.7298594 ↗

23. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 3431–344010.1109/CVPR.2015.7298965 ↗

24. Veit A, Wilber MJ, Belongie S (2016) Residual networks behave like ensembles of relatively shallow networks. Adv Neural Inf Process Syst 29

25. Abdi M, Nahavandi S (2016) Multi-residual networks: improving the speed and accuracy of residual networks. arXiv preprint arXiv:1609.05672

26. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention-MICCAI, 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18. Springer 2015:234–24110.1007/978-3-319-24574-4_28 ↗

27. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481–249510.1109/TPAMI.2016.2644615 ↗

28. Rühle B, Krumrey JF, Hodoroaba V-D (2021) Workflow towards automated segmentation of agglomerated, non-spherical particles from electron microscopy images using artificial neural networks. Sci Rep 11(1):494210.1038/s41598-021-84287-6 ↗

29. Iqbal T, Ali H (2018) Generative adversarial network for medical images (MI-GAN). J Med Syst 42:1–1110.1007/s10916-018-1072-9 ↗

30. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International conference on machine learning. PMLR, 214–223

31. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 4681–469010.1109/CVPR.2017.19 ↗

32. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, 2223–223210.1109/ICCV.2017.244 ↗

33. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784

34. Higgins I, Pal A, Rusu A, Matthey L, Burgess C, Pritzel A, Botvinick M, Blundell C, Lerchner A (2017) Darla: improving zero-shot transfer in reinforcement learning. In: International Conference on Machine Learning. PMLR, 1480–1490

35. Kingma DP, Welling M (2013) Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114

36. Guo M-H, Xu T-X, Liu J-J, Liu Z-N, Jiang P-T, Mu T-J, Zhang S-H, Martin RR, Cheng M-M, Hu S-M (2022) Attention mechanisms in computer vision: a survey. Comput visual Media 8(3):331–36810.1007/s41095-022-0271-y ↗

37. Khan S, Naseer M, Hayat M, Zamir SW, Khan FS, Shah M (2022) Transformers in vision: a survey. ACM Comput Surv (CSUR) 54(10s):1–4110.1145/3505244 ↗

38. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

39. Ye L, Rochan M, Liu Z, Wang Y (2019) Cross-modal self-attention network for referring image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10502–1051110.1109/CVPR.2019.01075 ↗

40. Zhu X, Su W, Lu L, Li B, Wang X, Dai J (2020) Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159

41. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision. Springer, 213–22910.1007/978-3-030-58452-8_13 ↗

42. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) Self-attention generative adversarial networks. In: International conference on machine learning. PMLR, 7354–7363

43. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, 10012–1002210.1109/ICCV48922.2021.00986 ↗

44. Han K, Xiao A, Wu E, Guo J, Xu C, Wang Y (2021) Transformer in transformer. Adv Neural Inf Process Syst 34:15908–15 919

45. Chu X, Tian Z, Wang Y, Zhang B, Ren H, Wei X, Xia H, Shen C (2021) Twins: revisiting the design of spatial attention in vision transformers. Adv Neural Inf Process Syst 34:9355–9366

46. Huang Z, Ben Y, Luo G, Cheng P, Yu G, Fu B (2021) Shuffle transformer: rethinking spatial shuffle for vision transformer. arXiv preprint arXiv:2106.03650

47. Larmuseau M, Sluydts M, Theuwissen K, Duprez L, Dhaene T, Cottenier S (2020) Compact representations of microstructure images using triplet networks. npj Comput Mater 6(1):15610.1038/s41524-020-00423-2 ↗

48. Campbell A, Murray P, Yakushina E, Marshall S, Ion W (2018) New methods for automatic quantification of microstructural features using digital image processing. Mater Design 141:395–40610.1016/j.matdes.2017.12.049 ↗

49. Duval L, Moreaud M, Couprie C, Jeulin D, Talbot H, Angulo J (2014) Image processing for materials characterization: issues, challenges and opportunities. In: IEEE international conference on image processing (icip). IEEE 2014:4862–486610.1109/ICIP.2014.7025985 ↗

50. Lindemann GR, Chao P, Nikitin V, De Andrade V, De Graef M, Shahani AJ (2024) Complexity and evolution of a three-phase eutectic during coarsening uncovered by 4d nano-imaging. Acta Mater 266, 119684. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S135964542400037510.1016/j.actamat.2024.119684 ↗

51. Marturi N, Dembélé S, Piat N (2014) Scanning electron microscope image signal-to-noise ratio monitoring for micro-nanomanipulation. Scanning J Scanning Microsc 36(4):419–429

52. Yang T, Luo Y, Ji W, Yang G (2021) Advancing biological super-resolution microscopy through deep learning: a brief review. Biophys Rep 7(4):25310.52601/bpr.2021.210019 ↗

53. Chen X, Chen J, Han X, Zhao C, Zhang D, Zhu K, Su Y (2020) A light-weighted CNN model for wafer structural defect detection. IEEE Access 8:24 006-24 01810.1109/ACCESS.2020.2970461 ↗

54. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 4510–452010.1109/CVPR.2018.00474 ↗

55. Gupta R, Anand V, Gupta S, Koundal D (2023) Deep learning model for defect analysis in industry using casting images. Expert Syst Appl 232:12075810.1016/j.eswa.2023.120758 ↗

56. Mulewicz B, Korpala G, Kusiak J, Prahl U (2019) Autonomous interpretation of the microstructure of steels and special alloys. In Materials science forum, vol. 949. Trans Tech Publ, 24–3110.4028/www.scientific.net/MSF.949.24 ↗

57. Elbana R, Mostafa R, Elkeran A (2020) Data processing for automatic classification of spheroidite microstructure using deep learning based on fcnns. Int J Mech Mechatron Eng 20(03):18–31

58. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

59. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 1251–125810.1109/CVPR.2017.195 ↗

60. Banerjee D, Sparks TD (2022) Comparing transfer learning to feature optimization in microstructure classification. Iscience, 25(2)10.1016/j.isci.2022.103774 ↗

61. Christen P, Hand DJ, Kirielle N (2023) A review of the F-measure: its history, properties, criticism, and alternatives. ACM Comput Surv 56(3). [Online]. https://doi.org/10.1145/360636710.1145/3606367 ↗

62. Muñoz-Rodenas J, García-Sevilla F, Coello-Sobrino J, Martínez-Martínez A, Miguel-Eguía V (2023) Effectiveness of machine-learning and deep-learning strategies for the classification of heat treatments applied to low-carbon steels based on microstructural analysis. Appl Sci 13(6):347910.3390/app13063479 ↗

63. Baskaran A, Kane G, Biggs K, Hull R, Lewis D (2020) Adaptive characterization of microstructure dataset using a two stage machine learning approach. Comput Mater Sci 177:10959310.1016/j.commatsci.2020.109593 ↗

64. Luo Q, Holm EA, Wang C (2021) A transfer learning approach for improved classification of carbon nanomaterials from tem images. Nanoscale Adv 3(1):206–21310.1039/D0NA00634C ↗

65. Ivo R F, Rodrigues D d A, Bezerra G M, Freitas F N, de Abreu H F G, Rebouças Filho P P (2020) Non-grain oriented electrical steel photomicrograph classification using transfer learning. J Market Res 9(4):8580–8591

66. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 4700–470810.1109/CVPR.2017.243 ↗

67. Xiang S, Chen X, Fan Z, Chen T, Lian X (2022) A deep learning-aided prediction approach for creep rupture time of Fe–Cr–Ni heat-resistant alloys by integrating textual and visual features. J Market Res 18:268–281

68. Cheloee Darabi A, Rastgordani S, Khoshbin M, Guski V, Schmauder S (2023) Hybrid data-driven deep learning framework for material mechanical properties prediction with the focus on dual-phase steel microstructures. Materials 16(1):44710.3390/ma16010447 ↗

69. Sadre R, Ophus C, Butko A, Weber GH (2021) Deep learning segmentation of complex features in atomic-resolution phase-contrast transmission electron microscopy images. Microsc Microanal 27(4):804–81410.1017/S1431927621000167 ↗

70. Alrfou K, Kordijazi A, Rohatgi P, Zhao T (2022) Synergy of unsupervised and supervised machine learning methods for the segmentation of the graphite particles in the microstructure of ductile iron. Mater Today Commun 30:10317410.1016/j.mtcomm.2022.103174 ↗

71. Shah A, Schiller JA, Ramos I, Serrano J, Adams DK, Tawfick S, Ertekin E (2023) Automated image segmentation of scanning electron microscopy images of graphene using u-net neural network. Mater Today Commun 35:10612710.1016/j.mtcomm.2023.106127 ↗

72. Ajioka F, Wang Z-L, Ogawa T, Adachi Y (2020) Development of high accuracy segmentation model for microstructure of steel by deep learning. ISIJ Int 60(5):954–95910.2355/isijinternational.ISIJINT-2019-568 ↗

73. Azimi SM, Britz D, Engstler M, Fritz M, Mücklich F (2018) Advanced steel microstructural classification by deep learning methods. Sci Rep 8(1):212810.1038/s41598-018-20037-5 ↗

74. DeCost BL, Lei B, Francis T, Holm EA (2019) High throughput quantitative metallography for complex microstructures using deep learning: a case study in ultrahigh carbon steel. Microsc Microanal 25(1):21–2910.1017/S1431927618015635 ↗

75. Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, Wang M (2022) Swin-unet: unet-like pure transformer for medical image segmentation. In: European conference on computer vision. Springer, 205–21810.1007/978-3-031-25066-8_9 ↗

76. Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4. Springer 2018:3–1110.1007/978-3-030-00889-5_1 ↗

77. Fotos G, Campbell A, Murray P, Yakushina E (2023) Deep learning enhanced watershed for microstructural analysis using a boundary class semantic segmentation. J Mater Sci 58(36):14 390-14 41010.1007/s10853-023-08901-w ↗

78. Ma DW, Wu XJ, Yang H (2019) Efficient small object detection with an improved region proposal networks. In: IOP Conference Series: Materials Science and Engineering, vol. 533, no. 1. IOP Publishing, 01206210.1088/1757-899X/533/1/012062 ↗

79. Girshick R (2015) Fast r-cnn. In: IEEE international conference on computer vision (ICCV) 2015:1440–144810.1109/ICCV.2015.169 ↗

80. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Cortes C, Lawrence N, Lee D, Sugiyama M, and Garnett R (eds) Advances in neural information processing systems, vol. 28. Curran Associates, Inc

81. He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask r-cnn. In: ICCV. IEEE Computer Society, 2980–298810.1109/ICCV.2017.322 ↗

82. Fu L, Yu H, Shah M, Simmons J, Wang S (2020) Crystallographic symmetry for data augmentation in detecting dendrite cores. Electron Imaging 32:1–710.1117/1.JEI.29.2.023020 ↗

83. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28

84. Fehlemann N, Suarez Aguilera AL, Sandfeld S, Bexter F, Neite M, Lenz D, Könemann M, Münstermann S (2023) Identification of martensite bands in dual-phase steels: a deep learning object detection approach using faster region-based-convolutional neural network. Steel Res Int 94:220083610.1002/srin.202200836 ↗

85. Masubuchi S, Watanabe E, Seo Y, Okazaki S, Sasagawa T, Watanabe K, Taniguchi T, Machida T (2020) Deep-learning-based image segmentation integrated with optical microscopy for automatically searching for two-dimensional materials. npj 2d Mater Appl 4:310.1038/s41699-020-0137-z ↗

86. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, 2961–296910.1109/ICCV.2017.322 ↗

87. Agbozo R, Jin W (2020) Quantitative metallographic analysis of gcr15 microstructure using mask r-cnn. J Korean Soc Precis Eng 37(5):361–36910.7736/JKSPE.019.144 ↗

88. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: Computer Vision-ECCV, 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part V 13. 740–75510.1007/978-3-319-10602-1_48 ↗

89. Li M, Chen D, Liu S, Liu F (2021) Prior mask r-cnn based on graph cuts loss and size input for precipitation measurement. IEEE Trans Instrum Meas 70:1–15

90. Wang Z, Fan L, Lu Y, Mao J, Huang L, Zhou J (2022) Tesn: transformers enhanced segmentation network for accurate nanoparticle size measurement of TEM images. Powder Technol 407:11767310.1016/j.powtec.2022.117673 ↗

91. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence N, and Weinberger K (eds) Advances in neural information processing systems vol. 27. Curran Associates, Inc

92. Kingma DP, Welling M (2019) An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4), 307–392, . [Online]. Available: http://dx.doi.org/10.1561/220000005610.1561/2200000056 ↗

93. Chen L, Zhang W, Nie Z, Li S, Pan F (2021) Generative models for inverse design of inorganic solid materials. J Mater Inform 1(4)10.20517/jmi.2021.07 ↗

94. Long T, Fortunato NM, Opahle I, Zhang Y, Samathrakis I, Shen C, Gutfleisch O, Zhang H (2021) Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures. npj Comput Mater 7(1):6610.1038/s41524-021-00526-4 ↗

95. Chun S, Roy S, Nguyen YT, Choi JB, Udaykumar H, Baek SS (2020) Deep learning for synthetic microstructure generation in a materials-by-design framework for heterogeneous energetic materials. Sci Rep 10(1):1330710.1038/s41598-020-70149-0 ↗

96. Kim SW, Kang S-H, Kim S-J, Lee S (2021) Estimating the phase volume fraction of multi-phase steel via unsupervised deep learning. Sci Rep 11(1):590210.1038/s41598-021-85407-y ↗

97. Ibtehaz N, Rahman MS (2020) Multiresunet: rethinking the u-net architecture for multimodal biomedical image segmentation. Neural Netw 121:74–8710.1016/j.neunet.2019.08.025 ↗

98. Cao Z, Liu Q, Liu Q, Yu X, Kruzic JJ, Li X (2023) A machine learning method to quantitatively predict alpha phase morphology in additively manufactured Ti–6Al–4V. npj Comput Mater 9(1):19510.1038/s41524-023-01152-y ↗

99. Ding G, Liu Y, Zhang R, Xin HL (2019) A joint deep learning model to recover information and reduce artifacts in missing-wedge sinograms for electron tomography and beyond. Sci Rep 9(1):1280310.1038/s41598-019-49267-x ↗

100. Yang X, De Andrade V, Scullin W, Dyer EL, Kasthuri N, De Carlo F, Gürsoy D (2018) Low-dose X-ray tomography through a deep convolutional neural network. Sci Rep 8(1):257510.1038/s41598-018-19426-7 ↗