Microstructure segmentation with deep learning encoders pre-trained on a large microscopy dataset
Source abstract
Abstract
Abstract This study examined the improvement of microscopy segmentation intersection over union accuracy by transfer learning from a large dataset of microscopy images called MicroNet. Many neural network encoder architectures were trained on over 100,000 labeled microscopy images from 54 material classes. These pre-trained encoders were then embedded into multiple segmentation architectures including UNet and DeepLabV3+ to evaluate segmentation performance on created benchmark microscopy datasets. Compared to ImageNet pre-training, models pre-trained on MicroNet generalized better to out-of-distribution micrographs taken under different imaging and sample conditions and were more accurate with less training data. When training with only a single Ni-superalloy image, pre-training on MicroNet produced a 72.2% reduction in relative intersection over union error. These results suggest that transfer learning from large in-domain datasets generate models with learned feature representations that are more useful for downstream tasks and will likely improve any microscopy image analysis technique that can leverage pre-trained encoders.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Electron Microscopy Techniques and Applications · Image Processing Techniques and Applications · Advanced X-ray and CT Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (60)
Show source-provided references
1. ASTM E112. Standard test methods for determining average grain size E112-10. ASTM E112-10 96, 1–27 (2010).
2. ASTM E45-18a, A. Standard test methods for determining the inclusion content of steel. ASTM Book of Standards Volume: 03.01 (2018).
3. Stuckner, J., Frei, K., McCue, I., Demkowicz, M. J. & Murayama, M. AQUAMI: An open source Python package and GUI for the automatic quantitative analysis of morphologically complex multiphase materials. Comput. Mater. Sci. 139, 320–329 (2017).10.1016/j.commatsci.2017.08.012 ↗
4. Smith, T. M. et al. Characterization of nanoscale precipitates in superalloy 718 using high resolution SEM imaging. Mater. Charact. 148, 178–187 (2019).10.1016/j.matchar.2018.12.018 ↗
5. Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition 248–255 (IEEE, 2009).10.1109/CVPR.2009.5206848 ↗
6. DeCost, B. L., Lei, B., Francis, T. & Holm, E. A. High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel. Microsc. Microanal. 25, 21–29 (2019).10.1017/S1431927618015635 ↗
7. Roberts, G. et al. DefectNet–a deep convolutional neural network for semantic segmentation of crystallographic defects in advanced microscopy images. Microsc. Microanal. 25, 164–165 (2019).10.1017/S1431927619001557 ↗
8. Goetz, A. et al. Addressing materials’ microstructure diversity using transfer learning. npj Comput. Mater. 8, 1–13 (2022).10.1038/s41524-022-00703-z ↗
9. Senanayake, N. M. & Carter, J. L. W. Computer vision approaches for segmentation of nanoscale precipitates in nickel-based superalloy IN718. Integr. Mater. Manuf. Innov. 9, 446–458 (2020).10.1007/s40192-020-00195-z ↗
10. Cohn, R. et al. Instance segmentation for direct measurements of satellites in metal powders and automated microstructural characterization from image data. JOM 73, 1–14 (2021).10.1007/s11837-021-04713-y ↗
11. Stan, T., Thompson, Z. T. & Voorhees, P. W. Building towards a universal neural network to segment large materials science imaging datasets. in Developments in X-Ray Tomography XII vol. 11113 297–302 (SPIE, 2019).10.1117/12.2525290 ↗
12. Groschner, C., Choi, C., Nguyen, D., Ophus, C. & Scott, M. Machine learning for high throughput HRTEM analysis. Microsc. Microanal. 25, 150–151 (2019).10.1017/S143192761900148X ↗
13. Potocek, P. et al. Sparse scanning electron microscopy data acquisition and deep neural networks for automated segmentation in connectomics. Microsc. Microanal. 26, 403–412 (2020).10.1017/S1431927620001361 ↗
14. Akers, S. et al. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Comput. Mater. 7, 1–9 (2021).10.1038/s41524-021-00652-z ↗
15. Kaufmann, K., Lane, H., Liu, X. & Vecchio, K. S. Efficient few-shot machine learning for classification of EBSD patterns. Sci. Rep. 11, 1–12 (2021).10.1038/s41598-020-79139-8 ↗
16. Durmaz, A. R. et al. A deep learning approach for complex microstructure inference. Nat. Commun. 12, 1–15 (2021).10.1038/s41467-021-26565-5 ↗
17. Cordts, M. et al. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition 3213–3223 (2016).10.1109/CVPR.2016.350 ↗
18. Luo, Q., Holm, E. A. & Wang, C. A transfer learning approach for improved classification of carbon nanomaterials from TEM images. Nanoscale Adv. 3, 206–213 (2021).10.1039/D0NA00634C ↗
19. Tan, M. & Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. in International conference on machine learning 6105–6114 (2019).
20. Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H. Rethinking atrous convolution for semantic image segmentation. arXiv Prepr. arXiv1706.05587 (2017).
21. Pan, S. J. & Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2009).10.1109/TKDE.2009.191 ↗
22. Wang, Z., Dai, Z., Póczos, B. & Carbonell, J. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 11293–11302 (2019).10.1109/CVPR.2019.01155 ↗
23. Noraas, R., Somanath, N., Giering, M. & Olusegun, O. O. Structural material property tailoring using deep neural networks. In AIAA Scitech 2019 Forum 1703 (2019).10.2514/6.2019-1703 ↗
24. Holm, E. A. et al. Overview: Computer vision and machine learning for microstructural characterization and analysis Department of Materials Science and Engineering, Carnegie Mellon University, Pittsburgh, PA USA Department of Materials Science and Engineering, MIT, Ca. 1–5.
25. Larmuseau, M. et al. Compact representations of microstructure images using triplet networks. npj Comput. Mater. 6, 1–11 (2020).10.1038/s41524-020-00423-2 ↗
26. Larmuseau, M. et al. Race against the Machine: can deep learning recognize microstructures as well as the trained human eye? Scr. Mater. 193, 33–37 (2021).10.1016/j.scriptamat.2020.10.026 ↗
27. Kusche, C. et al. Large-area, high-resolution characterisation and classification of damage mechanisms in dual-phase steel using deep learning. PLoS One 14, e0216493 (2019).10.1371/journal.pone.0216493 ↗
28. Azimi, S. M., Britz, D., Engstler, M., Fritz, M. & Mücklich, F. Advanced steel microstructural classification by deep learning methods. Sci. Rep. 8, 1–14 (2018).10.1038/s41598-018-20037-5 ↗
29. Kitahara, A. R. & Holm, E. A. Microstructure cluster analysis with transfer learning and unsupervised learning. Integr. Mater. Manuf. Innov. 7, 148–156 (2018).10.1007/s40192-018-0116-9 ↗
30. DeCost, B. L. et al. UHCSDB: UltraHigh carbon steel micrograph database: tools for exploring large heterogeneous microstructure datasets. Integr. Mater. Manuf. Innov. 6, 197–205 (2017).10.1007/s40192-017-0097-0 ↗
31. Aversa, R., Modarres, M. H., Cozzini, S., Ciancio, R. & Chiusole, A. Data descriptor: The first annotated set of scanning electron microscopy images for nanoscience. Sci. Data 5, 1–10 (2018).10.1038/sdata.2018.172 ↗
32. DeCost, B. L. & Holm, E. A. A large dataset of synthetic SEM images of powder materials and their ground truth 3D structures. Data Br. 9, 727–731 (2016).10.1016/j.dib.2016.10.011 ↗
33. Karimi, A. M. et al. Automated pipeline for photovoltaic module electroluminescence image processing and degradation feature classification. IEEE J. Photovolt. 9, 1324–1335 (2019).10.1109/JPHOTOV.2019.2920732 ↗
34. He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision 1026–1034 (2015).10.1109/ICCV.2015.123 ↗
35. Nair, V. & Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) 807–814 (2010).
36. Szegedy, C. et al. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition 1–9 (2015).10.1109/CVPR.2015.7298594 ↗
37. Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. arXiv Prepr. arXiv1912.01703 (2019).
38. Tan, M. et al. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2820–2828 (2019).10.1109/CVPR.2019.00293 ↗
39. Simonyan, K., & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv Prep. arXiv1409.1556 (2014).
40. Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning 448–456 (PMLR, 2015).
41. Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 4700–4708 (2017).10.1109/CVPR.2017.243 ↗
42. Chen, Y. et al. Dual path networks. Adv. Neural Inf. Process. Syst. 2017-December, 4468–4476 (2017).
43. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016).10.1109/CVPR.2016.90 ↗
44. Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence vol. 31 (2017).10.1609/aaai.v31i1.11231 ↗
45. Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition 1251–1258 (2017).10.1109/CVPR.2017.195 ↗
46. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition 4510–4520 (2018).10.1109/CVPR.2018.00474 ↗
47. Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 1492–1500 (2017).10.1109/CVPR.2017.634 ↗
48. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 7132–7141 (2018).10.1109/CVPR.2018.00745 ↗
49. Yakubovskiy, P. Segmentation Models Pytorch. GitHub Repos. (2020).
50. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv Prepr. arXiv1412.6980 (2014).
51. Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S. & Cardoso, M. J. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep learning in medical image analysis and multimodal learning for clinical decision support 240–248 (Springer, 2017).10.1007/978-3-319-67558-9_28 ↗
52. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention vol. 9351 234–241 (Springer, 2015).10.1007/978-3-319-24574-4_28 ↗
53. Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N. & Liang, J. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans. Med. Imaging 39, 1856–1867 (2019).10.1109/TMI.2019.2959609 ↗
54. Chaurasia, A. & Culurciello, E. Linknet: Exploiting encoder representations for efficient semantic segmentation. In 2017 IEEE Visual Communications and Image Processing (VCIP) 1–4 (IEEE, 2017).10.1109/VCIP.2017.8305148 ↗
55. Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117–2125 (2017).10.1109/CVPR.2017.106 ↗
56. Zhao, H., Shi, J., Qi, X., Wang, X. & Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition 2881–2890 (2017).10.1109/CVPR.2017.660 ↗
57. Li, H., Xiong, P., An, J. & Wang, L. Pyramid attention network for semantic segmentation. arXiv Prepr. arXiv1805.10180 (2018).
58. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) 801–818 (2018).10.1007/978-3-030-01234-2_49 ↗
59. Van der Walt, S. et al. scikit-image: image processing in Python. PeerJ 2, e453 (2014).10.7717/peerj.453 ↗
60. Fiorio, C. & Gustedt, J. Two linear time union-find strategies for image processing. Theor. Comput. Sci. 154, 165–181 (1996).10.1016/0304-3975(94)00262-2 ↗