Automated defect analysis in electron microscopic images
Source abstract
Abstract
Abstract Electron microscopy and defect analysis are a cornerstone of materials science, as they offer detailed insights on the microstructure and performance of a wide range of materials and material systems. Building a robust and flexible platform for automated defect recognition and classification in electron microscopy will result in the completion of analysis orders of magnitude faster after images are recorded, or even online during image acquisition. Automated analysis has the potential to be significantly more efficient, accurate, and repeatable than human analysis, and it can scale with the increasingly important methods of automated data generation. Herein, an automated recognition tool is developed based on a computer vison–based approach; it sequentially applies a cascade object detector, convolutional neural network, and local image analysis methods. We demonstrate that the automated tool performs as well as or better than manual human detection in terms of recall and precision and achieves quantitative image/defect analysis metrics close to the human average. The proposed approach works for images of varying contrast, brightness, and magnification. These promising results suggest that this and similar approaches are worth exploring for detecting multiple defect types and have the potential to locate, classify, and measure quantitative features for a range of defect types, materials, and electron microscopic techniques.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Electron and X-Ray Spectroscopy Techniques · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (50)
Show source-provided references
1. Jenkins, M. L. & Kirk, M. A. Characterisation of Radiation Damage by Transmission Electron Microscopy. (Boca Raton, FL, CRC Press, 2000).10.1201/9781420034646 ↗
2. Zinkle, S. J. & Busby, J. T. Structural materials for fission & fusion energy. Mater. Today 12, 12–19 (2009).10.1016/S1369-7021(09)70294-9 ↗
3. Jesse, S. et al. Big data analytics for scanning transmission electron microscopy ptychography. Sci. Rep. 6, 26348 (2016).10.1038/srep26348 ↗
4. Lowe, D. G. Object recognition from local scale-invariant features. In Proc. of the Seventh IEEE International Conference on Computer Vision. 1150–1157 (IEEE, Kerkyra, Greece, 1999).10.1109/ICCV.1999.790410 ↗
5. Zhao, J., Kong, Q.-J., Zhao, X., Liu, J. & Liu, Y. A method for detection and classification of glass defects in low resolution images. In 2011 Sixth International Conference on Image and Graphics (ICIG). 642–647 (IEEE Computer Society Washington, DC, USA 2011).10.1109/ICIG.2011.187 ↗
6. Dalal, N. & Triggs, B. Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 886–893 (IEEE, San Diego, CA, USA 2005).10.1109/CVPR.2005.177 ↗
7. Yu, K., Jia, L., Chen, Y. & Xu, W. Deep learning: yesterday, today, and tomorrow. J. Comput. Res. Dev. 50, 1799–1804 (2013).
8. Viola, P. & Jones, M. Rapid object detection using a boosted cascade of simple features. In Proc. of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1, 511-518 (IEEE, Kauai, HI, USA 2001).10.1109/CVPR.2001.990517 ↗
9. Belongie, S., Malik, J. & Puzicha, J. Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24, 509–522 (2002).10.1109/34.993558 ↗
10. Felzenszwalb, P. F., Girshick, R. B., McAllester, D. & Ramanan, D. Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1627–1645 (2010).10.1109/TPAMI.2009.167 ↗
11. Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Advances in Neural Information Processing Systems. (eds Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M. & Garnett, R.) In Neural Information Processing Systems 2015. 91–99 (NIPS, Montréal, Quebec, Canada, 2015).
12. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).10.1038/nature14539 ↗
13. Viola, P. & Jones, M. J. Robust real-time face detection. Int. J. Comput. Vision. 57, 137–154 (2004).10.1023/B:VISI.0000013087.49260.fb ↗
14. LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. (ed Touretzky, D.S.) In Advances in Neural Information Processing Systems. 396–404 (NIPS, Denver, Colorado, United States 1989).
15. Yang, Z., Tao, D.-p, Zhang, S.-y & Jin, L.-w Similar handwritten Chinese character recognition based on deep neural networks with big data. J. Commun. 35, 184–189 (2014).
16. Sharifara, A., Rahim, M. S. M. & Anisi, Y. A general review of human face detection including a study of neural networks and Haar feature-based cascade classifier in face detection. In 2014 International Symposium on Biometrics and Security Technologies (ISBAST). 73–78 (IEEE, Kuala Lumpur, Malaysia 2014).10.1109/ISBAST.2014.7013097 ↗
17. Jiang, H. & Learned-Miller, E. Face detection with the faster R-CNN. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). 650–657 (IEEE, Washington, DC, USA 2017).10.1109/FG.2017.82 ↗
18. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning. (MIT Press, Cambridge, Massachusetts, USA 2016).
19. Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In European Conference on Computer Vision. 818–833 (Springer, Cham, Zurich, Switzerland 2014).10.1007/978-3-319-10590-1_53 ↗
20. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at http://arXiv:1409.1556 (2014).
21. Szegedy, C. et al. Rethinking the Inception architecture for computer vision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 2818–2826 (IEEE, Las Vegas, NV, USA 2016).10.1109/CVPR.2016.308 ↗
22. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 3431–3440 (IEEE, Boston, MA, USA, 2015).10.1109/CVPR.2015.7298965 ↗
23. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask r-cnn. Preprint at http://arXiv:1703.06870 (2017).10.1109/ICCV.2017.322 ↗
24. Kreshuk, A. et al. Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images. PLoS ONE 6, e24899 (2011).10.1371/journal.pone.0024899 ↗
25. Ciresan, D., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in Neural Information Processing Systems. 2843–2851 (NIPS, Stateline, Nevada, United States 2012).
26. Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In International Conference on Medical Image Computing and Computer-assisted Intervention. 411–418 (Springer, Berlin, Heidelberg, Nagoya, Japan 2013).10.1007/978-3-642-40763-5_51 ↗
27. Lubbers, N., Lookman, T. & Barros, K. Inferring low-dimensional microstructure representations using convolutional neural networks. Phys. Rev. E 96, 052111 (2017).10.1103/PhysRevE.96.052111 ↗
28. Belianinov, A. et al. Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Adv. Struct. Chem. Imaging 1, 6 (2015).10.1186/s40679-015-0006-6 ↗
29. DeCost, B. L., Francis, T. & Holm, E. A. Exploring the microstructure manifold: image texture representations applied to ultrahigh carbon steel microstructures. Acta Mater. 133, 30–40 (2017).10.1016/j.actamat.2017.05.014 ↗
30. Chowdhury, A., Kautz, E., Yener, B. & Lewis, D. Image driven machine learning methods for microstructure recognition. Comput. Mater. Sci. 123, 176–187 (2016).10.1016/j.commatsci.2016.05.034 ↗
31. Ziatdinov, M. et al. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS Nano 11, 12742–12752 (2017).10.1021/acsnano.7b07504 ↗
32. Xu, W. & LeBeau, J. M. A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns. Preprint at http://arXiv:1708.00855 (2017).10.1016/j.ultramic.2018.03.004 ↗
33. Vincent, L. & Soille, P. Watersheds in digital spaces: an efficient algorithm based on immersion simulations. In IEEE Transactions on Pattern Analysis & Machine Intelligence 583–598 (IEEE, 1991).10.1109/34.87344 ↗
34. Mathworks. regionprops: Measure Properties of Image Regions https://www.mathworks.com/help/images/ref/regionprops.html (2017).
35. Parish, C. M., Field, K. G., Certain, A. G. & Wharry, J. P. Application of STEM characterization for investigating radiation effects in BCC Fe-based alloys. J. Mater. Res. 30, 1275–1289 (2015).10.1557/jmr.2015.32 ↗
36. Turcot, P. & Lowe, D. G. Better matching with fewer features: The selection of useful features in large database recognition problems. In 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops). 2109–2116 (IEEE, Kyoto, Japan 2009).10.1109/ICCVW.2009.5457541 ↗
37. Murphy, K. P. Machine Learning: A Probabilistic Perspective. (MIT Press, Cambridge, MA 2012).
38. Gonzalez, R. C. & Woods, R. E. Digital image analysis. 3rd edition (Prentice-Hall, Upper Saddle River, New Jersey, 2008).
39. Abràmoff, M. D., Magalhães, P. J. & Ram, S. J. Image processing with ImageJ. Biophotonics Int. 11, 36–42 (2004).
40. Field, K. G., Hu, X., Littrell, K. C., Yamamoto, Y. & Snead, L. L. Radiation tolerance of neutron-irradiated model Fe–Cr–Al alloys. J. Nucl. Mater. 465, 746–755 (2015).10.1016/j.jnucmat.2015.06.023 ↗
41. Field, K. G. et al. Heterogeneous dislocation loop formation near grain boundaries in a neutron-irradiated commercial FeCrAl alloy. J. Nucl. Mater. 483, 54–61 (2017).10.1016/j.jnucmat.2016.10.050 ↗
42. Field, K. G., Briggs, S. A., Sridharan, K., Yamamoto, Y. & Howard, R. H. Dislocation loop formation in model FeCrAl alloys after neutron irradiation below 1 dpa. J. Nucl. Mater. 495, 20–26 (2017).10.1016/j.jnucmat.2017.07.061 ↗
43. Briggs, S. A., Sridharan, K. & Field, K. G. Correlative microscopy of neutron-irradiated materials. Adv. Mater. Process. 174(10) (2016).10.31399/asm.amp.2016-10.p016 ↗
44. Kirk, M., Yi, X. & Jenkins, M. Characterization of irradiation defect structures and densities by transmission electron microscopy. J. Mater. Res. 30, 1195–1201 (2015).10.1557/jmr.2015.19 ↗
45. Phillips, P., Brandes, M., Mills, M. & De Graef, M. Diffraction contrast STEM of dislocations: imaging and simulations. Ultramicroscopy 111, 1483–1487 (2011).10.1016/j.ultramic.2011.07.001 ↗
46. Yao, B., Edwards, D. J. & Kurtz, R. J. TEM characterization of dislocation loops in irradiated bcc Fe-based steels. J. Nucl. Mater. 434, 402–410 (2013).10.1016/j.jnucmat.2012.12.002 ↗
47. Ahonen, T., Hadid, A. & Pietikainen, M. Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28, 2037–2041 (2006).10.1109/TPAMI.2006.244 ↗
48. Alionte, E. & Lazar, C. A practical implementation of face detection by using Matlab cascade object detector. In 2015 19th International Conference on System Theory, Control and Computing (ICSTCC) 785–790 (IEEE, Cheile Gradistei, Romania 2015).10.1109/ICSTCC.2015.7321390 ↗
49. Mathworks. Object Detection Using Deep Learning https://www.mathworks.com/help/vision/examples/object-detection-using-deep-learning.html (2017).
50. Li, W., Field, K. G. & Morgan, D. Detection of Open Loop Defects in STEM Images of Irradiation-Damaged Alloys – Source Code for Detection and Image Dataset (Materials Data Facility, doi: https://dx.doi.org/doi:10.18126/M2692Z, 2018).