Machine Learning Enhanced Image Segmentation for High-Fidelity STEM Data Analysis

Transmission electron microscopy (TEM) is an indispensable approach for the characterization of materials structures, chemistry, and functionality. The rapid advancement of TEM to allow for in situ, multimodal, multidimensional characterization generates huge datasets in spatial, temporal, and chemical domains, which, in turn, requires a new paradigm for data-intensive analysis. Artificial intelligence (AI) enabled TEM data analytics has become an emerging field that combines the power of computer vision and machine learning (ML) algorithms to automatically analyze and interpret TEM images, improving the speed and accuracy of the results [1]. TEM image analysis often involves image segmentation, which is the process of separating individual image segments containing representative features to facilitate further analysis. This is a crucial task that requires both accuracy and speed, particularly when dealing with large volumes of datasets generated by in situ TEM experiments. ML algorithms can provide an effective solution for both pixel-wise segmentation of individual atom columns and semantic segmentation of material features at the single-particle and microstructure level [2]. This approach can significantly improve the efficiency and accuracy of TEM data analysis by reducing human perception and intervention, leading to autonomous and accelerated analytical workflow. Here, we focused on a case study of bimetallic Pt-Ni catalyst nanoparticles and explored the effectiveness of image segmentation for high-angle annular dark-field scanning TEM (HAADF-STEM) images using both traditional global thresholding method and ML-enhanced segmentation approach. Specifically, we adopted the U-Net convolutional neural network (CNN) architecture [3] to investigate the accuracy and efficiency of ML algorithms for nanoparticle identification and segmentation. Fig. 1 shows the original HAADF-STEM image of Pt-Ni nanoparticle clusters and the corresponding segmentation by applying the global threshold and the ML model, respectively. The results indicate that the trained ML model can successfully identify all nanoparticles and effectively segment them by well-defined outlines with much higher accuracy of edge detection, outperforming the traditional global thresholding method. To identity the detection limit for segmentation, we also performed systematic assessments on the effect of imaging magnification on ML-based image segmentation quality (Fig. 1C). We compared the same nanoparticle cluster from the HAADF-STEM images taken at various magnification ranging from 450 kX to 2.5 MX and quantitatively analyzed their projected areas and accuracy using precisely measured manual segmentation as the grand truth (Fig. 2). We found that the edge detection accuracy remains at >95% when magnification is greater than 1.0 MX and it starts to decrease significantly at lowered magnifications due to the reduced pixel resolution. This allows us to determine practically reasonable imaging parameters, in terms of the optimized resolution and field-of-view combinations, for reliable ML-enhanced image segmentation. These findings provide useful guidelines for improving the accuracy and generalizability of segmentation tasks in data-intensive high-fidelity TEM/STEM image analyses [4]. Image segmentation of HAADF-STEM images. (A) Original HAADF-STEM image of Pt-Ni nanoparticles. Segmentation of nanoparticles using (B) the traditional global thresholding method, and (C) the ML trained model at different magnifications. Quantitative analysis of image segmentation accuracy of traditional global thresholding method and ML-enhanced method for HAADF-STEM images taken at various magnifications.