U-Net Implementation for High Throughput Grain Boundary Detection in Bright Field TEM Micrographs: Toward In Situ Grain Growth Studies

Quantitative understanding of grain growth is limited by the amount of data available as a function of annealing time and temperature. Advances in in situ microscopy have made it possible to capture grain coarsening as it happens in the transmission electron microscope (TEM). Extracting quantitative results like grain size distributions and other geometric and topological metrics for statistically significant populations of grains from the resulting video data requires reliable grain boundary (GB) identification, yet the only established technique is laborious hand-tracing of GBs in each image. Attempts have been made at automated GB detection [1], but conventional gradient-based schemes have shown limited success. These algorithms typically oversegment at sharp changes in contrast in regions like those near bend contours and undersegment between grains which show little contrast due to their similar diffracted intensity. For progress in quantitative in situ studies of grain growth, where thousands of images must be captured during any one experiment, a high-throughput technique for collecting microstructural information must be developed. Modern machine learning models provide an opportunity to break through this barrier but are limited by the size of the available ground truth datasets for training. This work applies a supervised learning approach using a U-Net convolutional neural network [2] to automate GB tracing, using training data carefully constructed from photographic plate-based bright field (BF) micrographs and associated expert hand-tracings of Al thin films, available from prior studies [3]. The model is validated on photographic data not used for training and then is applied to images of Al, recently acquired on a CCD camera, to observe >11,000 grains. Finally, a transfer learning approach is applied to obtain GB networks from in situ images of a Pt sample, demonstrating the transferability of the technique to in situ grain growth studies. The training dataset consists of BF TEM micrographs of 100 nm thick Al thin films which had attained a stagnant structure after annealing. These data were collected on photographic plates on a Philips EM420 microscope operating at 120 kV. Each field of view (FoV) was captured at one to five sample tilts to vary diffraction contrast and maximize the number of visible GBs across the images. Then, the GBs visible in the images were manually traced onto transparencies, which were scanned and digitally analyzed. For this work, both the hand tracings and the micrographs were scanned into digital images and image shifts were corrected using an in-house software package. Fourteen fields of view were used for 66 epochs of training U-Net; at each epoch, the data were randomly augmented using rotations and reflections to create a more robust training set. Twenty-eight FoVs remained from this dataset and were used to validate the trained U-Net model. The tracings of these images obtained from the U-Net model were double thresholded, combined by a max operator and subjected to a dilation, erosion, closing and pruning step [1]. Figs. 1a and b show a representative micrograph of one test FoV and its final U-Net tracing. To validate the model, the grain size distribution, shown in Fig. 1c, is extracted from the U-Net GB networks, and compared to the published distributions for the hand traced grains [3]. When both distributions are fit to lognormal density functions, the Hellinger distance between them is 0.061, deemed sufficiently close to proceed to newly collected data. Furthermore, the inset of Fig. 1c shows a Q-Q plot of the log of the two distributions, demonstrating that there is significant overlap, with divergence only at the smallest grain sizes. The model was then applied unmodified to new TEM BF images of a 100 nm-thick Al film as-deposited and after annealing for 15, 30, 60 and 120 minutes at 185°C, collected on an FEI Talos F200X on a CETA CCD camera at three sample tilts per FoV. The images were aligned to one another to correct image shifting from tilting in the same way as the training and validation images. Figures 1d and 1e show a representative micrograph from the 30-min sample with its U-Net output overlaid. >11,000 grains were automatically detected across 60 fields of view, representing a tremendous reduction in time and effort required to obtain such a large dataset. The grain size distribution from the four samples is shown in Fig. 1f, demonstrating the model is capable of generating new results from recent data collected on modern recording media, despite being visibly different from the training data. Finally, the model was applied to images of a 50 nm thick Pt film captured during an in situ heating experiment performed using a Protochips Fusion heating holder. Sample tilting is impractical for in situ experiments due to the image shifting, so images were recorded using a conical bright field (CBF) technique. Here, the beam was fixed with a constant tilt, but the beam’s rotation angle was changed to vary diffraction contrast from one frame to the next. Approximately every five frames, the beam completed one revolution. To apply the Al model to in situ Pt data, CBF images were first collected without heating. Two FoVs were hand traced using all five beam rotations to maximize the number of visible GBs; these 10 images and 2 tracings were then used for transfer learning on the decoder of the Al model. An in situ heating experiment was then performed and the updated model was applied using three beam positions for each traced image. Fig. 2 shows a representative frame, with its postprocessed U-Net tracing overlaid; the close match indicates that with a small amount of manual tracing, the model can be adapted to collect in situ data from a variety of materials, opening the door to high-throughput data collection and analysis for grain growth studies at the scale required to elucidate the dynamics of microstructural evolution [4]. (a) Micrograph acquired at one sample tilt of an Al film annealed 2 h at 400°C with (b) the U-Net output tracing overlaid. (c) The grain size distribution of reduced diameters obtained via U-Net from the validation Al dataset; the Q-Q plot for the validation-set tracings and the original hand-tracings is inset. (d) Micrograph of new Al film, captured on a CCD camera at one beam rotation, with (e) the U-Net output overlaid. (f) The grain size distribution of reduced circle-equivalent diameters, obtained via U-Net for new films as-deposited and after 15, 30, 60 and 120 minutes of annealing at 185°C; the Q-Q plot for the 11,163 grains is inset. Overlay of the U-Net boundary network on one of the CBF images captured during an in situ heating experiment on Pt.