Methodology for Collecting and Aligning Correlative SEM, CLSM and LOM Images of Bulk Material Microstructure to Create a Large Machine Learning Training Dataset

Even small changes in microstructure can cause great changes of mechanical properties. A great variation in microstructure – especially in the case of steel – leads to a wide range of mechanical properties [1]. Hence it is important to study microstructure. Unfortunately, a universal study method does not exist, thus we need to rely on a variety of different microstructure imaging techniques. However, navigating the sample and finding the corresponding region in scale of a few micrometers in different devices (and imaging techniques) is difficult, especially if devices are from different manufacturers. The existing multi-modal solutions typically assume either integrated multiple detectors in a chamber or a single sample carrier that can be switched among different machines of the same provider. But we needed to develop a customized technique to obtain a large number of images of corresponding regions with our equipment. Specifically, it is FEI Magellan 400L scanning electron microscope (SEM, using CBS detector) [2], the Keyence VK-X1000 confocal laser scanning microscope (CLSM) [3] and the Zeiss Axio Observer 7 for materials light optical microscope (LOM) [4]. We decided to create a grid on the fine etched metallographic sample in order to simplify the navigation. At first, we tried to glue a TEM grid to the surface of the sample, but the grid formed shadows and it was problematic due to the working distance of the LOM and CLSM microscopes (Fig. 1a). Later, we considered creating a grid using picosecond LASER [5], electron beam welder (insufficient accuracy), FIB-SEM (time-demanding and it sputters material on the surface), nanoindenter (forms a discontinuous grid) or electron beam lithography (requires layer deposition on the surface thus covering the pristine material). Although the LASER sprays microscopic droplets of molten metal nearby its trace, it provides sufficient accuracy and productivity. It has proven to be the best tool of the above mentioned ones. It also allows for more control over grid design for easier navigation. Each quadrant of the grid consists of two by two smaller grids, sub-grids, each with four rows and four columns of cells. The size of each cell is approximately 500 × 500 μm. The macro position of the sub-grid is indicated by the T shape in each given cell, therefore it is very easy to determine the current position even with a limited field of view. This shape also clearly determines the orientation of the grid, hence of the whole sample. In addition to the above, each cell has an exactly defined coordinate as on a chessboard (Fig. 1b). The above described auxiliary grid makes possible a correlative mapping of relatively large areas of the microstructure at the necessary magnification in different microscopes. The area of the grid cell was imaged by dozens of images (tiles). Problematically, image from each microscope has a different field of view at required magnification (e.g., SEM-CBS at 1200 × magnification: 124 × 82.67 μm, CLSM at 1500 × magnification: 97.18 × 72.88 μm, LOM at 1000 × magnification: 160 × 133.51 μm), a different aspect ratio, a different resolution, and a different overlap between the images. Hence, the position of the individual tiles obtained in the given microscopes was not the same. To simplify the alignment of the same areas, all tiles were stitched together in Adobe Photoshop CS6 [6]. We also examined free tools (e.g., BigStitcher in ImageJ/Fiji) [7, 8], but apart from Photoshop, all the other examined tools introduced clearly visible artifacts. These panoramas has been transformed (using bilinear scaling and nonlinear transformation to account for slightly different both aberration and a tilt of the sample in each modality) and manually aligned to match the underlying microstructure of each modality as closely as possible. The panoramas were cropped so that all modalities had the same number of pixels and occupied the same area. Unfortunately, the coarse alignment was not accurate enough to train the machine learning (ML) model with desired precision/accuracy. Thus, we sliced the panoramas using a Python script (image-split-with-overlap) [9] into matching tiles that finally had the same field of view and the same resolution. These tuples of tiles were then elastically aligned using Python script implementing the PyImageJ library [10]. Correct 3D Drift add-on [11] was first used for better rigid pre-alignment and bUnwarpJ [12] for final elastic alignment. We tested different tools for the final alignment. For example, DSH4 Image Alignment [13] in ImageJ/Fiji needed to manually annotate the landmarks, because the automatic landmark extraction did not work well due to the large variation between the modalities. Another ImageJ/Fiji tool TrakEM2 [14] was very user-friendly, but there was a difficult way to automated processing a large number of tuples. It also took over 200 s to align one tuple. Correlia [15] tool did not suit us well due to the limitation of the maximum resolution. Other tools were either built for biological imaging, hence they did not perform well on steel microstructures, or they were too complicated to install and operate (e.g., DeepCLEM). [16] Python macro using Correct 3D Drift, bUnwarpJ and Structural Similarity Index Metric (SSIM) [17] manages it under 10 s, including iterating through competing settings. The above described method is very fast. Nevertheless, large differences in modalities may result into a divergent alignment. The SSIM value easily detects very poorly aligned tuples, but despite our best efforts to automate the alignment, it is necessary for an expert to inspect the resulting dataset. The whole workflow is described in detail in Fig. 2. An example of pre-aligned and precisely aligned microstructure tuples can be seen in Fig. 3. We plan to use the resulting dataset to train a ML model. We intend such a model to detect poorly aligned stacks or to perform the alignment of panoramas [18]. Examples of navigation grids: a) glued TEM grid, b) navigation grid engraved by picosecond LASER (a sub-grid and its individual cell are highlighted in the top left corner). Schematic of the workflow from the engraving navigation grid to the final dataset of SEM, CLSM and LOM corelative images of the bulk metallographic sample. In the pre-aligned example (left), the TRIP steel microstructure shift is clearly visible, which disappears after fine elastic alignment of the layers (right). Micrographs was acquired by SEM (CBS detector), CLSM (LASER mode) and LOM (Micro LED lamp). Each modality is displayed as one of the three RGB channels (Red = SEM-CBS, Green = CLSM, Blue = LOM).