Machine Learning Enabled Image Classification for Automated Data Acquisition in the Electron Microscope

The development of high-speed detectors and associated data acquisition workflows in electron microscopes has enabled data generation at an unprecedented rate. It is not uncommon to generate terabytes of data per hour which can easily overwhelm computing infrastructure and downstream processing, thereby limiting the full utilization of the microscope hardware [1]. Implementing a selective approach to imaging to only acquire high-fidelity data from areas of interest is common in manual instrument operation, where bright- or dark-field STEM images are acquired first to identify regions of interest for collecting higher-dimensional spectroscopy or diffraction data. In the context of automated acquisitions workflows, this procedure becomes significantly more challenging, as it becomes necessary to quickly make decisions about regions of interest in the absence of a human operator. To that end, we have developed machine learning models working with low-fidelity dark-field STEM images of nanostructure arrays to perform a binary quality assessment to determine the highest priority areas to acquire high-fidelity data. Our models will be deployed in an automated electron microscopy system for the high-throughput analysis of nanomaterial megalibraries. Megalibraries contain arrays of 108 nanoparticles with spatially encoded size and composition gradients on a single chip, these libraries are uniquely suited for the discovery of new materials at high throughput [2]. Given the volume of materials in a megalibrary, we have designed our models to be highly effective at binary classification on unseen images while minimizing the single-inference times and the number of false positive results which would result in long latencies and high downstream processing cost. In this presentation, we will describe the computational challenges and solutions implemented in this design (e.g., memory constraints, training time, Neural Architecture Search (NAS) tools). We present the standard neural network evaluation metrics, training time, and single-inference time for the most accurate model; our best-performing model achieves a precision of >95%. We further report a weighted F-score of >90% on our test data set as a balancing metric of precision and recall for a given application. Given that false positives would incur a high downstream cost in the analysis pipeline, we chose to prioritize precision over recall by a factor of 9-to-1. This binary classification of low-fidelity data represents an important step towards fully automated electron microscopy workflows, as it introduces a methodology for autonomous decision-making in situ. The acceleration resulting from this automation effectively increases the experimental throughput, enabling the analysis of materials at an unprecedented rate [3]. Automated quality assessment on a nanoparticle megalibrary. (A) Schematic illustration of the screening process, acquiring images at nanoparticle locations. (B) Acquired high-angle annular dark-field (HAADF) images as seen by the model. Scale bars: 200 nm. (C) Illustration of the quality assessment pipeline, where each raw image is fed into the model, returning a binary classifier. If the image passes the test, the microscope can magnify and focus on that region to acquire high-fidelity datasets. Scale bars: 20 nm.