Post-Experiment Forensics and Human-in-the-Loop Interventions in Explainable Autonomous Scanning Transmission Electron Microscopy

The introduction of automated experiments (AE) in electron and scanning probe microscopy have become the North Star of scientific community in recent years. The central concept of AE is the workflow, i.e. the sequence of operations performed by the human or AI agent. Recent advances in machine learning (ML) algorithms are enabling the development of single-step beyond-human workflows, as exemplified by the property discovery in scanning probe and electron microscopy. The broad deployment of automated experiments necessitates development of a methodology to monitor the progression of the active learning process interacting with experimental physical systems. This includes both developing the tools for explainability of the AE in real-time and after the experiment, and creating frameworks that allow human intervention on the timescale and decision-making levels amenable to human scientists. Here we introduce the concept of after-experiment AE forensics and human-in-the-loop interventions, which can be applied to scanning transmission electron microscopy (STEM). The central element of experimental active learning is the ML agent interacting with the experimental system in an iterative fashion, both performing the experimentation and updating the state of the ML agent. Throughout the experiment, the state of the agent is updated in response to the incoming information from the active data generation process (i.e., microscope). Based on the current state and prior information, the agent makes decisions that are communicated to the microscope. This iterative cycle continues until the experimental budget is exhausted, or the predefined goal is achieved. At the end of the experiment, the change in the state of the agent represents the knowledge gained during the experiment. Correspondingly, AE forensics is defined as the analysis of the decision-making at each step of the experiment, comparing the decisions made by the agent in the actual knowledge state compared to the decision that would have been made by the fully trained agent. We develop the forensics for a specific case of a deep kernel learning automated experiment workflow for STEM-EELS and 4D STEM. We introduce the necessary definitions adapted from stochastic optimization and microscopy communities, including the real time, trained, and full models, the real- and latent space trajectory mapping, regret accumulation, and acquisition and scalarizer function optimization. Jointly, these high-level functions allow real time monitoring of the AE workflow, balancing the exploration and exploitation in the discovery, and injection of human knowledge before, during, and after the experiment. The proposed approach is universal and can be extended to other physical and chemical imaging techniques combining spatially resolved imaging and local probing. We will provide a publicly available forensic analysis notebook on GitHub for deep kernel learning-based AE STEM/EELS [1]. Learning curve for the STEM-EELS experiment with different acquisition functions. Experimental trajectories with different acquisition functions. Global latent representation and latent space trajectory for several acquisition functions