Revealing the evolution of order in materials microstructures using multi-modal computer vision
Source abstract
Abstract
The development of high-performance materials for microelectronics, energy storage, and extreme environments depends on our ability to describe and direct property-defining microstructural order. Our present understanding is typically derived from laborious manual analysis of imaging and spectroscopy data, which is difficult to scale, challenging to reproduce, and lacks the ability to reveal latent associations needed for mechanistic models. Here, we demonstrate a multi-modal machine learning (ML) approach to describe order from electron microscopy analysis of the complex oxide La 1− x Sr x FeO 3 . We construct a hybrid pipeline based on fully and semi-supervised classification, allowing us to evaluate both the characteristics of each data modality and the value each modality adds to the ensemble. We observe distinct differences in the performance of uni- and multi-modal models, from which we draw general lessons in describing crystal order using computer vision.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Electronic and Structural Properties of Oxides · Advanced Electron Microscopy Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (86)
Show source-provided references
1. Callister Jr, W. D., & Rethwisch, D. G. Materials Science and Engineering: An Introduction. 9th edn. (Wiley, 2013).
2. Eswarappa Prameela, S. et al. Materials for extreme environments. Nat. Rev. Mater. 8, 81–88 (2023).10.1038/s41578-022-00496-z ↗
3. Zhang, Y. & Weber, W. J. Ion irradiation and modification: the role of coupled electronic and nuclear energy dissipation and subsequent nonequilibrium processes in materials. Appl. Phys. Rev. 7, 041307 (2020).10.1063/5.0027462 ↗
4. Pernigoni, L. & Grande, A. Advantages and challenges of novel materials for future space applications. Front. Space Technol. 4, 1253419 (2023).10.3389/frspt.2023.1253419 ↗
5. Schmidt, F. et al. Effects of radiation-induced defects on corrosion. Annu. Rev. Mater. Res. 51, 293–328 (2021).10.1146/annurev-matsci-080819-123403 ↗
6. Norouzi, E., Waitelonis, J. & Sack, H. The landscape of ontologies in materials science and engineering: a survey and evaluation. https://doi.org/10.48550/arXiv.2408.06034 (2024).10.48550/arXiv.2408.06034 ↗
7. Ramasse, Q. M. Twenty years after: how “Aberration correction in the STEM” truly placed a “A synchrotron in a Microscope”. Ultramicroscopy 180, 41–51 (2017).10.1016/j.ultramic.2017.03.016 ↗
8. Petford-Long, A. K. & Chiaramonti, A. N. Transmission electron microscopy of multilayer thin films. Annu. Rev. Mater. Res. 38, 559–584 (2008).10.1146/annurev.matsci.38.060407.130326 ↗
9. Chen, Z. et al. Quantitative atomic resolution elemental mapping via absolute-scale energy dispersive X-ray spectroscopy. Ultramicroscopy 168, 7–16 (2016).10.1016/j.ultramic.2016.05.008 ↗
10. Spurgeon, S. R., Du, Y. & Chambers, S. A. Measurement error in atomic-scale scanning transmission electron microscopy-energy-dispersive X-ray spectroscopy (STEM-EDS) mapping of a model oxide interface. Microsc. Microanal. 23, 513–517 (2017).10.1017/S1431927617000368 ↗
11. Schwartz, J. et al. Imaging atomic-scale chemistry from fused multi-modal electron microscopy. npj Comput. Mater. 8, 16 (2022).10.1038/s41524-021-00692-5 ↗
12. Gunkel, F., Christensen, D. V., Chen, Y. & Pryds, N. Oxygen vacancies: the (in) visible friend of oxide electronics. Appl. Phys. Lett. 116, 120505 (2020).10.1063/1.5143309 ↗
13. Tuller, H. L. & Bishop, S. R. Point defects in oxides: tailoring materials through defect engineering. Annu. Rev. Mater. Res. 41, 369–398 (2011).10.1146/annurev-matsci-062910-100442 ↗
14. Shi, J. et al. Wide bandgap oxide semiconductors: from materials physics to optoelectronic devices. Adv. Mater. 33, 2006230 (2021).10.1002/adma.202006230 ↗
15. Geng, D. et al. Thin-film transistors for large-area electronics. Nat. Electron. 6, 963–972 (2023).10.1038/s41928-023-01095-8 ↗
16. Yan, A. et al. Thin-film transistors for integrated circuits: fundamentals and recent progress. Adv. Funct. Mater. 34, 2304409 (2024).10.1002/adfm.202304409 ↗
17. Wang, M., Zhu, H., Nomura, K., Noh, Y.-Y. & Liu, A. Revolutionizing electronics with oxide thin-film transistor technology. Device 2, 100396 (2024).10.1016/j.device.2024.100396 ↗
18. Faber, H. et al. Heterojunction oxide thin-film transistors with unprecedented electron mobility grown from solution. Sci. Adv. 3, e1602640 (2017).10.1126/sciadv.1602640 ↗
19. Li, Y. et al. Recent advances of solution-processed heterojunction oxide thin-film transistors. Nanomaterials 10, 965 (2020).10.3390/nano10050965 ↗
20. Lee, J. & Chung, D. S. Heterojunction oxide thin film transistors: a review of recent advances. J. Mater. Chem. C. 11, 5241–5256 (2023).10.1039/D3TC00584D ↗
21. Sickafus, K. et al. Radiation tolerance of complex oxides. Science 289, 748–751 (2000).10.1126/science.289.5480.748 ↗
22. Stemmer, S. & Allen, S. J. Two-dimensional electron gases at complex oxide interfaces. Annu. Rev. Mater. Res. 44, 151–171 (2014).10.1146/annurev-matsci-070813-113552 ↗
23. Zubko, P., Gariglio, S., Gabay, M., Ghosez, P. & Triscone, J.-m. Interface physics in complex oxide heterostructures. Annu. Rev. Condens. Matter Phys. 2, 141–165 (2011).10.1146/annurev-conmatphys-062910-140445 ↗
24. Yano, K. H. et al. Dose rate dependent cation & anion radiation enhanced diffusion in hematite. J. Mater. Chem. A 10, 24167–24177 (2022).10.1039/D2TA03403D ↗
25. Kaspar, T. C. et al. Role of structural defects in mediating disordering processes at irradiated epitaxial Fe3O4/Cr2O3 interfaces. Phys. Rev. Mater. 7, 093604 (2023).10.1103/PhysRevMaterials.7.093604 ↗
26. Schmidt, F. et al. Effects of radiation-induced defects on corrosion. Annu. Rev. Mater. Res. 51, 1–36 (2021).10.1146/annurev-matsci-080819-123403 ↗
27. Beyerlein, I., Demkowicz, M., Misra, A. & Uberuaga, B. Defect-interface interactions. Prog. Mater. Sci. 74, 125–210 (2015).10.1016/j.pmatsci.2015.02.001 ↗
28. Beyerlein, I. et al. Radiation damage-tolerant nanomaterials. Mater. Today 16, 443–449 (2013).10.1016/j.mattod.2013.10.019 ↗
29. Matthews, B. E. et al. Percolation of ion-irradiation-induced disorder in complex oxide interfaces. Nano Lett. 21, 5353–5359 (2021).10.1021/acs.nanolett.1c01651 ↗
30. Sassi, M., Kaspar, T., Rosso, K. M. & Spurgeon, S. R. Effect of structure and composition on the electronic excitation induced amorphization of La2Ti2-xZrxO7 ceramics. Sci. Rep. 9, 8190 (2019).10.1038/s41598-019-44621-5 ↗
31. Spurgeon, S. R. et al. Asymmetric lattice disorder induced at oxide interfaces. Adv. Mater. Interfaces 7, 1901944 (2020).10.1002/admi.201901944 ↗
32. Choudhury, S. et al. Non-uniform solute segregation at semi-coherent metal/oxide interfaces. Sci. Rep. 5, 1–9 (2015).10.1038/srep13086 ↗
33. Aguiar, J. A. et al. Linking interfacial step structure and chemistry with locally enhanced radiation-induced amorphization at oxide heterointerfaces. Adv. Mater. Interfaces 1, 1300142 (2014).10.1002/admi.201300142 ↗
34. Aguiar, J. A. et al. Orientation-specific amorphization and intercalated recrystallization at ion-irradiated SrTiO3/MgO interfaces. J. Mater. Res. 29, 1699–1710 (2014).10.1557/jmr.2014.217 ↗
35. Dholabhai, P. P., Aguiar, J. A., Misra, A. & Uberuaga, B. P. Defect interactions with stepped CeO 2 /SrTiO 3 interfaces: implications for radiation damage evolution and fast ion conduction. J. Chem. Phys. 140, 194701 (2014).10.1063/1.4876225 ↗
36. Bi, Z. et al. Role of the interface on radiation damage in the SrTiO3/ LaAlO3 heterostructure under Ne2+ ion irradiation. J. Appl. Phys. 115, 124315 (2014).10.1063/1.4870052 ↗
37. Bruno, G., Lynch, M. J., Jacobs, R., Morgan, D. D. & Field, K. G. Evaluation of human-bias in labeling of ambiguous features in electron microscopy machine learning models. Microsc. Microanal. 29, 1493–1494 (2023).10.1093/micmic/ozad067.767 ↗
38. Kalinin, S. V. et al. Machine learning for automated experimentation in scanning transmission electron microscopy. npj Comput. Mater. 9, 227 (2023).10.1038/s41524-023-01142-0 ↗
39. Olszta, M. et al. An automated scanning transmission electron microscope guided by sparse data analytics. Microsc. Microanal. 28, 1611–1621 (2022).10.1017/S1431927622012065 ↗
40. Roccapriore, K. M. et al. Probing electron beam induced transformations on a single-defect level via automated scanning transmission electron microscopy. ACS Nano 16, 17116–17127 (2022).10.1021/acsnano.2c07451 ↗
41. Sytwu, K., Groschner, C. & Scott, M. C. Understanding the influence of receptive field and network complexity in neural network-guided TEM image analysis. Microsc. Microanal. 28, 1896–1904 (2022).10.1017/S1431927622012466 ↗
42. Li, K. et al. Single-image-based deep learning for precise atomic defect identification. Nano Lett. 24, 10275–10283 (2023).10.1021/acs.nanolett.4c02654 ↗
43. Akers, S. et al. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Comput. Mater. 7, 187 (2021).10.1038/s41524-021-00652-z ↗
44. Li, F. & Xi, Q. Defectnet: Toward fast and effective defect detection. IEEE Trans. Instrum. Meas. 70, 1–9 (2021).
45. Stuckner, J., Harder, B. & Smith, T. M. Microstructure segmentation with deep learning encoders pre-trained on a large microscopy dataset. npj Comput. Mater. 8, 200 (2022).10.1038/s41524-022-00878-5 ↗
46. Ziatdinov, M., Ghosh, A., Wong, C. Y. T. & Kalinin, S. V. Atom AI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy. Nat. Mach. Intell. 4, 1011–1112 (2022).10.1038/s42256-022-00555-8 ↗
47. Sadre, R., Ophus, C., Butko, A. & Weber, G. H. Deep learning segmentation of complex features in atomic-resolution phase-contrast transmission electron microscopy images. Microsc. Microanal. 27, 804–814 (2021).10.1017/S1431927621000167 ↗
48. Groschner, C. K., Choi, C. & Scott, M. C. Machine learning pipeline for segmentation and defect identification from high-resolution transmission electron microscopy data. Microsc. Microanal. 27, 549–556 (2021).10.1017/S1431927621000386 ↗
49. Lee, K. et al. Stem image analysis based on deep learning: Identification of vacancy defects and polymorphs of MoS2. Nano Lett. 22, 4677–4685 (2022).10.1021/acs.nanolett.2c00550 ↗
50. Burns, K. et al. Deep learning-enabled probing of irradiation-induced defects in time-series micrographs. APL Mach. Learn 2, 016117 (2024).10.1063/5.0186046 ↗
51. Shi, C. et al. Uncovering material deformations via machine learning combined with four-dimensional scanning transmission electron microscopy. npj Comput. Mater 8, 114 (2024).10.1038/s41524-022-00793-9 ↗
52. Cui, Y. et al. Deep learning for image and point cloud fusion in autonomous driving: a review. IEEE Trans. Intell. Transport. Syst. 23, 722–739 (2022).10.1109/TITS.2020.3023541 ↗
53. Rizzoli, G., Barbato, F. & Zanuttigh, P. Multimodal semantic segmentation in autonomous driving: a review of current approaches and future perspectives. Technologies 10, 90 (2022).10.3390/technologies10040090 ↗
54. Li, J. L., Cai, Y., Li, Q., Kou, M. & Zhang, T. A review of remote sensing image segmentation by deep learning methods. Int. J. Digit. Earth 17, 2328827 (2024).10.1080/17538947.2024.2328827 ↗
55. Jiang, B., An, X., Xu, S. & Chen, Z. Intelligent image semantic segmentation: a review through deep learning techniques for remote sensing image analysis. J. Indian Soc. Remote Sens. 51, 1865–1878 (2023).10.1007/s12524-022-01496-w ↗
56. Walter, A. et al. Correlated multimodal imaging in life sciences: expanding the biomedical horizon. Front. Phys. 8, 47 (2020).10.3389/fphy.2020.00047 ↗
57. Fernandez-Gonzalez, R. et al. PyJAMAS: open-source, multimodal segmentation and analysis of microscopy images. Bioinformatics 38, 594–596 (2021).10.1093/bioinformatics/btab589 ↗
58. Cameron, W. D., Bennett, A. M., Bui, C. V., Chang, H. H. & Rocheleau, J. V. Leveraging multimodal microscopy to optimize deep learning models for cell segmentation. APL Bioeng. 5, 016101 (2021).10.1063/5.0027993 ↗
59. Ibtehaz, N. & Rahman, M. S. Multiresunet: rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw. 121, 74–87 (2020).10.1016/j.neunet.2019.08.025 ↗
60. Muroga, S., Miki, Y. & Hata, K. A comprehensive and versatile multimodal deep-learning approach for predicting diverse properties of advanced materials. Adv. Sci. 10, 2302508 (2023).10.1002/advs.202302508 ↗
61. Kimoto, K. et al. Unsupervised machine learning combined with 4D scanning transmission electron microscopy for bimodal nanostructural analysis. npj Comput. Mater 14, 2901 (2024).
62. Jiao, X., Chen, Y. & Dong, R. An unsupervised image segmentation method combining graph clustering and high-level feature representation. Neurocomputing 409, 83–92 (2020).10.1016/j.neucom.2020.05.073 ↗
63. Wang, Y. et al. Self-supervised transformers for unsupervised object discovery using normalized cut. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14543–14553 (2022).10.1109/CVPR52688.2022.01414 ↗
64. Melas-Kyriazi, L., Rupprecht, C., Laina, I. & Vedaldi, A. Deep spectral methods: a surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8364–8375 (2022).10.1109/CVPR52688.2022.00818 ↗
65. Aflalo, A., Bagon, S., Kashti, T. et al. Deepcut: unsupervised segmentation using graph neural network clustering. https://doi.org/10.48550/arXiv.2212.05853 (2022).10.48550/arXiv.2212.05853 ↗
66. Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M. & Zitnik, M. Multimodal learning with graphs. Nat. Mach. Intell. 5, 340–350 (2023).10.1038/s42256-023-00624-6 ↗
67. Iyer, G., Chanussot, J. & Bertozzi, A. L. A graph-based approach for data fusion and segmentation of multimodal images. IEEE Trans. Geosci. Remote Sens. 59, 4419–4429 (2020).10.1109/TGRS.2020.2971395 ↗
68. He, Q. et al. Multimodal remote sensing image segmentation with intuition-inspired hypergraph modeling. IEEE Trans. Image Process. 32, 1474–1487 (2023).10.1109/TIP.2023.3245324 ↗
69. Felzenszwalb, P. F. & Huttenlocher, D. P. Efficient graph-based image segmentation. Int. J. Comput. Vis. 59, 167–181 (2004).10.1023/B:VISI.0000022288.19776.77 ↗
70. Camilus, K. S. & Govindan, V. A review on graph-based segmentation. Int. J. Image, Graph. Signal Process. 4, 1 (2012).
71. Taylor, S. D. et al. Resolving diverse oxygen transport pathways across Sr-doped lanthanum ferrite and metal-perovskite heterostructures. Adv. Mater. Interfaces 10, 2202276 (2023).10.1002/admi.202202276 ↗
72. Dan, J. et al. Learning motifs and their hierarchies in atomic resolution microscopy. Sci. Adv. 8, eabk1005 (2022).10.1126/sciadv.abk1005 ↗
73. Ter-Petrosyan, A. H. et al. Unsupervised segmentation of irradiation-induced order-disorder phase transitions in electron microscopy. In Proc. Machine Learning and the Physical Sciences Workshop, NeurIPS 2023 (2023).
74. Snell, J., Swersky, K. & Zemel, R. Prototypical networks for few-shot learning. Adv. Neural Inf. Process. Syst. 30, https://arxiv.org/abs/1703.05175 (2017).
75. Williams, D. B. & Carter, C. B. Transmission Electron Microscopy. http://link.springer.com/10.1007/978-0-387-76501-3 (Springer, 2009).
76. Zega, T. J. et al. Mass-thickness measurements in the transmission electron microscope: a single-standard approach to quantitative EDS analysis. Microsc. Microanal. 31, ozaf005 (2025).10.1093/mam/ozaf005 ↗
77. Creange, N., Dyck, O., Vasudevan, R. K., Ziatdinov, M. & Kalinin, S. V. Towards automating structural discovery in scanning transmission electron microscopy. Mach. Learn.: Sci. Technol. 3, 015024 (2022).
78. Lewis, N. R. et al. Forecasting of in situ electron energy loss spectroscopy. npj Comput. Mater. 8, 252 (2022).10.1038/s41524-022-00940-2 ↗
79. Nord, M., Vullum, P. E., Hallsteinsen, I., Tybell, T. & Holmestad, R. Assessing electron beam sensitivity for SrTiO3 and La0.7Sr0.3MnO3 using electron energy loss spectroscopy. Ultramicroscopy 169, 98–106 (2016).10.1016/j.ultramic.2016.07.004 ↗
80. Stoerzinger, K. A. et al. Linking surface chemistry to photovoltage in Sr-substituted LaFeO3 for water oxidation. J. Mater. Chem. A 6, 2050 (2018).10.1039/C8TA05741A ↗
81. Burton, A. R. et al. Thickness-dependent OER electrocatalysis of epitaxial LaFeO3 thin films. J. Mater. Chem. A 10, 1909 (2022).10.1039/D1TA07142D ↗
82. Ziegler, J. F. (2013). Stopping range of ions in matter (SRIM) (SRIM Company, 2013).
83. de la Peña, F. et al. hyperspy/hyperspy: v2.1.1. https://doi.org/10.5281/zenodo.12724131 (2024).10.5281/zenodo.12724131 ↗
84. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. https://doi.org/10.48550/arXiv.1409.1556 (2015).10.48550/arXiv.1409.1556 ↗
85. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
86. Ward Jr, J. H. Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58, 236–244 (1963).10.1080/01621459.1963.10500845 ↗