nNPipe: a neural network pipeline for automated analysis of morphologically diverse catalyst systems
Source abstract
Abstract
Abstract We describe nNPipe for the automated analysis of morphologically diverse catalyst materials. Automated imaging routines and direct-electron detectors have enabled the collection of large data stacks over a wide range of sample positions at high temporal resolution. Simultaneously, traditional image analysis approaches are slow and hence unsuitable for large data stacks and consequently, researchers have progressively turned towards machine learning and deep learning approaches. Previous studies often detail work on morphologically uniform material systems with clearly discernible features, limited workable image sizes and training data that may be biased due to manual labelling. The nNPipe data-processing method consists of two standalone convolutional neural networks that were exclusively trained on multislice image simulations and enables fast analysis of 2048 × 2048 pixel images. Inference performance compared between idealised and real industrial catalytic samples and insights derived from subsequent data analysis are placed into the context of an automated imaging scenario.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Electron and X-Ray Spectroscopy Techniques · Advanced Electron Microscopy Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (55)
Show source-provided references
1. Morishita, S. et al. Attainment of 40.5 pm spatial resolution using 300 kV scanning transmission electron microscope equipped with fifth-order aberration corrector. Microscopy 67, 46–50 (2018).10.1093/jmicro/dfx122 ↗
2. Jones, L., MacArthur, K. E., Fauske, V. T., van Helvoort, A. T. J. & Nellist, P. D. Rapid estimation of catalyst nanoparticle morphology and atomic-coordination by high-resolution Z-contrast electron microscopy. Nano Lett. 14, 6336–6341 (2014).10.1021/nl502762m ↗
3. Michen, B. et al. Avoiding drying-artifacts in transmission electron microscopy: characterizing the size and colloidal state of nanoparticles. Sci. Rep. 5, 9793 (2015).10.1038/srep09793 ↗
4. Zhao, Y., Tang, R. & Huang, R. Palladium supported on graphitic carbon nitride: an efficient and recyclable heterogeneous catalyst for reduction of nitroarenes and suzuki coupling reaction. Catal. Lett. 145, 1961–1971 (2015).10.1007/s10562-015-1600-x ↗
5. Ward, M. R., Theobald, B., Sharman, J., Boyes, E. D. & Gai, P. L. Direct observations of dynamic PtCo interactions in fuel cell catalyst precursors at the atomic level using E(S)TEM. J. Microsc. 269, 143–150 (2017).10.1111/jmi.12600 ↗
6. He, L. et al. Morphology-dependent catalytic activity of Ru/CeO2 in dry reforming of methane. Molecules 24, 526 (2019).10.3390/molecules24030526 ↗
7. Crozier, P. A. & Hansen, T. W. In situ and operando transmission electron microscopy of catalytic materials. MRS Bull. 40, 38–45 (2015).10.1557/mrs.2014.304 ↗
8. Kumar, S., Varma, R. S., Zboril, R. & Gawande, M. B. Nanoparticle Design and Characterization for Catalytic Applications in Sustainable Chemistry (eds Prinsen, P. & Luque, R.) Ch. 3 (Royal Society of Chemistry, 2019).
9. Gontard, L. C., Ozkaya, D. & Dunin-Borkowski, R. E. A simple algorithm for measuring particle size distributions on an uneven background from TEM images. Ultramicroscopy 111, 101–106 (2010).10.1016/j.ultramic.2010.10.011 ↗
10. Park, C. et al. A multistage, semi-automated procedure for analyzing the morphology of nanoparticles. IIE Trans. 44, 507–522 (2011).10.1080/0740817X.2011.587867 ↗
11. Mondini, S., Ferretti, A. M., Puglisi, A. & Ponti, A. Pebbles and PebbleJuggler: software for accurate, unbiased, and fast measurement and analysis of nanoparticle morphology from transmission electron microscopy (TEM) micrographs. Nanoscale 4, 5356–5372 (2012).10.1039/c2nr31276j ↗
12. Park, C., Huang, J. Z., Ji, J. X. & Ding, Y. Segmentation, inference and classification of partially overlapping nanoparticles. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1–1 (2013).10.1109/TPAMI.2012.163 ↗
13. Epicier, T., Sato, K., Tournus, F. & Konno, T. Chemical composition dispersion in bi-metallic nanoparticles: semi-automated analysis using HAADF-STEM. J. Nanopart. Res. 14, 1106 (2012).10.1007/s11051-012-1106-z ↗
14. Groom, D. J. et al. Automatic segmentation of inorganic nanoparticles in BF TEM micrographs. Ultramicroscopy 194, 25–34 (2018).10.1016/j.ultramic.2018.06.002 ↗
15. Uusimaeki, T., Wagner, T., Lipinski, H.-G. & Kaegi, R. AutoEM: a software for automated acquisition and analysis of nanoparticles. J. Nanopart. Res. 21, 122 (2019).10.1007/s11051-019-4555-9 ↗
16. Clough, R. & Kirkland, A. I. In Advances in Imaging and Electron Physics (Ed. Hawkes, P. W.) Ch. 1 (Academic Press, 2016).10.1016/bs.aiep.2016.09.001 ↗
17. Faruqi, A. R. & McMullan, G. Direct imaging detectors for electron microscopy. Nucl. Instrum. Methods Phys. Res. A 878, 180–190 (2018).10.1016/j.nima.2017.07.037 ↗
18. MacLaren, I., Macgregor, T. A., Allen, C. S. & Kirkland, A. I. Detectors—The ongoing revolution in scanning transmission electron microscopy and why this important to material characterization. APL Mater. 8, 110901 (2020).10.1063/5.0026992 ↗
19. Schorb, M., Haberbosch, I., Hagen, W. J. H., Schwab, Y. & Mastronarde, D. N. Software tools for automated transmission electron microscopy. Nat. Methods 16, 471–477 (2019).10.1038/s41592-019-0396-9 ↗
20. Spurgeon, S. R. et al. Towards data-driven next-generation transmission electron microscopy. Nat. Mater. 20, 274–279 (2021).10.1038/s41563-020-00833-z ↗
21. Lee, B. et al. Statistical characterization of the morphologies of nanoparticles through machine learning based electron microscopy image analysis. ACS Nano 14, 17125–17133 (2020).10.1021/acsnano.0c06809 ↗
22. Muneesawang, P. & Sirisathitkul, C. Size measurement of nanoparticle assembly using multilevel segmented TEM images. J. Nanomater. 2015, 1–8 (2015).10.1155/2015/790508 ↗
23. Wang, X. et al. AutoDetect-mNP: an unsupervised machine learning algorithm for automated analysis of transmission electron microscope images of metal nanoparticles. JACS Au 1, 316–327 (2021).10.1021/jacsau.0c00030 ↗
24. Kalinin, S. V., Sumpter, B. G. & Archibald, R. K. Big–deep–smart data in imaging for guiding materials design. Nat. Mater. 14, 973–980 (2015).10.1038/nmat4395 ↗
25. Kalinin, S. V. et al. Lab on a beam—Big data and artificial intelligence in scanning transmission electron microscopy. MRS Bull. 44, 565–575 (2019).10.1557/mrs.2019.159 ↗
26. Dan, J., Zhao, X. & Pennycook, S. J. A machine perspective of atomic defects in scanning transmission electron microscopy. InfoMat 1, 359–375 (2019).10.1002/inf2.12026 ↗
27. Ge, M., Su, F., Zhao, Z. & Su, D. Deep learning analysis on microscopic imaging in materials science. Mater. Today Nano 11, 100087 (2020).10.1016/j.mtnano.2020.100087 ↗
28. Ede, J. M. Deep learning in electron microscopy. Mach. Learn. Sci. Technol. 2, 011004 (2021).10.1088/2632-2153/abd614 ↗
29. Treder, K. P., Huang, C., Kim, J. S. & Kirkland, A. I. Applications of deep learning in electron microscopy. Microscopy 71, i100–i115 (2022).10.1093/jmicro/dfab043 ↗
30. Madsen, J. et al. A deep learning approach to identify local structures in atomic‐resolution transmission electron microscopy images. Adv. Theory Simul. 1, 1800037 (2018).10.1002/adts.201800037 ↗
31. Groschner, C. K., Choi, C. & Scott, M. C. Machine learning pipeline for segmentation and defect identification from high-resolution transmission electron microscopy data. Microsc. Microanal. 27, 549–556 (2021).10.1017/S1431927621000386 ↗
32. Oktay, A. B. & Gurses, A. Automatic detection, localization and segmentation of nano-particles with deep learning in microscopy images. Micron 120, 113–119 (2019).10.1016/j.micron.2019.02.009 ↗
33. Zafari, S., Eerola, T., Ferreira, P., Kälviäinen, H. & Bovik A. Automated segmentation of nanoparticles in BF TEM images by U-Net binarization and branch and bound. Computer Analysis of Images and Patterns—CAIP 2019 113–125 (Springer, 2019).10.1007/978-3-030-29888-3_10 ↗
34. Colliard-Granero, A. et al. Deep learning for the automation of particle analysis in catalyst layers for polymer electrolyte fuel cells. Nanoscale 14, 10–18 (2021).10.1039/D1NR06435E ↗
35. Zhang, F. et al. Rodlike nanoparticle parameter measurement method based on improved mask R-CNN segmentation. SIViP 15, 579–587 (2021).10.1007/s11760-020-01779-0 ↗
36. Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015 234–241 (Springer, 2015).10.1007/978-3-319-24574-4_28 ↗
37. Horwath, J. P., Zakharov, D. N., Mégret, R. & Stach, E. A. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. NPJ Comput. Mater. 6, 108 (2020).10.1038/s41524-020-00363-x ↗
38. Saaim, K. M., Afridi, S. K., Nisar, M. & Islam, S. In search of best automated model: explaining nanoparticle TEM image segmentation. Ultramicroscopy 233, 113437 (2021).10.1016/j.ultramic.2021.113437 ↗
39. Yao, L., Ou, Z., Luo, B., Xu, C. & Chen, Q. Machine learning to reveal nanoparticle dynamics from liquid- phase TEM videos. ACS Cent. Sci. 6, 1421–1430 (2020).10.1021/acscentsci.0c00430 ↗
40. Jamali, V. et al. Anomalous nanoparticle surface diffusion in LCTEM is revealed by deep learning-assisted analysis. Proc. Natl Acad. Sci. USA 118, e2017616118 (2021).10.1073/pnas.2017616118 ↗
41. Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 2481–2495 (2017).10.1109/TPAMI.2016.2644615 ↗
42. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2961–2969 (IEEE, 2017).10.1109/ICCV.2017.322 ↗
43. Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: unified, real-time object detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 779–788 (2016).10.1109/CVPR.2016.91 ↗
44. Redmon, J. & Farhadi, A. YOLOv3: an incremental improvement. Preprint at https://arxiv.org/abs/1804.02767 (2018).
45. Frei, M. & Kruis, F. E. Image-based size analysis of agglomerated and partially sintered particles via convolutional neural networks. Powder Technol. 360, 324–336 (2020).10.1016/j.powtec.2019.10.020 ↗
46. Cowley, J. M. & Moodie, A. F. The scattering of electrons by atoms and crystals. I. A new theoretical approach. Acta Cryst. 10, 609–619 (1957).10.1107/S0365110X57002194 ↗
47. Goodman, P. & Moodie, A. F. Numerical evaluations of N-beam wave functions in electron scattering by the multi-slice method. Acta Cryst. A30, 280–290 (1974).10.1107/S056773947400057X ↗
48. Lobato, I. & van Dyck, D. MULTEM: a new multislice program to perform accurate and fast electron diffraction and imaging simulations using graphics processing units with CUDA. Ultramicroscopy 156, 9–17 (2015).10.1016/j.ultramic.2015.04.016 ↗
49. Lobato, I., Aert, S. V. & Verbeeck, J. Progress and new advances in simulating electron microscopy datasets using MULTEM. Ultramicroscopy 168, 17–27 (2016).10.1016/j.ultramic.2016.06.003 ↗
50. Ito, E., Sato, T., Sano, D., Utagawa, E. & Kato, T. Virus particle detection by convolutional neural network in transmission electron microscopy images. Food Environ. Virol. 10, 201–208 (2018).10.1007/s12560-018-9335-7 ↗
51. Bell, C. G. et al. Trainable segmentation for transmission electron microscope images of inorganic nanoparticles, J. Microsc. https://doi.org/10.1111/jmi.13110. (2022).10.1111/jmi.13110 ↗
52. Typke, D. & Dierksen, K. Determination of image aberrations in high-resolution electron microscopy using diffractogram and cross-correlation methods. Optik 4, 155–166 (1995).
53. Scott, D. W. in Kernel Density Estimators Ch. 6 (John Wiley & Sons, Incorporated, 2015).
54. Shiojiri, M., Saito, Y., Okada, H. & Sasaki, H. Densities of amorphous thin films. Jpn J. Appl. Phys. 18, 1931 (1979).10.1143/JJAP.18.1931 ↗
55. Wypych, G. Databook of Antistatics (ChemTec Publishing, 2014).