Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images
Source abstract
Abstract
Abstract Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Electron and X-Ray Spectroscopy Techniques · Advanced Electron Microscopy Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (43)
Show source-provided references
1. Zheng, H., Meng, Y. S. & Zhu, Y. Frontiers of in situ electron microscopy. MRS Bull. 40, 12–18 (2015).10.1557/mrs.2014.305 ↗
2. Tao, F. & Salmeron, M. In situ studies of chemistry and structure of materials in reactive environments. Science 331, 171–174 (2011).10.1126/science.1197461 ↗
3. Taheri, M. L. et al. Current status and future directions for in situ transmission electron microscopy. Ultramicroscopy 170, 86–95 (2016).10.1016/j.ultramic.2016.08.007 ↗
4. Hill, J. et al. Materials science with large-scale data and informatics: unlocking new opportunities. MRS Bull. 41, 399–409 (2016).10.1557/mrs.2016.93 ↗
5. Simonsen, S. B. et al. Direct observations of oxygen-induced platinum nanoparticle ripening studied by in situ TEM. J. Am. Chem. Soc. 132, 7968–7975 (2010).10.1021/ja910094r ↗
6. Badea, M. et al. The use of deep learning in image segmentation, classification, and detection. arxiv 1605.09612 (2016).
7. Chen, X. W. & Lin, X. Big data deep learning: challenges and perspectives. IEEE Access 2, 514–525 (2014).10.1109/ACCESS.2014.2325029 ↗
8. Dheeba, J. & Tamil Selvi, S. Classification of malignant and benign microcalcification using SVM classifier. 2011 Int. Conf. Emerg. Trends Electr. Comput. Technol. ICETECT 2011, 686–690 (2011).10.1109/ICETECT.2011.5760205 ↗
9. Bengio, Y., Lee, D.-H., Bornschein, J., Mesnard, T. & Lin, Z. Towards biologically plausible deep learning. arxiv 1502.04156 (2015).
10. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).
11. Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).10.1038/s41592-019-0403-1 ↗
12. Yang, W., Zhang, X., Tian, Y., Wang, W. & Xue, J.-H. Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimedia 21, 1–17 (2018).
13. Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. arxiv 1611.03530 (2017).
14. Wang, Z. Deep learning for Image segmentation-a short survey. arxiv 1904.08483 (2019).
15. Dietterich, T. Overftting Overlifting and undercomputing in machine learning. ACM Comput. Surv. 27, 326–327 (1995).10.1145/212094.212114 ↗
16. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
17. Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Proc. IEEE Int. Conf. Comput. Vis. 2017, 618–626 (2017).
18. Umehara, M. et al. Analyzing machine learning models to accelerate generation of fundamental materials insights. npj Comput. Mater 5, 1–9 (2019).
19. Madsen, J. et al. A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images. Adv. Theory. Simul. 1, 1–12 (2018).10.1002/adts.201800037 ↗
20. Schneider, N. M., Park, J. H., Norton, M. M., Ross, F. M. & Bau, H. H. Automated analysis of evolving interfaces during in situ electron microscopy. Adv. Struct. Chem.Imaging 2, 1–11 (2017).
21. Ziatdinov, M. et al. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS Nano 11, 12742–12752 (2017).10.1021/acsnano.7b07504 ↗
22. Zakharov, D. N. et al. Towards Real time quantitative analysis of supported nanoparticle ensemble evolution investigated by environmental TEM. Microsc. Microanal. 24, 540–541 (2018).10.1017/S1431927618003197 ↗
23. Hansen, T. W., Delariva, A. T., Challa, S. R. & Datye, A. K. Sintering of catalytic nanoparticles: particle migration or ostwald ripening? Acc. Chem. Res. 46, 1720–1730 (2013).10.1021/ar3002427 ↗
24. Ostwald, W. Uber die vermeintliche Isomerie des roten und gelben Quecksilberoxyds und die Oberflachenspannung fester Korper. Zeritschrift fur Phys. Chem. 34, 495 (1900).
25. Shen, D., Wu, G. & Suk, H.-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017).10.1146/annurev-bioeng-071516-044442 ↗
26. Wilson, R. S. et al. Automated single particle detection and tracking for large microscopy datasets. R. Soc. Open Sci. 3, 160225 (2016).10.1098/rsos.160225 ↗
27. Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2481–2495 (2017).10.1109/TPAMI.2016.2644615 ↗
28. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. IEEE Conf. Comput. Vis. Pattern Recognit. 3431–3440 (2015).10.1109/CVPR.2015.7298965 ↗
29. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. Lect. Notes Comput. Sci. 9351, 234–241 (2015).10.1007/978-3-319-24574-4_28 ↗
30. Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks?. Adv. Neural. Inf. Process Syst 27, 3320–3328 (2014).
31. Baudat, G. & Anouar, F. Kernel-based methods and function approximation. Proc. Int. Jt. Conf. Neural. Netw. 2, 1244–1249 (2001).10.1109/IJCNN.2001.939539 ↗
32. Lu, Z. et al. The expressive power of neural networks: a view form the width. Adv. Neural. Inf. Proc. Sys. 30 (2017).
33. Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. How does batch normalization help optimization?. Adv. Neural. Inf. Process. Syst. 31, 2488–2498 (2018).
34. Maas, A. L., Hannun, A. Y. & Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. Icml ’13 28, 6 (2013).
35. Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man. Cybern. SMC-9, 62–66 (1979).10.1109/TSMC.1979.4310076 ↗
36. Xu, X., Xu, S., Jin, L. & Song, E. Characteristic analysis of Otsu threshold and its applications. Pattern Recognit. Lett. 32, 956–961 (2011).10.1016/j.patrec.2011.01.021 ↗
37. Kabkab, M., Hand, E. & Chellappa, R. On the size of convolutional neural networks and generalization performance. In 2016 23rd International Conference on Pattern Recognition (ICPR) 3572–3577 (IEEE, 2016).10.1109/ICPR.2016.7900188 ↗
38. Han, S. et al. Learning both weights and connections for efficient neural network. Adv. Neural. Inf. Proc. Sys. 2015, 1135–1143 (2015).
39. van der Walt, S. et al. scikit-image: image processing in Python. PeerJ 2, e453 (2014).10.7717/peerj.453 ↗
40. Paszke, A. et al. Automatic differentiation in PyTorch. NIPS 2017 (2017).
41. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. ICLR 2015, 1–15 (2015).
42. Nair, V. & Hinton, G. Rectified linear units improve restricted Boltzmann machines. Int. Conf. on Mach. Learn. (2010).
43. Scherer, D., Müller, A. & Behnke, S. Evaluation of pooling operations in convolutional architectures for object recocnition. Art. Neural. Net-ICANN 6354, 92–101 (2010).