Deep-learning-based image segmentation integrated with optical microscopy for automatically searching for two-dimensional materials
Source abstract
Abstract
Abstract Deep-learning algorithms enable precise image recognition based on high-dimensional hierarchical image features. Here, we report the development and implementation of a deep-learning-based image segmentation algorithm in an autonomous robotic system to search for two-dimensional (2D) materials. We trained the neural network based on Mask-RCNN on annotated optical microscope images of 2D materials (graphene, hBN, MoS2, and WTe2). The inference algorithm is run on a 1024 × 1024 px2optical microscope images for 200 ms, enabling the real-time detection of 2D materials. The detection process is robust against changes in the microscopy conditions, such as illumination and color balance, which obviates the parameter-tuning process required for conventional rule-based detection algorithms. Integrating the algorithm with a motorized optical microscope enables the automated searching and cataloging of 2D materials. This development will allow researchers to utilize a large number of 2D materials simply by exfoliating and running the automated searching process. To facilitate research, we make the training codes, dataset, and model weights publicly available.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Electron and X-Ray Spectroscopy Techniques · Machine Learning in Materials Science
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (47)
Show source-provided references
1. Zhao, Z.-Q., Zheng, P., Xu, S.-t. & Wu, X. Object detection with deep learning: a review. IEEE Transactions on Neural Networks and Learning Systems 30, 3212–3232 (2019).10.1109/TNNLS.2018.2876865 ↗
2. Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 91–99 (Neural Information Processing Systems Foundation, 2015).
3. Girshick, R. Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision, 1440–1448 (IEEE, 2015).10.1109/ICCV.2015.169 ↗
4. Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 580–587 (IEEE, 2014).10.1109/CVPR.2014.81 ↗
5. Liu, W. et al. SSD: Single shot multibox detector. European Conference on Computer Vision, 21–37 (Springer, 2016).10.1007/978-3-319-46448-0_2 ↗
6. Garcia-Garcia, A., Orts-Escolano, S., Oprea, S. O., Villena-Martinez, V. & Garcia-Rodriguez, J. A review on deep learning techniques applied to semantic segmentation. Preprint at https://arxiv.org/abs/1704.06857 (2017).10.1016/j.asoc.2018.05.018 ↗
7. Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2481–2495 (2017).10.1109/TPAMI.2016.2644615 ↗
8. Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-assisted Intervention, 234–241 (Springer, 2015).10.1007/978-3-319-24574-4_28 ↗
9. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3431–3440 (IEEE, 2015).10.1109/CVPR.2015.7298965 ↗
10. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2017).10.1109/TPAMI.2017.2699184 ↗
11. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, 2961–2969 (IEEE, 2017).10.1109/ICCV.2017.322 ↗
12. Goodfellow, I. et al. Generative adversarial nets. Advances in Neural Information Processing Systems, 2672–2680 (Neural Information Processing Systems Foundation, 2014).
13. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning. (MIT Press, 2016).
14. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).10.1038/nature14539 ↗
15. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 1097–1105 (Neural Information Processing Systems Foundation, 2012).
16. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).10.1016/j.media.2017.07.005 ↗
17. Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).10.1038/s41592-018-0261-2 ↗
18. Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods https://doi.org/10.1038/s41592-019-0403-1 (2019).10.1038/s41592-019-0403-1 ↗
19. Karpathy, A. Software 2.0. https://medium.com/@karpathy/software-2-0-a64152b37c35 (2017).
20. Novoselov, K. S., Mishchenko, A., Carvalho, A. & Castro Neto, A. H. 2D materials and van der Waals heterostructures. Science 353, aac9439 (2016).10.1126/science.aac9439 ↗
21. Novoselov, K. S. et al. Two-dimensional atomic crystals. Proc. Natl Acad. Sci. USA 102, 10451–10453 (2005).10.1073/pnas.0502848102 ↗
22. Novoselov, K. S. et al. Electric field effect in atomically thin carbon films. Science 306, 666–669 (2004).10.1126/science.1102896 ↗
23. Masubuchi, S. et al. Autonomous robotic searching and assembly of two-dimensional crystals to build van der Waals superlattices. Nat. Commun. 9, 1413 (2018).10.1038/s41467-018-03723-w ↗
24. Masubuchi, S. & Machida, T. Classifying optical microscope images of exfoliated graphene flakes by data-driven machine learning. npj 2D Mater. Appl. 3, 4 (2019).10.1038/s41699-018-0084-0 ↗
25. Nixon, M. S. & Aguado, A. S. Feature Extraction & Image Processing for Computer Vision (Academic Press, 2012).
26. Szeliski, R. Computer Vision: Algorithms and Applications. (Springer Science & Business Media, 2010).10.1007/978-1-84882-935-0 ↗
27. Yu, Y. et al. High-temperature superconductivity in monolayer Bi2Sr2CaCu2O8+δ. Nature 575, 156–163 (2019).10.1038/s41586-019-1718-x ↗
28. Ghosh, S., Das, N., Das, I. & Maulik, U. Understanding deep learning techniques for image segmentation. Preprint at https://arxiv.org/abs/1907.06119 (2019).
29. Pinheiro, P. O., Lin, T.-Y., Collobert, R. & Dollár, P. Learning to refine object segments. European Conference on Computer Vision, 75–91 (Springer, 2016).10.1007/978-3-319-46448-0_5 ↗
30. Zhao, H., Shi, J., Qi, X., Wang, X. & Jia, J. Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2881–2890 (IEEE, 2017).10.1109/CVPR.2017.660 ↗
31. Lin, T.-Y. et al. Microsoft COCO: common objects in context. European Conference on Computer Vision, 740–755 (Springer, 2014).10.1007/978-3-319-10602-1_48 ↗
32. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (IEEE, 2016).10.1109/CVPR.2016.90 ↗
33. Abdulla, W. Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow https://github.com/matterport/Mask_RCNN (2017).
34. Chollet, F. Keras: Deep learning for humans https://github.com/keras-team/keras (2015).
35. Abadi, M. et al. Tensorflow: a system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation, 265–283 (USENIX Association, 2016).
36. Hestness, J. et al. Deep learning scaling is predictable, empirically. Preprint at https://arxiv.org/abs/1712.00409 (2017).
37. Labelbox, “Labelbox,” Online, [Online]. https://labelbox.com (2019).
38. Lin, X. et al. Intelligent identification of two-dimensional nanostructures by machine-learning optical microscopy. Nano Res. 11, 6316–6324 (2018).10.1007/s12274-018-2155-0 ↗
39. Li, H. et al. Rapid and reliable thickness identification of two-dimensional nanosheets using optical microscopy. ACS Nano 7, 10344–10353 (2013).10.1021/nn4047474 ↗
40. Ni, Z. H. et al. Graphene thickness determination using reflection and contrast spectroscopy. Nano Lett. 7, 2758–2763 (2007).10.1021/nl071254m ↗
41. Nolen, C. M., Denina, G., Teweldebrhan, D., Bhanu, B. & Balandin, A. A. High-throughput large-area automated identification and quality control of graphene and few-layer graphene films. ACS Nano 5, 914–922 (2011).10.1021/nn102107b ↗
42. Taghavi, N. S. et al. Thickness determination of MoS2, MoSe2, WS2 and WSe2 on transparent stamps used for deterministic transfer of 2D materials. Nano Res. 12, 1691–1695 (2019).10.1007/s12274-019-2424-6 ↗
43. Zhang, P., Zhong, Y., Deng, Y., Tang, X. & Li, X. A survey on deep learning of small sample in biomedical image analysis. Preprint at https://arxiv.org/abs/1908.00473 (2019).
44. Saito, Y. et al. Deep-learning-based quality filtering of mechanically exfoliated 2D crystals. npj Computational Materials 5, 1–6 (2019).10.1038/s41524-019-0262-4 ↗
45. Han, B. et al. Deep learning enabled fast optical characterization of two-dimensional materials. Preprint at https://arxiv.org/abs/1906.11220 (2019).
46. Greplova, E. et al. Fully automated identification of 2D material samples. Preprint at https://arxiv.org/abs/1911.00066 (2019).
47. Quigley, M. et al. ROS: an open-source Robot Operating System. ICRA Workshop on Open Source Software (Open Robotics, 2009).