journal article·2024-12-18Open access

Enabling dynamic 3D coherent diffraction imaging via adaptive latent space tuning of generative autoencoders

Alexander Scheinker, Reeju Pokharel

npj Computational Materials· Volume 10· Issue 1

Abstract

Abstract Coherent diffraction imaging (CDI) is an advanced non-destructive 3D X-ray imaging technique for measuring a sample’s electron density. The main challenge of CDI is loss of phase information in diffraction intensity measurements, resulting in lengthy iterative reconstruction processes that can return non-unique solutions, which pose challenges for experiments attempting to track dynamic sample evolution through multiple states. As the increased brightness of fourth-generation light sources enables faster sample measurements and drives operando experiments with Bragg CDI, there is a growing need for faster reconstruction techniques that can keep pace. We have developed an adaptive generative autoencoder approach for uniquely tracking a sample’s electron density as it dynamically evolves. Our approach adaptively tunes the low-dimensional latent embedding of a generative autoencoder, enabling a computationally efficient manner to account for time-varying shifting distributions in real-time. Analytic proof of convergence is provided as well as numerical demonstration of sample tracking with noisy measurements.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced X-ray Imaging Techniques · Digital Holography and Microscopy · Advanced Electron Microscopy Techniques and Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

References (75)

Show source-provided references
1. Miao, J., Charalambous, P., Kirz, J. & Sayre, D. Extending the methodology of X-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens. Nature 400, 342–344 (1999).10.1038/22498 ↗
2. Miao, J., Sandberg, R. L. & Song, C. Coherent X-ray diffraction imaging. IEEE J. Sel. Top. quantum Electron. 18, 399–410 (2011).10.1109/JSTQE.2011.2157306 ↗
3. Song, C. et al. Quantitative imaging of single, unstained viruses with coherent x rays. Phys. Rev. Lett. 101, 158101 (2008).10.1103/PhysRevLett.101.158101 ↗
4. Miao, J. et al. Imaging whole Escherichia coli bacteria by using single-particle x-ray diffraction. Proc. Natl Acad. Sci. 100, 110–112 (2003).10.1073/pnas.232691299 ↗
5. Miao, J. et al. Quantitative image reconstruction of GaN quantum dots from oversampled diffraction intensities alone. Phys. Rev. Lett. 95, 085503 (2005).10.1103/PhysRevLett.95.085503 ↗
6. Williams, G. J. et al. High-resolution X-ray imaging of plasmodium falciparum-infected red blood cells. Cytom. Part A J. Int. Soc. Anal. Cytol. 73, 949–957 (2008).10.1002/cyto.a.20616 ↗
7. Nishino, Y., Takahashi, Y., Imamoto, N., Ishikawa, T. & Maeshima, K. Three-dimensional visualization of a human chromosome using coherent X-ray diffraction. Phys. Rev. Lett. 102, 018101 (2009).10.1103/PhysRevLett.102.018101 ↗
8. Clark, J., Huang, X., Harder, R. & Robinson, I. High-resolution three-dimensional partially coherent diffraction imaging. Nat. Commun. 3, 1–6 (2012).10.1038/ncomms1994 ↗
9. Robinson, I. K., Vartanyants, I. A., Williams, G., Pfeifer, M. & Pitney, J. Reconstruction of the shapes of gold nanocrystals using coherent x-ray diffraction. Phys. Rev. Lett. 87, 195505 (2001).10.1103/PhysRevLett.87.195505 ↗
10. Pfeifer, M. A., Williams, G. J., Vartanyants, I. A., Harder, R. & Robinson, I. K. Three-dimensional mapping of a deformation field inside a nanocrystal. Nature 442, 63–66 (2006).10.1038/nature04867 ↗
11. Robinson, I. & Harder, R. Coherent X-ray diffraction imaging of strain at the nanoscale. Nat. Mater. 8, 291–298 (2009).10.1038/nmat2400 ↗
12. McDowell, D. L. A perspective on trends in multiscale plasticity. Int. J. Plasticity 26, 1280–1309 (2010).10.1016/j.ijplas.2010.02.008 ↗
13. Crabtree, G. & Sarrao, J. Opportunities for mesoscale science. MRS Bull. 37, 1079–1088 (2012).10.1557/mrs.2012.274 ↗
14. Huang, X., Harder, R., Leake, S., Clark, J. & Robinson, I. Three-dimensional Bragg coherent diffraction imaging of an extended ZnO crystal. J. Appl. Crystallogr. 45, 778–784 (2012).10.1107/S0021889812018900 ↗
15. Cherukara, M. J. et al. Three-dimensional X-ray diffraction imaging of dislocations in polycrystalline metals under tensile loading. Nat. Commun. 9, 1–6 (2018).10.1038/s41467-018-06166-5 ↗
16. Robinson, I., Clark, J. & Harder, R. Materials science in the time domain using bragg coherent diffraction imaging. J. Opt. 18, 054007 (2016).10.1088/2040-8978/18/5/054007 ↗
17. Fohtung, E., Karpov, D., & Baumbach, T. Bragg Coherent Diffraction Imaging Techniques at 3rd and 4th Generation Light Sources pp. 203–215. Springer, Cham https://doi.org/10.1007/978-3-319-99465-9_8 (2018).10.1007/978-3-319-99465-9_8 ↗
18. Clark, J. et al. Ultrafast three-dimensional imaging of lattice dynamics in individual gold nanocrystals. Science 341, 56–59 (2013).10.1126/science.1236034 ↗
19. Clark, J. N. et al. Imaging transient melting of a nanocrystal using an x-ray laser. Proc. Natl Acad. Sci. 112, 7444–7448 (2015).10.1073/pnas.1417678112 ↗
20. Cherukara, M. J. et al. Ultrafast three-dimensional integrated imaging of strain in core/shell semiconductor/metal nanostructures. Nano Lett. 17, 7696–7701 (2017).10.1021/acs.nanolett.7b03823 ↗
21. Ulvestad, A. et al. Bragg coherent diffractive imaging of zinc oxide acoustic phonons at picosecond timescales. Sci. Rep. 7, 9823 (2017).10.1038/s41598-017-09999-0 ↗
22. Karpov, D., Fohtung, E. Bragg coherent diffractive imaging of strain at the nanoscale. J. Appl. Phys. 125(12), https://doi.org/10.1063/1.5054294 (2019).10.1063/1.5054294 ↗
23. Gao, Y. et al. Modeling and experimental validation of dynamical effects in bragg coherent x-ray diffractive imaging of finite crystals. Phys. Rev. B 106, 184111 (2022).10.1103/PhysRevB.106.184111 ↗
24. Ulvestad, A. et al. Topological defect dynamics in operando battery nanoparticles. Science 348, 1344–1347 (2015).10.1126/science.aaa1313 ↗
25. Chen, C.-C., Miao, J., Wang, C. & Lee, T. Application of optimization technique to noncrystalline x-ray diffraction microscopy: Guided hybrid input-output method. Phys. Rev. B Condens. Matter Mater. Phys. 76, 064113 (2007).10.1103/PhysRevB.76.064113 ↗
26. Truong, N. X., Whittaker, E. & Denecke, M. A. Phase retrieval of coherent diffractive images with global optimization algorithms. J. Appl. Crystallogr. 50, 1637–1645 (2017).10.1107/S1600576717013012 ↗
27. Miao, J., Sayre, D. & Chapman, H. Phase retrieval from the magnitude of the Fourier transforms of nonperiodic objects. JOSA A 15, 1662–1669 (1998).10.1364/JOSAA.15.001662 ↗
28. Fienup, J. R. Reconstruction of an object from the modulus of its Fourier transform. Opt. Lett. 3, 27–29 (1978).10.1364/OL.3.000027 ↗
29. Fienup, J. R. Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758–2769 (1982).10.1364/AO.21.002758 ↗
30. Elser, V. Solution of the crystallographic phase problem by iterated projections. Acta Crystallogr. Sect. A Found. Crystallogr. 59, 201–209 (2003).10.1107/S0108767303002812 ↗
31. Chen, C.-C., Miao, J., Wang, C. & Lee, T. Application of optimization technique to noncrystalline x-ray diffraction microscopy: Guided hybrid input-output method. Phys. Rev. B 76, 064113 (2007).10.1103/PhysRevB.76.064113 ↗
32. Marchesini, S. Invited article: A unified evaluation of iterative projection algorithms for phase retrieval. Rev. Sci. Instrum. 78, 011301 (2007).10.1063/1.2403783 ↗
33. Bauschke, H. H., Combettes, P. L. & Luke, D. R. Hybrid projection–reflection method for phase retrieval. JOSA A 20, 1025–1034 (2003).10.1364/JOSAA.20.001025 ↗
34. Luke, D. R. Relaxed averaged alternating reflections for diffraction imaging. Inverse Probl. 21, 37 (2004).10.1088/0266-5611/21/1/004 ↗
35. Sayre, D. Imaging processes and coherence in physics. Springe. Lect. Notes Phys. 112, 229–235 (1980).10.1007/3-540-09727-9_82 ↗
36. Colombo, A., Galli, D. E., De Caro, L., Scattarella, F. & Carlino, E. Facing the phase problem in coherent diffractive imaging via memetic algorithms. Sci. Rep. 7, 1–12 (2017).10.1038/srep42236 ↗
37. Fevola, G., Bergbäck Knudsen, E., Ramos, T., Carbone, D., Wenzel Andreasen, J. A Monte Carlo ray-tracing simulation of coherent X-ray diffractive imaging. J. Synchrotron Radiat. 27(1), 134–145 (2020)10.1107/S1600577519014425 ↗
38. Wilkin, M. J. et al. Experimental demonstration of coupled multi-peak bragg coherent diffraction imaging with genetic algorithms. Phys. Rev. B 103, 214103 (2021).10.1103/PhysRevB.103.214103 ↗
39. Attal, B. & O’Toole, M. Towards mixed-state coded diffraction imaging. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, (IEEE, 2022).
40. Chatelier, C. et al. Unveiling core–shell structure formation in a ni3fe nanoparticle with in situ multi-bragg coherent diffraction imaging (ACS nano, 2024).10.1021/acsnano.3c11534 ↗
41. Ulvestad, A. et al. Coherent diffractive imaging of time-evolving samples with improved temporal resolution. Phys. Rev. B 93, 184105 (2016).10.1103/PhysRevB.93.184105 ↗
42. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 https://doi.org/10.48550/arXiv.1511.04587 (2016).10.48550/arXiv.1511.04587 ↗
43. Baraja, M. C. & Bhattacharya, K. Learning the effective adhesive properties of heterogeneous substrates. Extrem. Mech. Lett. 65, 102110 (2023).10.1016/j.eml.2023.102110 ↗
44. Li, Z. et. al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations. Preprint at https://openreview.net/forum?id=c8P9NQVtmnO (2021).
45. Badrinarayanan, V., Kendall, A. & Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2481–2495 (2017).10.1109/TPAMI.2016.2644615 ↗
46. Jing, L. & Tian, Y. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 4037–4058 (2020).10.1109/TPAMI.2020.2992393 ↗
47. Li, S., Jia, K., Wen, Y., Liu, T. & Tao, D. Orthogonal deep neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 43, 1352–1368 (2019).10.1109/TPAMI.2019.2948352 ↗
48. Dong, W. et al. Denoising prior driven deep neural network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 41, 2305–2318 (2018).10.1109/TPAMI.2018.2873610 ↗
49. Bessadok, A., Mahjoub, M. A. & Rekik, I. Graph neural networks in network neuroscience. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, (IEEE, 2022).10.1109/TPAMI.2022.3209686 ↗
50. Ziabari, A., Rose, D. C., Shirinifard, A. & Solecki, D. YOLO2U-Net: Detection-guided 3D instance segmentation for microscopy. Pattern Recognit. Lett. 181, 37–42 (2024).10.1016/j.patrec.2024.03.015 ↗
51. Ziabari, A. et al. Enabling rapid X-ray CT characterisation for additive manufacturing using CAD models and deep learning-based reconstruction. npj Computational Mater. 9, 91 (2023).10.1038/s41524-023-01032-5 ↗
52. Shen, Y.-F., Pokharel, R., Nizolek, T. J., Kumar, A. & Lookman, T. Convolutional neural network-based method for real-time orientation indexing of measured electron backscatter diffraction patterns. Acta Materialia 170, 118–131 (2019).10.1016/j.actamat.2019.03.026 ↗
53. Cherukara, M. J., Nashed, Y. S. & Harder, R. J. Real-time coherent diffraction inversion using deep generative networks. Sci. Rep. 8, 1–8 (2018).10.1038/s41598-018-34525-1 ↗
54. Scheinker, A. & Pokharel, R. Adaptive 3D convolutional neural network-based reconstruction method for 3D coherent diffraction imaging. J. Appl. Phys. 128, 184901 (2020).10.1063/5.0014725 ↗
55. Wu, L. et al. Three-dimensional coherent X-ray diffraction imaging via deep convolutional neural networks. npj Computational Mater. 7, 1–8 (2021).10.1038/s41524-021-00644-z ↗
56. Chan, H. et al. Rapid 3D nanoscale coherent imaging via physics-aware deep learning. Appl. Phys. Rev. 8, 021407 (2021).10.1063/5.0031486 ↗
57. Yao, Y. et al. AutoPhaseNN: unsupervised physics-aware deep learning of 3D nanoscale Bragg coherent diffraction imaging. npj Computational Mater. 8, 1–8 (2022).10.1038/s41524-022-00803-w ↗
58. Meziere, J., Carpenter, A. H., Pateras, A., Harder, R. & Sandberg, R. L. Atomic resolution coherent x-ray imaging with physics-based phase retrieval. npj Computational Mater. 10, 167 (2024).10.1038/s41524-024-01340-4 ↗
59. Nugent, K., Peele, A., Quiney, H. & Chapman, H. Diffraction with wavefront curvature: a path to unique phase recovery. Acta Crystallogr. Sect. A: Found. Crystallogr. 61, 373–381 (2005).10.1107/S010876730501055X ↗
60. Scheinker, A. Latent Space Movies. https://github.com/alexscheinker/Adaptive-3D-Autoencoder-Latent-Space-Tuning-for-CDI Accessed (2023).
61. Scheinker, A. & Scheinker, D. Bounded extremum seeking with discontinuous dithers. Automatica 69, 250–257 (2016).10.1016/j.automatica.2016.02.023 ↗
62. Scheinker, A. & Krstić, M. Minimum-seeking for CLFs: Universal semiglobally stabilizing feedback under unknown control directions. IEEE Trans. Autom. Control 58, 1107–1122 (2012).10.1109/TAC.2012.2225514 ↗
63. Scheinker, A. Simultaneous stabilization and optimization of unknown, time-varying systems. In: 2013 American Control Conference, pp. 2637–2642 https://doi.org/10.1109/ACC.2013.6580232 (2013).10.1109/ACC.2013.6580232 ↗
64. Scheinker, A., Huang, X. & Wu, J. Minimization of betatron oscillations of electron beam injected into a time-varying lattice via extremum seeking. IEEE Trans. Control Syst. Technol. 26, 336–343 (2017).10.1109/TCST.2017.2664728 ↗
65. Scheinker, A., Bland, M., Krstić, M. & Audia, J. Extremum seeking-based optimization of high voltage converter modulator rise-time. IEEE Trans. Control Syst. Technol. 22, 34–43 (2013).10.1109/TCST.2013.2240387 ↗
66. Scheinker, A., Cropp, F., Paiagua, S. & Filippetto, D. An adaptive approach to machine learning for compact particle accelerators. Sci. Rep. 11, 1–11 (2021).10.1038/s41598-021-98785-0 ↗
67. Khalil, H.K. Nonlinear systems, 120,651,129. (Prentice-Hall, Upper Saddle River, 2002).
68. Partha, P., Sajjadi, M. S. M., Antonio, V. & Michael, B. From variational to deterministic autoencoders. 8th International Conference on Learning Representations (2020).
69. Scheinker, A. Adaptive machine learning for time-varying systems: low dimensional latent space tuning. J. Instrum. 16, 10008 (2021).10.1088/1748-0221/16/10/P10008 ↗
70. Scheinker, A., Cropp, F. & Filippetto, D. Adaptive autoencoder latent space tuning for more robust machine learning beyond the training set for six-dimensional phase space diagnostics of a time-varying ultrafast electron-diffraction compact accelerator. Phys. Rev. E 107, 045302 (2023).10.1103/PhysRevE.107.045302 ↗
71. Han, Y. et al. Dynamic neural networks: A survey. In IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE, 2021).
72. Newton, M. C. Compressed sensing for phase retrieval. Phys. Rev. E 85, 056706 (2012).10.1103/PhysRevE.85.056706 ↗
73. Candes, E. J., Li, X. & Soltanolkotabi, M. Phase retrieval via Wirtinger flow: Theory and algorithms. IEEE Trans. Inf. Theory 61, 1985–2007 (2015).10.1109/TIT.2015.2399924 ↗
74. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (IEEE, 2017).10.1109/CVPR.2017.35 ↗
75. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-assisted intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241 https://doi.org/10.1007/978-3-319-24574-4_28 (2015).10.1007/978-3-319-24574-4_28 ↗