Multimodal Sparse Reconstruction and Deep Generative Networks: A Paradigm Shift in MR-PET Neuroimaging
Source abstract
Abstract
A novel multimodal super-resolution framework is introduced, combining GAN-based synthesis, perceptual constraints, and joint low-rank sparsity regularization to noticeably enhance MR-PET image quality. The architecture integrates modality-specific ResNet encoders, a transformer-based attention fusion block, and a multi-scale PatchGAN discriminator. Training is guided by a hybrid loss function incorporating adversarial, pixel-wise, perceptual (VGG19), and structured Hankel constraints. The proposed method outperforms all baselines in PSNR, SSIM, LPIPS, and diagnostic confidence metrics. Clinical PET metrics, such as SUV recovery and lesion detectability, show substantial improvement. A thorough analysis of computational complexity, dataset composition, training reproducibility, and motion compensation is provided. These findings are visually supported by processed scan panels and benchmark tables. This framework advances reproducible and interpretable hybrid neuroimaging with strong clinical and technical validation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Medical Imaging Techniques and Applications · Advanced Image Processing Techniques · Advanced MRI Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Information Technology, Warsaw University of Life Sciences, Nowoursynowska St. 159 Building 34, 02-776 Warsaw, Poland
References (74)
Show source-provided references
1. An outlook on future design of hybrid PET/MRI systems10.1118/1.3633909 ↗
2. Positron emission tomography/magnetic resonance imaging evaluation of lung cancer: Current status and future prospects10.1097/RTI.0000000000000062 ↗
3. Multimodality imaging of structure and function10.1088/0031-9155/53/4/R01 ↗
4. Kim, J.K., Kwon Lee, J., and Mu Lee, K. (2016, January 27–30). Accurate image super-resolution using very deep convolutional networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
5. Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017, January 21–26). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
6. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. Computer Vision—ECCV 2018. ECCV 2018, Springer.10.1007/978-3-030-01234-2_18 ↗
7. Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., and Sun, J. (2019, January 15–20). Meta-SR: A Magnification-Arbitrary Network for Super-Resolution. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00167 ↗
8. Sparse MRI: The application of compressed sensing for rapid MR imaging10.1002/mrm.21391 ↗
9. Compressed sensing10.1109/TIT.2006.871582 ↗
10. Image super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
11. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
12. Dong, C., Loy, C.C., He, K., and Tang, X. (2014). Learning a Deep Convolutional Network for Image Super-Resolution. Computer Vision—ECCV 2014, Springer.10.1007/978-3-319-10593-2_13 ↗
13. Johnson, J., Alahi, A., and Fei-Fei, L. (2016). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Computer Vision—ECCV 2016, Springer.10.1007/978-3-319-46475-6_43 ↗
14. Multimodal Image fusion via self-supervised transformer10.1109/JSEN.2023.3263336 ↗
15. STFNet: Self-supervised transformer for infrared and visible image fusion10.1109/TETCI.2024.3352490 ↗
16. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. (2018, January 18–23). Residual dense network for image super-resolution. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00262 ↗
17. Generative adversarial nets
18. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein GAN. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
19. Improved Training of Wasserstein GANs
20. MRI-to-PET cross-modality translation using globally & locally aware GAN (GLA-GAN) for multi-modal diagnosis of alzheimer’s disease10.1016/j.premed.2025.100004 ↗
21. Huang, J., Ding, W., Lv, J., Yang, J., Dong, H., Del Ser, J., Xia, J., Ren, T., Wong, S., and Yang, G. (2021). Edge-enhanced dual discriminator generative adversarial network for fast MRI with Parallel imaging using multi-view information. arXiv.10.1007/s10489-021-03092-w ↗
22. Second order total generalized variation (TGV) for MRI10.1002/mrm.22595 ↗
23. A fast iterative shrinkage–thresholding algorithm for linear inverse problems10.1137/080716542 ↗
24. Generalized autocalibrating partially parallel acquisitions (GRAPPA)10.1002/mrm.10171 ↗
25. Cross-modality deep feature learning for brain tumor segmentation10.1016/j.patcog.2020.107562 ↗
26. VoxelMorph: A learning framework for deformable medical image registration10.1109/TMI.2019.2897538 ↗
27. Ravishankar, S., and Bresler, Y. (2013, January 7–11). Sparsifying Transform Learning for Compressed Sensing MRI. Proceedings of the 2013 IEEE 10th International Symposium on Biomedical Imaging, San Francisco, CA, USA.10.1109/ISBI.2013.6556401 ↗
28. Rousseau, F. Multiscale brain MRI super-resolution using deep 3D convolutional networks10.1016/j.compmedimag.2019.101647 ↗
29. Joint MR–PET reconstruction using a multi-channel image regularizer
30. Mass univariate analysis of event-related brain potentials/fields I: A critical tutorial review10.1111/j.1469-8986.2011.01273.x ↗
31. A generalized structured low-rank matrix completion algorithm for MR image recovery10.1109/TMI.2018.2886290 ↗
32. A deep cascade of convolutional neural networks for MR image reconstruction10.1109/TMI.2017.2760978 ↗
33. Learning-Based Optimization of the Under-Sampling Pattern in MRI10.1109/TCI.2020.3006727 ↗
34. Motion correction with PROPELLER MRI: Application to head motion and free-breathing cardiac imaging10.1002/(SICI)1522-2594(199911)42:5<963::AID-MRM17>3.0.CO;2-L ↗
35. Bridson, R. (2007). Fast Poisson Disk Sampling in Arbitrary Dimensions, Association for Computing Machinery. ACM SIGGRAPH Sketches.10.1145/1278780.1278807 ↗
36. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA.
37. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
38. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
39. Simard, P.Y., Steinkraus, D., and Platt, J.C. (2003, January 6). Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. Proceedings of the Seventh International Conference on Document Analysis and Recognition, Edinburgh, UK.
40. A deep learning framework for unsupervised affine and deformable image registration10.1016/j.media.2018.11.010 ↗
41. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
42. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7–13). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.123 ↗
43. Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration for Images and Surfaces
44. Shen, Z., Han, X., Xu, Z., and Niethammer, M. (2019, January 15–20). Networks for Joint Affine and Non-Parametric Image Registration. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00435 ↗
45. Computing large deformation metric mappings via geodesic flows of diffeomorphisms10.1023/B:VISI.0000043755.93987.aa ↗
46. A fast diffeomorphic image registration algorithm10.1016/j.neuroimage.2007.07.007 ↗
47. Balakrishnan, G., Zhao, A., Guttag, J., and Sabuncu, M.R. (2018, January 18–23). An Unsupervised Learning Model for Deformable Medical Image Registration. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00964 ↗
48. Zhang, J., He, X., Qing, L., Gao, F., and Wang, B. (2022). BPGAN: Brain PET synthesis from MRI using generative adversarial network for multi-modal alzheimer’s disease diagnosis. Comput. Methods Programs Biomed., 217.10.1016/j.cmpb.2022.106676 ↗
49. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., and Matas, J. (2018, January 18–23). DeblurGAN: Blind Motion Deblurring Using Conditional GANs. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00854 ↗
50. Nah, S., Kim, T.H., and Lee, K.M. (2017, January 21–26). Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.35 ↗
51. Adaptive non-local means denoising of MR images with spatially varying noise levels10.1002/jmri.22003 ↗
52. Cohen, T.S., and Welling, M. (2016, January 20–22). Group Equivariant Convolutional Networks. In Proceedings of The 33rd International Conference on Machine Learning, New York, NY, USA. Available online: https://proceedings.mlr.press/v48/cohenc16.html.
53. Chen, Y., Xie, Y., Shi, F., Zhou, Z., Lin, W., and Wang, Y. (2018, January 4–7). Efficient and Accurate MRI Super-Resolution Using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network. Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI), Washington, DC, USA.
54. Noise and signal estimation in magnitude MRI and rician-distributed images: A LMMSE approach10.1109/TIP.2008.925382 ↗
55. Wang, S., Xiao, T., Liu, Q., and Zheng, H. (2020). Deep learning for fast MR imaging: A review for learning reconstruction from incomplete k-space data. arXiv.10.1016/j.bspc.2021.102579 ↗
56. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information10.1109/TIT.2005.862083 ↗
57. PET image reconstruction using deep image prior10.1109/TMI.2018.2888491 ↗
58. DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction10.1109/TMI.2017.2785879 ↗
59. Learning a variational network for reconstruction of accelerated MRI data10.1002/mrm.26977 ↗
60. Efficient and Accurate MRI Super-Resolution Using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network
61. Xie, T., Cui, Z.-X., Luo, C., Wang, H., Liu, C., Zhang, Y., Wang, X., Zhu, Y., Chen, G., and Liang, D. (2023). Joint diffusion: Mutual consistency-driven diffusion model for PET-MRI co-reconstruction. arXiv.10.1088/1361-6560/ad6117 ↗
62. Compressed sensing MRI Reconstruction using a generative adversarial network with a cyclic loss10.1109/TMI.2018.2820120 ↗
63. VoxResNet: Deep Voxelwise Residual Networks for Brain Segmentation from 3D MR Images10.1016/j.neuroimage.2017.04.041 ↗
64. A joint group sparsity-based deep learning for multi-contrast MRI reconstruction10.1016/j.jmr.2022.107354 ↗
65. Image Reconstruction Using UNET-Transformer Network for Fast and Low-Dose PET Scans10.1016/j.compmedimag.2023.102315 ↗
66. Nonlinear Total variation based noise removal algorithms10.1016/0167-2789(92)90242-F ↗
67. The alzheimer’s disease neuroimaging initiative: A review of papers published since its inception10.1016/j.jalz.2011.09.172 ↗
68. Improved optimization for the robust and accurate linear registration and motion correction of brain images10.1006/nimg.2002.1132 ↗
69. Quicksilver: Fast Predictive Image Registration Using Deep Learning10.1016/j.neuroimage.2017.07.008 ↗
70. Accurate and Robust Brain Image Alignment Using Boundary-Based Registration10.1016/j.neuroimage.2009.06.060 ↗
71. Surface-based mixed effects multilevel analysis of grouped human electrocorticography10.1016/j.neuroimage.2014.07.006 ↗
72. MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration10.1016/j.media.2012.05.008 ↗
73. Yang, J., Zhang, C., Wang, Z., Cao, X., Ouyang, X., Zhang, X., Zeng, Z., Zeng, Z., Lu, B., and Xia, Z. (2024). 3D Registration in 30 Years: A Survey. arXiv.
74. TransReg: Transformer-Based Registration Network for Medical Image Registration