Learnable Diffusion Framework for Mouse V1 Neural Decoding
Source abstract
Abstract
ABSTRACT Decoding visual stimuli from neural signals is an essential step toward understanding how sensory information is represented in the brain. While most existing approaches reconstruct visual stimuli from human functional magnetic resonance imaging (fMRI), utilizing calcium imaging in mice opens the door to single‐neuron‐level insights into non‐primate visual systems with distinct spectral sensitivities. Here, we present Sensorium‐Viz, a diffusion‐based framework specifically designed for decoding activity in the mouse primary visual cortex. The model is among the first to reliably reconstruct complex, high‐resolution images from previously unseen single‐neuron responses. At its core, Sensorium‐Viz introduces two key advances for neuron‐to‐image decoding: a synthetic‐response augmentation strategy that improves reconstruction performance by more than 30% while enabling cross‐mouse generalization through fine‐tuning, and an architectural design that integrates a Diffusion Transformer (DiT) with a spatial neuron‐embedding module, thereby achieving up to a 10.65% performance gain over leading fMRI‐based reconstruction methods across pixel‐ and content‐level benchmarks. Analysis of the neural responses and corresponding reconstructions reveals that neurons sensitive to low‐level visual features form the primary basis of V1's representation of external stimuli. These findings establish Sensorium‐Viz as a biologically grounded and technically robust tool for vision decoding.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face Recognition and Perception · Neural dynamics and brain function · Visual perception and processing mechanisms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computational Medicine and Bioinformatics University of Michigan Ann Arbor Michigan USA
Department of Electrical Engineering and Computer Science University of Michigan Ann Arbor Michigan USA
Department of Internal Medicine University of Michigan Ann Arbor Michigan USA
Funding
National Institute of General Medical Sciences
References (30)
Show source-provided references
1. 10.1167/19.10.81c10.1167/19.10.81c ↗
2. 10.1016/j.neuroimage.2006.06.06210.1016/j.neuroimage.2006.06.062 ↗
3. 10.1016/j.neuron.2008.11.00410.1016/j.neuron.2008.11.004 ↗
4. 10.1162/NECO_a_0042310.1162/NECO_a_00423 ↗
5. 10.1038/ncomms1503710.1038/ncomms15037 ↗
6. 10.1016/j.tics.2018.04.00110.1016/j.tics.2018.04.001 ↗
7. F.Ozcelik B.Choksi M.Mozafari L.Reddy andR.VanRullen “Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration Using Instance‐Conditioned GANs ”International Joint Conference on Neural Networks (IJCNN) (2022):1–8 https://doi.org/10.48550/ARXIV.2202.12692.10.1109/IJCNN55064.2022.9892673 ↗
8. Reconstructing Perceptive Images From Brain Activity By Shape‐Semantic GAN
9. 10.48550/ARXIV.2210.0176910.48550/ARXIV.2210.01769 ↗
10. Z.Chen J.Qing T.Xiang W. L.Yue andJ. H.Zhou “Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding ” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition(2022):22710–22720.10.1109/CVPR52729.2023.02175 ↗
11. Y.TakagiandS.Nishimoto “High‐resolution Image Reconstruction With Latent Diffusion Models From Human Brain Activity ” in2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(IEEE 2023) 14453–14463 https://doi.org/10.1109/cvpr52729.2023.01389.10.1109/CVPR52729.2023.01389 ↗
12. 10.1038/nrn241110.1038/nrn2411 ↗
13. 10.1523/JNEUROSCI.0712-14.201410.1523/JNEUROSCI.0712-14.2014 ↗
14. 10.1038/s41586-022-05270-310.1038/s41586-022-05270-3 ↗
15. 10.1038/s41467-020-14645-x10.1038/s41467-020-14645-x ↗
16. 10.1007/s11042-016-3414-210.1007/s11042-016-3414-2 ↗
17. S.Garasto A. A.Bharath andS. R.Schultz “Visual Reconstruction From 2‐photon Calcium Imaging Suggests Linear Readout Properties of Neurons in Mouse Primary Visual Cortex ”preprint bioRxiv April 12 (2018):300392 https://doi.org/10.1101/300392.10.1101/300392 ↗
18. 10.3389/fnins.2023.113060610.3389/fnins.2023.1130606 ↗
19. K. F.Willeke P. G.Fahey M.Bashiri et al. “The Sensorium Competition on Predicting Large‐Scale Mouse Primary Visual Cortex Activity ”preprint arXiv [q‐bio.NC] June 17 2022.
20. 10.1016/j.visres.2005.09.00610.1016/j.visres.2005.09.006 ↗
21. W.PeeblesandS.Xie “Scalable Diffusion Models With Transformers ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2023):4195–4205.10.1109/ICCV51070.2023.00387 ↗
22. B. M.Li I. M.Cornacchia N. L.Rochefort andA.Onken “V1T: Large‐Scale Mouse V1 Response Prediction Using a Vision Transformer ”preprint arXiv [cs.CV]February 6 2023.
23. 10.3389/fncom.2019.0002110.3389/fncom.2019.00021 ↗
24. R.Zhang P.Isola A. A.Efros E.Shechtman andO.Wang “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric ” in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(IEEE 2018) 586–595 https://doi.org/10.1109/cvpr.2018.00068.10.1109/CVPR.2018.00068 ↗
25. J.Betker G.Goh L.Jing A.Ramesh et al. “Improving Image Generation With Better Captions ” Computer Science https://cdn.openai.com/papers/dall‐e‐3. pdf 2.3 (2023):8.
26. Predicting Single Neuron Responses of the Primary Visual Cortex With Deep Learning Model
27. J.Sobotka L.Baroni andJ. M. E. I.Antolík “Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs ”preprint arXiv [cs.LG] October 23 2025 https://doi.org/10.48550/arXiv.2510.20762.
28. 10.1038/s41586-023-06031-610.1038/s41586-023-06031-6 ↗
29. 10.7717/peerj.45310.7717/peerj.453 ↗
30. 10.21105/joss.0410110.21105/joss.04101 ↗