Sound-to-Image Translation Through Direct Cross-Modal Connection Using a Convolutional–Attention Generative Model
Source abstract
Abstract
Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multisensory perception and integration · Generative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Signal Theory and Communications Department, Polytechnic University of Catalonia (UPC), C. Jordi Girona 1-3, 08034 Barcelona, Spain
Funding
Brazilian National Council for Scientific and Technological Development · Spanish State Research Agency
References (49)
Show source-provided references
1. Fanzeres, L.A., Vivacqua, A.S., and Biscainho, L.W.P. (2018). Mobile Sound Recognition for the Deaf and Hard of Hearing. arXiv.
2. Asakura, T. (2023). Augmented-Reality Presentation of Household Sounds for Deaf and Hard-of-Hearing People. Sensors, 23.10.2139/ssrn.4469562 ↗
3. Matthews, T., Fong, J., and Mankoff, J. (2005, January 9–12). Visualizing Non-Speech Sounds for the Deaf. Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility, Baltimore, MD, USA.10.1145/1090785.1090797 ↗
4. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems 27: Proceedings of the 28th Conference on Neural Information Processing Systems (NeurIPS 2014), Montreal, QC, Canada, 8–13 December 2014, Curran Associates, Inc.
5. Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv.
6. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Curran Associates, Inc.
7. Fanzeres, L.A., and Nadeu, C. (2023). Sound-to-Imagination: An Exploratory Study on Cross-Modal Translation Using Diverse Audiovisual Data. Appl. Sci., 13.10.3390/app131910833 ↗
8. Petermann, D., and Kalayeh, M.M. (2025). Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation. arXiv.
9. Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., and Yu, D. (2024). MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv.10.18653/v1/2024.findings-acl.738 ↗
10. Neubert, A., and Shreve, G.M. (1992). Translation as Text, Kent State University Press.
11. Representation Learning: A Review and New Perspectives10.1109/TPAMI.2013.50 ↗
12. Pons, J., and Serra, X. (2017, January 5–9). Designing Efficient Architectures for Modeling Temporal Features with Convolutional Neural Networks. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952601 ↗
13. Zeiler, M.D., and Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer. Proceedings, Part I.
14. Yosinski, J., Clune, J., Fuchs, T., and Lipson, H. (2015, January 6–11). Understanding Neural Networks Through Deep Visualization. Proceedings of the Deep Learning Workshop at the International Conference on Machine Learning (ICML), Lille, France.
15. Do Semantic Parts Emerge in Convolutional Neural Networks?10.1007/s11263-017-1048-0 ↗
16. Bag-of-Features Methods for Acoustic Event Detection and Classification10.1109/TASLP.2017.2690574 ↗
17. Kato, H., and Harada, T. (2014, January 23–28). Image Reconstruction from Bag-of-Visual-Words. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.127 ↗
18. Kavalerov, I., Wisdom, S., Erdogan, H., Patton, B., Wilson, K., Le Roux, J., and Hershey, J.R. (2019, January 20–23). Universal Sound Separation. Proceedings of the 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2019.8937253 ↗
19. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
20. Visual Instruction Tuning
21. CogVLM: Visual Expert for Pretrained Language Models
22. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. (2023, January 4–10). Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes, Greece.10.1109/ICASSP49357.2023.10095969 ↗
23. Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., and Lu, H. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. arXiv.10.1007/978-3-031-73411-3_5 ↗
24. Biner, B.C., Sofian, F.M., Karakaş, U.B., Ceylan, D., Erdem, E., and Erdem, A. (2024). SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models. arXiv.
25. van den Oord, A., Li, Y., and Vinyals, O. (2019). Representation Learning with Contrastive Predictive Coding. arXiv.
26. Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems 33: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada, 6–12 December 2020, Curran Associates, Inc.
27. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.10.1007/978-3-319-24574-4_28 ↗
28. Sung-Bin, K., Senocak, A., Ha, H., Owens, A., and Oh, T.-H. (2023, January 18–22). Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00622 ↗
29. Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. (2020, January 4–8). Vggsound: A Large-Scale Audio-Visual Dataset. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9053174 ↗
30. Zhou, Y., Wang, Z., Fang, C., Bui, T., and Berg, T.L. (2018, January 18–23). Visual to Sound: Generating Natural Sound for Videos in the Wild. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00374 ↗
31. Qin, C., Yu, N., Xing, C., Zhang, S., Chen, Z., Ermon, S., Fu, Y., Xiong, C., and Xu, R. (2023, January 2–6). GlueGen: Plug and Play Multi-Modal Encoders for X-to-Image Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.02110 ↗
32. Guzhov, A., Raue, F., Hees, J., and Dengel, A. (2022, January 23–27). Audioclip: Extending Clip to Image, Text and Audio. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022), Singapore.10.1109/ICASSP43922.2022.9747631 ↗
33. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Conference.
34. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., and Misra, I. (2023, January 18–22). ImageBind: One Embedding Space To Bind Them All. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01457 ↗
35. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv.
36. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Conference.
37. Any-to-Any Generation via Composable Diffusion
38. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 19–24). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
39. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
40. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
41. Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. (2024, January 21–27). NExT-GPT: Any-to-Any Multimodal LLM. Proceedings of the Forty-First International Conference on Machine Learning (ICML), Vienna, Austria.
42. Ku, L.-W., Martins, A., and Srikumar, V. (2024). GroundingGPT: Language Enhanced Multi-Modal Grounding Model. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, 11–16 August 2024, Association for Computational Linguistics.
43. Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. (2023). BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs. arXiv.
44. Wang, X., Zhuang, B., and Wu, Q. (2024, January 17–21). ModaVerse: Efficiently Transforming Modalities with LLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02512 ↗
45. Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., and Bansal, M. (2024, January 17–21). CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02589 ↗
46. Gal, Y., and Ghahramani, Z. (2016, January 19–24). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA.
47. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-To-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
48. Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., and Ritter, M. (2017, January 5–9). Audio Set: An Ontology and Human-Labeled Dataset for Audio Events. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952261 ↗
49. Understanding the Difficulty of Training Deep Feedforward Neural Networks