EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance
Source abstract
Abstract
With the flourishing development of generative models, image manipulation is receiving increasing attention. Rather than text modality, several elegant designs have delved into leveraging audio to manipulate images. However, existing methodologies mainly focus on image generation conditional on semantic alignment, ignoring the vivid affective information depicted in the audio. We propose an Emotion-aware StyleGAN Manipulator (EmoStyle), a framework where affective information from audio can be explicitly extracted and further utilized during image manipulation. Specifically, we first leverage the multi-modality model ImageBind for initial cross-modal retrieval between images and music, and select the music-related image for further manipulation. Simultaneously, by extracting sentiment polarity from the lyrics of the audio, we generate an emotionally rich auxiliary music branch to accentuate the affective information. We then leverage pre-trained encoders to encode audio and the audio-related image into the same embedding space. With the aligned embeddings, we manipulate the image via a direct latent optimization method. We conduct objective and subjective evaluations on the generated images, and our results show that our framework is capable of generating images with specified human emotions conveyed in the audio.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Software Engineering Institute, East China Normal University, Shanghai 200062, China
School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
School of Computer Science, Fudan University, Shanghai 200433, China
References (48)
Show source-provided references
1. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
2. Karras, T., Laine, S., and Aila, T. (2019, January 15–20). A style-based generator architecture for generative adversarial networks. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
3. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 18–24). High-resolution image synthesis with latent diffusion models. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
4. Johnson, J., Gupta, A., and Li, F.-F. (2018, January 18–23). Image generation from scene graphs. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00133 ↗
5. Chen, L., Srivastava, S., Duan, Z., and Xu, C. (2017, January 23–27). Deep cross-modal audio-visual generation. Proceedings of the ACM International Conference on Multimedia (ACM MM), Silicon Valley, CA, USA.10.1145/3126686.3126723 ↗
6. Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. (2021, January 11–17). Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the International Conference on Computer Vision (ICCV), Montreal, BC, Canada.10.1109/ICCV48922.2021.00209 ↗
7. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
8. Lee, S.H., Roh, W., Byeon, W., Yoon, S.H., Kim, C., Kim, J., and Kim, S. (2022, January 18–24). Sound-guided semantic image manipulation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00337 ↗
9. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., and Misra, I. (2023, January 17–24). Imagebind: One embedding space to bind them all. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01457 ↗
10. Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., and Yang, H. (2021, January 6–14). Cogview: Mastering text-to-image generation via transformers. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Virtual.
11. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., and Gafni, O. (2022). Make-a-video: Text-to-video generation without text-video data. arXiv.
12. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., and Reynolds, M. (December, January 28). Flamingo: A visual language model for few-shot learning. Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LA, USA.
13. Mokady, R., Hertz, A., and Bermano, A.H. (2021). Clipcap: Clip prefix for image captioning. arXiv.
14. Sun, J., Li, Q., Wang, W., Zhao, J., and Sun, Z. (2021, January 20–24). Multi-caption text-to-face synthesis: Dataset and algorithm. Proceedings of the ACM International Conference on Multimedia (ACM MM), Chengdu, China.10.1145/3474085.3475391 ↗
15. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–22). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
16. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D.N. (2017, January 22–29). Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.629 ↗
17. Stackgan++: Realistic image synthesis with stacked generative adversarial networks10.1109/TPAMI.2018.2856256 ↗
18. Cheng, J., Wu, F., Tian, Y., Wang, L., and Tao, D. (2020, January 13–19). RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01092 ↗
19. Dong, H., Yu, S., Wu, C., and Guo, Y. (2017, January 22–29). Semantic image synthesis via adversarial learning. Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.608 ↗
20. Kim, G., Kwon, T., and Ye, J.C. (2022, January 18–24). Diffusionclip: Text-guided diffusion models for robust image manipulation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00246 ↗
21. Li, B., Qi, X., Lukasiewicz, T., and Torr, P.H. (2020, January 13–19). Manigan: Text-guided image manipulation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00790 ↗
22. Liu, Y., De Nadai, M., Cai, D., Li, H., Alameda-Pineda, X., Sebe, N., and Lepri, B. (2020, January 16). Describe what to change: A text-guided unsupervised image-to-image translation approach. Proceedings of the ACM International Conference on Multimedia (ACM MM), Seattle, WA, USA.10.1145/3394171.3413505 ↗
23. Nam, S., Kim, Y., and Kim, S.J. (2018, January 3–8). Text-adaptive generative adversarial networks: Manipulating images with natural language. Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, QC, Canada.
24. Xia, W., Yang, Y., Xue, J.H., and Wu, B. (2021, January 20–25). Tedigan: Text-guided diverse face image generation and manipulation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00229 ↗
25. Xu, Z., Lin, T., Tang, H., Li, F., He, D., Sebe, N., Timofte, R., Van Gool, L., and Ding, E. (2022, January 18–24). Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01769 ↗
26. Aytar, Y., Vondrick, C., and Torralba, A. (2016, January 5–10). Soundnet: Learning sound representations from unlabeled video. Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.10.1109/CVPR.2016.18 ↗
27. Korbar, B., Tran, D., and Torresani, L. (2018, January 3–8). Cooperative learning of audio and video models from self-supervised synchronization. Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, QC, Canada.
28. Owens, A., Wu, J., McDermott, J.H., Freeman, W.T., and Torralba, A. (2016, January 11–14). Ambient sound provides supervision for visual learning. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_48 ↗
29. Gan, C., Huang, D., Zhao, H., Tenenbaum, J.B., and Torralba, A. (2020, January 13–19). Music gesture for visual sound separation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01049 ↗
30. Su, K., Liu, X., and Shlizerman, E. (2020, January 6–12). Audeo: Audio generation for a silent performance video. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada.
31. Morgado, P., Li, Y., and Nvasconcelos, N. (2020, January 6–12). Learning representations from audio-visual spatial alignment. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada.
32. Mo, S., and Morgado, P. (2023). A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition. arXiv.
33. Gao, R., Feris, R., and Grauman, K. (2018, January 8–14). Learning to separate object sounds by watching unlabeled video. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_3 ↗
34. Tian, Y., Hu, D., and Xu, C. (2021, January 20–25). Cyclic co-learning of sounding object visual grounding and sound separation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00277 ↗
35. Gao, R., and Grauman, K. (2021, January 20–25). Visualvoice: Audio-visual speech separation with cross-modal consistency. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01524 ↗
36. Tian, Y., Shi, J., Li, B., Duan, Z., and Xu, C. (2018, January 8–14). Audio-visual event localization in unconstrained videos. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_16 ↗
37. Tian, Y., Li, D., and Xu, C. (2020, January 23–28). Unified multisensory perception: Weakly-supervised audio-visual video parsing. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58580-8_26 ↗
38. Lin, Y.B., Tseng, H.Y., Lee, H.Y., Lin, Y.Y., and Yang, M.H. (2021, January 6–14). Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Virtual.
39. Mo, S., and Tian, Y. (December, January 28). Multi-modal grouping network for weakly-supervised audio-visual video parsing. Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LA, USA.
40. Morgado, P., Nvasconcelos, N., Langlois, T., and Wang, O. (2018, January 3–8). Self-supervised generation of spatial audio for 360 video. Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, QC, Canada.
41. Gao, R., and Grauman, K. (2019, January 15–20). 2. 5D visual sound. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00041 ↗
42. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020, January 13–19). Analyzing and improving the image quality of stylegan. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00813 ↗
43. Gong, Y., Khurana, S., Karlinsky, L., and Glass, J. (2023). Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers. arXiv.10.21437/Interspeech.2023-2193 ↗
44. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., and Xia, X. (2023, January 1–5). GLM-130B: An Open Bilingual Pre-trained Model. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
45. Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019, January 15–20). Arcface: Additive angular margin loss for deep face recognition. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00482 ↗
46. Lee, S.H., Oh, G., Byeon, W., Yoon, S.H., Kim, J., and Kim, S. (2022). Robust sound-guided image manipulation. arXiv.10.2139/ssrn.4437061 ↗
47. Skorokhodov, I., Sotnikov, G., and Elhoseiny, M. (2021, January 11–17). Aligning latent and image spaces to connect the unconnectable. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01388 ↗
48. Ke, J., Ye, K., Yu, J., Wu, Y., Milanfar, P., and Yang, F. (2023, January 17–24). VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00968 ↗