Generating Synthetic Facial Expression Images Using EmoStyle
Source abstract
Abstract
Synthetic data has emerged as a significant alternative to more costly and time-consuming data collection methods. This assertion is particularly salient in the context of training facial expression recognition (FER) and generation models. The EmoStyle model represents a state-of-the-art method for editing images of facial expressions in the latent space of StyleGAN2, using a continuous valence–arousal (VA) representation of emotions. While the model has demonstrated promising results in terms of high-quality image generation and strong identity preservation, its accuracy in reproducing facial expressions across the VA space remains to be systematically examined. To address this gap, the present study proposes a systematic evaluation of EmoStyle’s ability to generate facial expressions across the full VA space, including four levels of emotional intensity. While prior work on expression manipulation has mainly focused its evaluations on perceptual quality, diversity, identity preservation, or classification accuracy, to the best of our knowledge, no study to date has systematically evaluated the accuracy of generated expressions across the VA space. The evaluation’s findings include a consistent weakness in the VA direction range of 242–329°, where EmoStyle demonstrates the inability to produce distinct expressions. Building on these findings, we outline recommendations for enhancing the generation pipeline and release an open-source EmoStyle-based toolkit that integrates fixes to the original EmoStyle repository, an API wrapper, and our experiment scripts. Collectively, these contributions furnish both novel insights into the model’s capacities and practical resources for further research.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Emotion and Mood Recognition · Face and Expression Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Systems Informatics, Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe 657-8501, Japan
Grenoble Institute of Technology–Ensimag, Grenoble Alpes University, 681 Rue de la Passerelle–BP 72, 38402 Saint Martin d’Hères CEDEX, France
Funding
JSPS KAKENHI
References (47)
Show source-provided references
1. Deep Facial Expression Recognition: A Survey10.1109/TAFFC.2020.2981446 ↗
2. Villalobos, P., Sevilla, J., Besiroglu, T., Heim, L., Ho, A., and Hobbhahn, M. (2022). Machine Learning Model Sizes and the Parameter Gap. arXiv.
3. Dynamic Programming10.1126/science.153.3731.34 ↗
4. Crespo Márquez, A. (2022). The curse of dimensionality. Digital Maintenance Management: Guiding Digital Transformation in Maintenance, Springer.10.1007/978-3-030-97660-6 ↗
5. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild10.1109/TAFFC.2017.2740923 ↗
6. Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.10.1371/journal.pone.0196391 ↗
7. Kollias, D., and Zafeiriou, S. (2019). Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition. arXiv.
8. Fabian Benitez-Quiroz, C., Srinivasan, R., and Martinez, A.M. (2016, January 27–30). EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.600 ↗
9. A Novel Approach to Cross dataset studies in Facial Expression Recognition10.1007/s11042-022-13117-2 ↗
10. Boutros, F., Huber, M., Siebke, P., Rieber, T., and Damer, N. (2022). SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data. arXiv.10.1109/IJCB54206.2022.10007961 ↗
11. Harvey, A. (2025, September 29). Exposing.ai. Available online: https://exposing.ai.
12. Kemelmacher-Shlizerman, I., Seitz, S., Miller, D., and Brossard, E. (2015). The MegaFace Benchmark: 1 Million Faces for Recognition at Scale. arXiv.10.1109/CVPR.2016.527 ↗
13. Guo, Y., Zhang, L., Hu, Y., He, X., and Gao, J. (2016). MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition. arXiv.10.1007/978-3-319-46487-9_6 ↗
14. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., and Zisserman, A. (2018). VGGFace2: A dataset for recognising faces across pose and age. arXiv.10.1109/FG.2018.00020 ↗
15. Peng, K., Mathur, A., and Narayanan, A. (2021). Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers. arXiv.
16. Nikolenko, S.I. (2019). Synthetic Data for Deep Learning. arXiv.
17. Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., and Zhou, D. (2024). Best Practices and Lessons Learned on Synthetic Data. arXiv.
18. Wood, E., Baltrušaitis, T., Hewitt, C., Dziadzio, S., Johnson, M., Estellers, V., Cashman, T.J., and Shotton, J. (2021). Fake It Till You Make It: Face analysis in the wild using synthetic data alone. arXiv.10.1109/ICCV48922.2021.00366 ↗
19. Jaipuria, N., Zhang, X., Bhasin, R., Arafa, M., Chakravarty, P., Shrivastava, S., Manglani, S., and Murali, V.N. (2020, January 14–19). Deflating Dataset Bias Using Synthetic Data Augmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00394 ↗
20. Xu, T., White, J., Kalkan, S., and Gunes, H. (2020). Investigating Bias and Fairness in Facial Expression Recognition. arXiv.10.1007/978-3-030-65414-6_35 ↗
21. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020). Analyzing and Improving the Image Quality of StyleGAN. arXiv.10.1109/CVPR42600.2020.00813 ↗
22. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Networks. arXiv.
23. Face Generation and Editing with StyleGAN: A Survey10.1109/TPAMI.2024.3350004 ↗
24. StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows10.1145/3447648 ↗
25. Azari, B., and Lim, A. (2024, January 3–8). EmoStyle: One-Shot Facial Expression Editing Using Continuous Emotion Parameters. Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV57701.2024.00626 ↗
26. A Circumplex Model of Affect10.1037/h0077714 ↗
27. Pumarola, A., Agudo, A., Martinez, A.M., Sanfeliu, A., and Moreno-Noguer, F. (2018, January 8–14). Ganimation: Anatomically-aware facial animation from a single image. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_50 ↗
28. A chinese face dataset with dynamic expressions and diverse ages synthesized by deep learning10.1038/s41597-023-02701-2 ↗
29. Yang, C., and Lim, S.N. (2019). Unconstrained Facial Expression Transfer using Style-based Generator. arXiv.
30. Hu, X., Aldausari, N., and Mohammadi, G. (2022). 2CET-GAN: Pixel-Level GAN Model for Human Facial Expression Transfer. arXiv.10.1145/3607541.3616822 ↗
31. Kollias, D., and Zafeiriou, S. (2021). Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework. arXiv.
32. Ding, H., Sricharan, K., and Chellappa, R. (2017). ExprGAN: Facial Expression Editing with Controllable Expression Intensity. arXiv.10.1609/aaai.v32i1.12277 ↗
33. Ekman, P., and Friesen, W.V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press.10.1037/t27734-000 ↗
34. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., and Choo, J. (2018). StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. arXiv.10.1109/CVPR.2018.00916 ↗
35. Kim, D., and Song, B.C. (2022). Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition. arXiv.
36. Abbasnejad, I., Sridharan, S., Nguyen, D., Denman, S., Fookes, C., and Lucey, S. (2017, January 22–29). Using Synthetic Data to Improve Facial Expression Analysis with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Venice, Italy.10.1109/ICCVW.2017.189 ↗
37. He, X., Luo, C., Xian, X., Li, B., Song, S., Khan, M.H., Xie, W., Shen, L., and Ge, Z. (2024). SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data. arXiv.
38. Karras, T., Laine, S., and Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. arXiv.10.1109/CVPR.2019.00453 ↗
39. Face Image Quality Assessment: A Literature Survey10.1145/3507901 ↗
40. Meng, Q., Zhao, S., Huang, Z., and Zhou, F. (2021). MagFace: A Universal Representation for Face Recognition and Quality Assessment. arXiv.10.1109/CVPR46437.2021.01400 ↗
41. Paplu, S.H., Mishra, C., and Berns, K. (2022). Real-time Emotion Appraisal with Circumplex Model for Human-Robot Interaction. arXiv.
42. Estimation of continuous valence and arousal levels from faces in naturalistic conditions10.1038/s42256-020-00280-0 ↗
43. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network10.1109/TAFFC.2022.3188390 ↗
44. Facial Expression Recognition with Adaptive Frame Rate based on Multiple Testing Correction
45. Gu, S., Bao, J., Chen, D., and Wen, F. (2020). GIQA: Generated Image Quality Assessment. arXiv.10.1007/978-3-030-58621-8_22 ↗
46. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 9–15). Parameter-efficient transfer learning for NLP. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
47. Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-Free Generative Adversarial Networks. arXiv.