GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer
Source abstract
Abstract
Deepfakes have raised significant concerns due to their potential to spread false information and compromise the integrity of digital media. Current deepfake detection models often struggle to generalize across a diverse range of deepfake generation techniques and video content. In this work, we propose a Generative Convolutional Vision Transformer (GenConViT) for deepfake video detection. Our model combines ConvNeXt and Swin Transformer models for feature extraction, and it utilizes an Autoencoder and Variational Autoencoder to learn from latent data distributions. By learning from the visual artifacts and latent data distribution, GenConViT achieves an improved performance in detecting a wide range of deepfake videos. The model is trained and evaluated on DFDC, FF++, TM, DeepfakeTIMIT, and Celeb-DF (v2) datasets. The proposed GenConViT model demonstrates strong performance in deepfake video detection, achieving high accuracy across the tested datasets. While our model shows promising results in deepfake video detection by leveraging visual and latent features, we demonstrate that further work is needed to improve its generalizability when encountering out-of-distribution data. Our model provides an effective solution for identifying a wide range of fake videos while preserving the integrity of media.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
IDLab, Department of Electronics and Information Systems, Ghent University—imec, 9000 Gent, Belgium
Department of Computer Science, Addis Ababa University, Addis Ababa P.O. Box 1176, Ethiopia
Department of Network and Computer Security, State University of New York Polytechnic Institute, C135, Kunsela Hall, Utica, NY 13502, USA
Funding
Addis Ababa University Research Grant for the Adaptive Problem-Solving Research · Research Foundation—Flanders · Flemish government · IDLab (Ghent University—imec) · Flanders Innovation and Entrepreneurship (VLAIO) · European Union
References (88)
Show source-provided references
1. Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward10.1007/s10489-022-03766-z ↗
2. Rathgeb, C., Tolosana, R., Vera-Rodriguez, R., and Busch, C. (2022). An Introduction to Digital Face Manipulation. Handbook of Digital Face Manipulation and Detection: From DeepFakes to Morphing Attacks, Springer International Publishing.10.1007/978-3-030-87664-7 ↗
3. A literature review and perspectives in deepfakes: Generation, detection, and applications10.1007/s13735-022-00241-w ↗
4. Countering Malicious DeepFakes: Survey, Battleground, and Horizon10.1007/s11263-022-01606-8 ↗
5. Digital Manipulation and the Future of Electoral Democracy in the U.S10.1109/TTS.2020.2992666 ↗
6. Deep learning for deepfakes creation and detection: A survey10.1016/j.cviu.2022.103525 ↗
7. Rathgeb, C., Tolosana, R., Vera-Rodriguez, R., and Busch, C. (2022). The Threat of Deepfakes to Computer and Human Visions. Handbook of Digital Face Manipulation and Detection: From DeepFakes to Morphing Attacks, Springer International Publishing.10.1007/978-3-030-87664-7 ↗
8. Engaging with deepfakes: A meta-synthesis from the perspective of social shaping of technology theory10.1108/INTR-06-2022-0465 ↗
9. Emergence of deepfakes and video tampering detection approaches: A survey10.1007/s11042-022-13100-x ↗
10. Liu, P., Lin, Y., He, Y., Wei, Y., Zhen, L., Zhou, J.T., Goh, R.S.M., and Liu, J. (2021). Automated deepfake detection. arXiv.
11. Visual Deepfake Detection: Review of Techniques, Tools, Limitations, and Future Prospects10.1109/ACCESS.2024.3523288 ↗
12. Afchar, D., Nozick, V., Yamagishi, J., and Echizen, I. (2018, January 11–13). Mesonet: A compact facial video forgery detection network. Proceedings of the 2018 IEEE international workshop on information forensics and security (WIFS), Hong Kong.10.1109/WIFS.2018.8630761 ↗
13. Yang, X., Li, Y., and Lyu, S. (2019, January 12–17). Exposing Deep Fakes Using Inconsistent Head Poses. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683164 ↗
14. Trabelsi, A., Pic, M.M., and Dugelay, J.L. (September, January 29). Improving deepfake detection by mixing top solutions of the DFDC. Proceedings of the 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia.10.23919/EUSIPCO55093.2022.9909905 ↗
15. Matern, F., Riess, C., and Stamminger, M. (2019, January 7–11). Exploiting visual artifacts to expose deepfakes and face manipulations. Proceedings of the 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), Waikoloa Village, HI, USA.10.1109/WACVW.2019.00020 ↗
16. DeepFake detection algorithm based on improved vision transformer10.1007/s10489-022-03867-9 ↗
17. Gu, Z., Chen, Y., Yao, T., Ding, S., Li, J., Huang, F., and Ma, L. (2021, January 20–24). Spatiotemporal inconsistency learning for deepfake video detection. Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event.10.1145/3474085.3475508 ↗
18. Khan, S.A., and Dai, H. (2021, January 20–24). Video transformer for deepfake detection with incremental learning. Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event.10.1145/3474085.3475332 ↗
19. Generative adversarial networks10.1145/3422622 ↗
20. A Style-Based Generator Architecture for Generative Adversarial Networks10.1109/TPAMI.2020.2970919 ↗
21. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
22. Kingma, D.P., and Welling, M. (2022). Auto-Encoding Variational Bayes. arXiv.
23. An introduction to variational autoencoders10.1561/2200000056 ↗
24. Diffusion models beat gans on image synthesis
25. Ciftci, U.A., Demir, I., and Yin, L. (2020). Fakecatcher: Detection of synthetic portrait videos using biological signals. IEEE Trans. Pattern Anal. Mach. Intell., Early Access.
26. Yan, Z., Sun, P., Lang, Y., Du, S., Zhang, S., Wang, W., and Liu, L. (2022). Multimodal graph learning for deepfake detection. arXiv.
27. Wesselkamp, V., Rieck, K., Arp, D., and Quiring, E. (2022, January 22–26). Misleading deep-fake detection with gan fingerprints. Proceedings of the 2022 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.10.1109/SPW54247.2022.9833860 ↗
28. Zhang, D., Lin, F., Hua, Y., Wang, P., Zeng, D., and Ge, S. (2022, January 10–14). Deepfake video detection with spatiotemporal dropout transformer. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3547913 ↗
29. Kim, D.K., and Kim, K. (2022, January 3–8). Generalized Facial Manipulation Detection with Edge Region Feature Extraction. Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV51458.2022.00284 ↗
30. Yu, N., Skripniuk, V., Abdelnabi, S., and Fritz, M. (2021, January 11–17). Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01418 ↗
31. Guillaro, F., Cozzolino, D., Sud, A., Dufour, N., and Verdoliva, L. (2023, January 18–22). Trufor: Leveraging all-round clues for trustworthy image forgery detection and localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01974 ↗
32. Li, B., Sun, J., Poskitt, C.M., and Wang, X. (2024). How Generalizable are Deepfake Image Detectors? An Empirical Study. arXiv.
33. Rokach, L., Maimon, O., and Shmueli, E. (2023). Autoencoders. Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, Springer International Publishing.10.1007/978-3-031-24628-9 ↗
34. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
35. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
36. O’shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv.
37. Convolutional neural networks: A comprehensive survey10.37622/IJAER/14.3.2019.780-789 ↗
38. Generative Adversarial Networks for Face Generation: A Survey10.1145/3527850 ↗
39. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., and Nießner, M. (2016, January 27–30). Face2face: Real-time face capture and reenactment of rgb videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.262 ↗
40. Hsu, G.S., Tsai, C.H., and Wu, H.Y. (2022, January 18–24). Dual-generator face reenactment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00072 ↗
41. Sun, J., Deng, Q., Li, Q., Sun, M., Ren, M., and Sun, Z. (2022, January 18–24). AnyFace: Free-Style Text-To-Face Synthesis and Manipulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01813 ↗
42. Korshunova, I., Shi, W., Dambre, J., and Theis, L. (2017, January 22–29). Fast face-swap using convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.397 ↗
43. Li, L., Bao, J., Yang, H., Chen, D., and Wen, F. (2020). FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping. arXiv.
44. Zhu, Y., Li, Q., Wang, J., Xu, C.Z., and Sun, Z. (2021, January 19–25). One Shot Face Swapping on Megapixels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00480 ↗
45. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2018). Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv.
46. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein Generative Adversarial Networks. Proceedings of the the 34th International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
47. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-To-Image Translation With Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
48. Image-to-Image Translation: Methods and Applications10.1109/TMM.2021.3109419 ↗
49. Rathgeb, C., Tolosana, R., Vera-Rodriguez, R., and Busch, C. (2022). Detection of AI-Generated Synthetic Faces. Handbook of Digital Face Manipulation and Detection: From DeepFakes to Morphing Attacks, Springer International Publishing.10.1007/978-3-030-87664-7 ↗
50. Mareen, H., D’haeseleer, S., Van Damme, K., Evens, T., Lambert, P., and Van Wallendael, G. (2024, January 5–7). COM-PRESS: Dashboard to Detect (AI-Based) Image Manipulations. Proceedings of the 2024 IEEE Gaming, Entertainment, and Media Conference (GEM), Turin, Italy.10.1109/GEM61861.2024.10585560 ↗
51. Nguyen, H., Tieu, T.N.D.T., Nguyen-Son, H.Q., Nozick, V., Yamagishi, J., and Echizen, I. (2018, January 27–30). Modular Convolutional Neural Network for Discriminating between Computer-Generated Images and Photographic Images. Proceedings of the ARES 2018 Proceedings of the 13th International Conference on Availability, Reliability and Security, Hambourg, Germany.10.1145/3230833.3230863 ↗
52. Nguyen, H.H., Yamagishi, J., and Echizen, I. (2019, January 12–17). Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682602 ↗
53. Li, Y., and Lyu, S. (2019, January 16–20). Exposing DeepFake Videos By Detecting Face Warping Artifacts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA.
54. Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., and Guo, B. (2020, January 14–19). Face X-Ray for More General Face Forgery Detection. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00505 ↗
55. Sun, Y., Zhang, Z., Echizen, I., Nguyen, H.H., Qiu, C., and Sun, L. (2023, January 3–7). Face Forgery Detection Based on Facial Region Displacement Trajectory Series. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, Waikoloa, HI, USA.10.1109/WACVW58289.2023.00070 ↗
56. Li, Y., Chang, M.C., and Lyu, S. (2018, January 11–13). In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking. Proceedings of the 2018 IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong.10.1109/WIFS.2018.8630787 ↗
57. Chintha, A., Rao, A., Sohrawardi, S., Bhatt, K., Wright, M., and Ptucha, R. (October, January 28). Leveraging edges and optical flow on faces for deepfake detection. Proceedings of the 2020 IEEE International Joint Conference on Biometrics (IJCB), Houston, TX, USA.10.1109/IJCB48548.2020.9304936 ↗
58. Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., and Xia, W. (2021, January 11–17). Learning Self-Consistency for Deepfake Detection. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01475 ↗
59. Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., and Natarajan, P. (2019, January 15–20). Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.
60. Deressa, D.W., Lambert, P., Van Wallendael, G., Atnafu, S., and Mareen, H. (2024, January 5–7). Improved Deepfake Video Detection Using Convolutional Vision Transformer. Proceedings of the 2024 IEEE Gaming, Entertainment, and Media Conference (GEM), Turin, Italy.10.1109/GEM61861.2024.10585593 ↗
61. Heo, Y.J., Choi, Y.J., Lee, Y.W., and Kim, B.G. (2021). Deepfake Detection Scheme Based on Vision Transformer and Distillation. arXiv.
62. Coccomini, D.A., Messina, N., Gennaro, C., and Falchi, F. (2022). Combining EfficientNet andVision Transformers for Video Deepfake Detection. Image Analysis and Processing—ICIAP 2022: 21st International Conference, Lecce, Italy, 23–27 May 2022, Proceedings, Part III, Springer.10.1007/978-3-031-06433-3_19 ↗
63. Chen, T., Yang, S., Hu, S., Fang, Z., Fu, Y., Wu, X., and Wang, X. (July, January 30). Masked Conditional Diffusion Model for Enhancing Deepfake Detection. Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN), Yokohama, Japan.10.1109/IJCNN60899.2024.10651330 ↗
64. Bhattacharyya, C., Wang, H., Zhang, F., Kim, S., and Zhu, X. (2024). Diffusion Deepfake. arXiv.
65. DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion
66. Xu, Y., Liang, J., Jia, G., Yang, Z., Zhang, Y., and He, R. (2023, January 1–6). TALL: Thumbnail Layout for Deepfake Video Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.02071 ↗
67. Le, B.M., and Woo, S.S. (2023, January 1–6). Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.02045 ↗
68. FreqBlender: Enhancing DeepFake Detection by Blending Frequency Knowledge
69. Seferbekov, S. (2025, June 09). A Prize Winning Solution for DFDC Challenge. Github. Available online: https://github.com/selimsef/dfdc_deepfake_challenge.
70. Khan, S.A., and Dang-Nguyen, D.T. (2022). Hybrid Transformer Network for Deepfake Detection. arXiv.10.1145/3549555.3549588 ↗
71. Wightman, R. (2025, June 09). PyTorch Image Models, G. Github. Available online: https://github.com/huggingface/pytorch-image-models.
72. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A.A. (2020). Albumentations: Fast and Flexible Image Augmentations. Information, 11.10.3390/info11020125 ↗
73. Dolhansky, B., Howes, R., Pflaum, B., Baram, N., and Ferrer, C.C. (2019). The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv.
74. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C.C. (2020). The DeepFake Detection Challenge (DFDC) Dataset. arXiv.
75. Chen, W., Chua, S.L.B., Winkler, S., and Ng, S.K. (2022, January 17–21). Trusted Media Challenge Dataset and User Study. Proceedings of the 31st ACM International Conference on Information and Knowledge Management, Atlanta, GA, USA.10.1145/3511808.3557715 ↗
76. Korshunov, P., and Marcel, S. (2018). DeepFakes: A New Threat to Face Recognition? Assessment and Detection. arXiv.
77. Sanderson, C., and Lovell, B.C. (2009). Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference. Advances in Biometrics, Springer.10.1007/978-3-642-01793-3_21 ↗
78. Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, S. (2020, January 14–19). Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00327 ↗
79. Rathgeb, C., Tolosana, R., Vera-Rodriguez, R., and Busch, C. (2022). Toward the Creation and Obstruction of DeepFakes. Handbook of Digital Face Manipulation and Detection: From DeepFakes to Morphing Attacks, Springer International Publishing.10.1007/978-3-030-87664-7 ↗
80. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Nießner, M. (2020, January 27). FaceForensics++: Learning to Detect Manipulated Facial Images. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00009 ↗
81. Deferred neural rendering: Image synthesis using neural textures10.1145/3306346.3323035 ↗
82. Geitgey, A. (2025, June 09). Face_Recognition, 2017. GitHub Repository. Available online: https://github.com/ageitgey/face_recognition.
83. Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., and Grundmann, M. (2019). BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs. arXiv.
84. Thing, V.L.L. (2023). Deepfake Detection with Deep Learning: Convolutional Neural Networks versus Transformers. arXiv.10.1109/CSR57506.2023.10225004 ↗
85. Ojha, U., Li, Y., and Lee, Y.J. (2023, January 18–22). Towards universal fake image detectors that generalize across generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02345 ↗
86. Maiano, L., Casadei, F., and Amerini, I. (2025). Enhancing Abnormality Identification: Robust Out-of-Distribution Strategies for Deepfake Detection. arXiv.
87. Tian, J., Yu, C., Wang, X., Chen, P., Xiao, Z., Dai, J., Han, J., and Chai, Y. (October, January 29). Real appearance modeling for more general deepfake detection. Proceedings of the European Conference on Computer Vision, Milan, Italy.
88. Kim, A. (2025, March 03). Open-Source Deepfake Detection Models from TrueMedia.org. Available online: https://automata88.medium.com/open-source-deepfake-detection-models-from-truemedia-org-29f9ce59882d.