Spatially-Precise Video Editing with Reference Imitation: A Region-Aware Cross-Modal Framework
Source abstract
Abstract
Video content is widely applied in the multimedia field. Meanwhile, artificial intelligence generative content (AIGC) is continuously developing. As a result, video editing techniques in the field of computer vision are constantly evolving. However, at present, there are still numerous challenges in achieving fine-grained local editing of videos and maintaining temporal consistency.In this paper, we propose a novel framework for local video editing that integrates image-based local editing techniques with CoDeF (Content Deformation Field). Unlike existing methods, our approach explicitly models temporal deformation, ensuring spatial precision in local edits while maintaining high temporal consistency. Our framework first utilizes CoDeF to generate canonical images that aggregate static video content while preserving semantic integrity. A dual diffusion-based local editing module is then applied to modify these canonical images with user-specified editing regions and reference images. Finally, the learned temporal deformation field propagates the edits back to the entire video, achieving coherent and seamless video editing. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches in both editing quality and user satisfaction, offering a significant improvement in local precision and temporal stability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Analysis and Summarization · Advanced Vision and Imaging · Computer Graphics and Visualization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China
Guangxi Academy of Sciences, Nanning 530007, China
References (46)
Show source-provided references
1. Dancyger, K. (2014). The Technique of Film and Video Editing: History, Theory, and Practice, Routledge.10.4324/9780240813967 ↗
2. Wang, Z. (2021, January 25–27). Analysis on the Application of Video Editing Skills Based on Image Mosaic in Film and Television Works. Proceedings of the 2021 2nd International Conference on Computers, Information Processing and Advanced Education, CIPAE 2021, New York, NY, USA.10.1145/3456887.3459697 ↗
3. Importance of multi-camera film/video editing techniques in film/tv programs making
4. Video Editing: Technical Vs.Creative Tool10.1149/10701.10045ecst ↗
5. Meng, J., and Chang, S.F. (1996, January 18–22). CVEPS-A compressed video editing and parsing system. Proceedings of the Fourth ACM International Conference on Multimedia, MULTIMEDIA ’96, Boston, MA, USA.10.1145/244130.244145 ↗
6. Interactive Video Manipulation Using Object Trajectories and Scene Backgrounds10.1109/TCSVT.2013.2248972 ↗
7. Siddique, A., and Lee, S. (2021). Object-Wise Video Editing. Appl. Sci., 11.10.3390/app11020671 ↗
8. Artistic Style Transfer for Videos and Spherical Images10.1007/s11263-018-1089-z ↗
9. Video style transfer based on convolutional neural networks10.1155/2022/8918722 ↗
10. Peebles, W.S., Zhu, J.Y., Zhang, R., Torralba, A., Efros, A.A., and Shechtman, E. (2022, January 18–24). GAN-Supervised Dense Visual Alignment. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01311 ↗
11. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, NIPS’14, Cambridge, MA, USA.
12. DeepFaceVideoEditing: Sketch-based deep editing of face videos
13. Tzaban, R., Mokady, R., Gal, R., Bermano, A., and Cohen-Or, D. (2022, January 6–9). Stitch it in Time: GAN-Based Facial Editing of Real Videos. Proceedings of the SIGGRAPH Asia 2022 Conference Papers, SA ’22, New York, NY, USA.10.1145/3550469.3555382 ↗
14. A Style-Based Generator Architecture for Generative Adversarial Networks10.1109/TPAMI.2020.2970919 ↗
15. Layered neural atlases for consistent video editing10.1145/3478513.3480546 ↗
16. Huang, J., Sigal, L., Yi, K.M., Wang, O., and Lee, J.Y. (2023). INVE: Interactive Neural Video Editing. arXiv.
17. Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. (2023, January 18–22). Imagic: Text-Based Real Image Editing with Diffusion Models. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.10.1109/CVPR52729.2023.00582 ↗
18. Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., and Wen, F. (2023, January 17–24). Paint by Example: Exemplar-based Image Editing with Diffusion Models. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01763 ↗
19. Shi, Y., Xue, C., Liew, J.H., Pan, J., Yan, H., Zhang, W., Tan, V.Y.F., and Bai, S. (2024, January 16–22). DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.10.1109/CVPR52733.2024.00844 ↗
20. Feng, R., Weng, W., Wang, Y., Yuan, Y., Bao, J., Luo, C., Chen, Z., and Guo, B. (2024, January 16–22). CCEdit: Creative and Controllable Video Editing via Diffusion Models. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00641 ↗
21. Ouyang, W., Dong, Y., Yang, L., Si, J., and Pan, X. (2024). I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models. arXiv.10.1145/3680528.3687656 ↗
22. Chen, X., Feng, Y., Chen, M., Wang, Y., Zhang, S., Liu, Y., Shen, Y., and Zhao, H. (2024). Zero-shot Image Editing with Reference Imitation. arXiv.
23. Ouyang, H., Wang, Q., Xiao, Y., Bai, Q., Zhang, J., Zheng, K., Zhou, X., Chen, Q., and Chen, Q. (2024, January 16–22). CoDeF: Content Deformation Fields for Temporally Consistent Video Processing. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00773 ↗
24. Zhang, Z., Dai, Z., Qin, L., and Wang, W. (2024). EffiVED:Efficient Video Editing via Text-instruction Diffusion Models. arXiv.
25. Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., and Ritter, H. (2024). Video Diffusion Models: A Survey. arXiv.
26. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
27. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
28. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
29. Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Munich, Germany.10.1007/978-3-319-24571-3 ↗
30. Pan, X., Tewari, A., Leimkühler, T., Liu, L., Meka, A., and Theobalt, C. (2023, January 6–10). Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. Proceedings of the ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ’23, New York, NY, USA.10.1145/3588432.3591500 ↗
31. A Survey on Generative Adversarial Networks: Variants, Applications, and Training10.1145/3463475 ↗
32. Ho, J., Jain, A., and Abbeel, P. (2020, January 6–12). Denoising diffusion probabilistic models. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA.
33. Song, J., Meng, C., and Ermon, S. (2020). Denoising Diffusion Implicit Models. arXiv.
34. Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., and Yang, F. (2023, January 1–6). SVDiff: Compact Parameter Space for Diffusion Fine-Tuning. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00673 ↗
35. Zhang, L., Rao, A., and Agrawala, M. (2023, January 1–6). Adding Conditional Control to Text-to-Image Diffusion Models. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00355 ↗
36. Li, S., Zeng, B., Feng, Y., Gao, S., Liu, X., Liu, J., Li, L., Tang, X., Hu, Y., and Liu, J. (2024, January 16–22). ZONE: Zero-Shot Instruction-Guided Local Editing. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00598 ↗
37. Chai, W., Guo, X., Wang, G., and Lu, Y. (2023, January 1–6). StableVideo: Text-driven Consistency-aware Diffusion Video Editing. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.02106 ↗
38. Low, W.F., and Lee, G.H. (2022). Minimal Neural Atlas: Parameterizing Complex Surfaces with Minimal Charts and Distortion. arXiv.10.1007/978-3-031-20086-1_27 ↗
39. Lu, E., Cole, F., Dekel, T., Xie, W., Zisserman, A., Salesin, D., Freeman, W.T., and Rubinstein, M. (2020). Layered neural rendering for retiming people in video. arXiv.10.1145/3414685.3417760 ↗
40. Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M.Z. (2023, January 1–6). Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00701 ↗
41. Ku, M., Wei, C., Ren, W., Yang, H., and Chen, W. (2024). AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks. arXiv.
42. Brooks, T., Holynski, A., and Efros, A.A. (2023, January 17–24). InstructPix2Pix: Learning to Follow Image Editing Instructions. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01764 ↗
43. Chen, X., Huang, L., Liu, Y., Shen, Y., Zhao, D., and Zhao, H. (2024, January 16–22). AnyDoor: Zero-shot Object-level Image Customization. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.10.1109/CVPR52733.2024.00630 ↗
44. Hu, J.E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
45. Zhao, R., Gu, Y., Wu, J.Z., Zhang, D.J., Liu, J.W., Wu, W., Keppo, J., and Shou, M.Z. (October2024, January 29). MotionDirector: Motion Customization of Text-to-Video Diffusion Models. Proceedings of the Computer Vision—ECCV 2024: 18th European Conference, Milan, Italy. Proceedings, Part LVI.10.1007/978-3-031-72992-8_16 ↗
46. Tu, S., Dai, Q., Cheng, Z.Q., Hu, H., Han, X., Wu, Z., and Jiang, Y.G. (2024, January 16–22). MotionEditor: Editing Video Motion via Content-Aware Diffusion. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00753 ↗