Video Super Resolution Using a Selective Edge Aggregation Network
Source abstract
Abstract
An edge map is a feature map representing the contours of the object in the image. There was a Single Image Super Resolution (SISR) method using the edge map, which achieved a notable SSIM performance improvement. Unlike SISR, Video Super Resolution (VSR) uses video, which consists of consecutive images with temporal features. Therefore, some VSR models adopted motion estimation and motion compensation to apply spatio-temporal feature maps. Unlike the models above, we tried a different method by adding edge structure information and its related post-processing to the existing model. Our model “Video Super Resolution Using a Selective Edge Aggregation Network (SEAN)” consists of a total of two stages. First, the model selectively generates an edge map using the target frame and also the neighboring frame. At this stage, we adopt the magnitude loss function so that the output of SEAN more clearly learns the contours of each object. Second, the final output is generated using the refinement (post-processing) module. SEAN shows more distinct object contours and better color correction compared to other existing models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image and Signal Denoising Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Artificial Intelligence Laboratory, Hanyang University, Seoul 04763, Korea
Funding
Institute for Information and Communications Technology Promotion
References (22)
Show source-provided references
1. Video enhancement with task-oriented flow10.1007/s11263-018-01144-2 ↗
2. Haris, M., Shakhnarovich, G., and Ukita, N. (2019, January 16–20). Recurrent back-projection network for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.10.1109/CVPR.2019.00402 ↗
3. Liu, C., and Sun, D. (2011, January 20–25). A bayesian approach to adaptive video super resolution. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995614 ↗
4. Li, W., Tao, X., Guo, T., Qi, L., Lu, J., and Jia, J. (2020). Mucan: Multi-correspondence aggregation network for video super-resolution. Lecture Notes in Computer Science, Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020, Springer.10.1007/978-3-030-58607-2_20 ↗
5. Kim, J., Lee, J.K., and Lee, K.M. (July, January 26). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
6. Image super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
7. Nazeri, K., Thasarathan, H., and Ebrahimi, M. (November, January 27). Edge-informed single image super-resolution. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Korea.
8. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
9. Tian, Y., Zhang, Y., Fu, Y., and Xu, C. (2020, January 14–19). Tdan: Temporally-deformable alignment network for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00342 ↗
10. Wang, X., Chan, K.C., Yu, K., Dong, C., and Change Loy, C. (2019, January 16–20). Edvr: Video restoration with enhanced deformable convolutional networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00247 ↗
11. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. (2017, January 22–29). Deformable convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.89 ↗
12. Haris, M., Shakhnarovich, G., and Ukita, N. (2020, January 14–19). Space-time-aware multi-resolution video enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00293 ↗
13. Park, W.S., and Kim, M. (2016, January 11–12). CNN-based in-loop filtering for coding efficiency improvement. Proceedings of the 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Bordeaux, France.10.1109/IVMSPW.2016.7528223 ↗
14. Dai, Y., Liu, D., and Wu, F. (2017). A convolutional neural network approach for post-processing in HEVC intra coding. Lecture Notes in Computer Science, Proceedings of the International Conference on Multimedia Modeling, Reykjavik, Iceland, 4–6 January 2017, Springer.10.1007/978-3-319-51811-4_3 ↗
15. Kang, J., Kim, S., and Lee, K.M. (2017, January 17–20). Multi-modal/multi-scale convolutional neural network based in-loop filter design for next generation video codec. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296236 ↗
16. Wang, T., Chen, M., and Chao, H. (2017, January 4–7). A novel deep learning-based method of improving coding efficiency from the decoder-end for HEVC. Proceedings of the 2017 Data Compression Conference (DCC), Snowbird, UT, USA.10.1109/DCC.2017.42 ↗
17. Ma, L., Tian, Y., and Huang, T. (2018, January 13–16). Residual-based video restoration for HEVC intra coding. Proceedings of the 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), Xi’an, China.10.1109/BigMM.2018.8499072 ↗
18. Liu, C. (2009). Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. [Ph.D. Thesis, Massachusetts Institute of Technology].
19. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
20. Timofte, R., De Smet, V., and Van Gool, L. (2013, January 1–8). Anchored neighborhood regression for fast example-based super-resolution. Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia.10.1109/ICCV.2013.241 ↗
21. Image super-resolution via sparse representation10.1109/TIP.2010.2050625 ↗
22. Nah, S., Baik, S., Hong, S., Moon, G., Son, S., Timofte, R., and Lee, K.M. (2019, January 16–20). NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study. Proceedings of the CVPR Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00251 ↗