A Comprehensive Survey on Real-Time Image Super-Resolution for IoT and Delay-Sensitive Applications
Source abstract
Abstract
In contemporary computer vision, deep learning-based real-time single image super-resolution approaches have gained significant attention for their ability to enhance the resolution of images in real time. These approaches are interconnected with various other computer vision domains, including image segmentation and object detection. Numerous surveys have summarized the state of the image SR domain. However, there is no survey that specifically addresses real-time single image SR on IoT devices. Therefore, in this study, we aim to explore strategies, identify the technical challenges, and outline the future directions of SR research, with a special emphasis on real-time super-resolution techniques. We begin with an overview of the core concepts related to real-time SR, recent challenges, and algorithm classification and delve into potential application scenarios that merit attention. Additionally, we explore the challenges and identify promising research areas related to real-time SR specifically related to IoT devices, highlighting potential advancements, limitations, and opportunities for future innovation in this rapidly evolving field.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image and Signal Denoising Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of IT Convergence Engineering, Gachon University, Sujeong-Gu, Seongnam-Si 461-701, Gyeonggi-do, Republic of Korea
Funding
Korea Creative Content Agency grant funded by the Ministry of Culture, Sports, and Tourism in 2023
References (87)
Show source-provided references
1. Salvetti, F., Mazzia, V., Khaliq, A., and Chiaberge, M. (2020). Multi-image super resolution of remotely sensed images using residual attention deep neural networks. Remote Sens., 12.10.3390/rs12142207 ↗
2. Gendy, G., Mohammed, H., Sabor, N., and He, G. (2021, January 13–14). A deep pyramid attention network for single image super-resolution. Proceedings of the 2021 9th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), Alexandria, Egypt.10.1109/JAC-ECC54461.2021.9691443 ↗
3. Lightweight image super-resolution for IoT devices using deep residual feature distillation network10.1016/j.knosys.2023.111343 ↗
4. Rasool, M.A., Jeong, W., Ahmed, S., and Whangbo, T.K. (2024). Stellar SR: A Convolutional Local Feature Network for Lightweight Image Super-Resolution. Korean Soc. Broad Eng., 298–301.
5. Image super-resolution reconstruction based on feature map attention mechanism10.1007/s10489-020-02116-1 ↗
6. Single image super-resolution using a polymorphic parallel CNN10.1007/s10489-018-1270-7 ↗
7. Jo, Y., Oh, S.W., Kang, J., and Kim, S.J. (2018, January 18–22). Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00340 ↗
8. Rasool, M.A., Ahmed, S., Sabina, U., and Whangbo, T.J. (2023). MRESR: Multi-agent Reinforcement learning for Efficient Super-Resolution. Korean Soc. Broad Eng., 460–463.
9. Thawakar, O., Patil, P.W., Dudhane, A., Murala, S., and Kulkarni, U. (2019, January 18–21). Image and video super resolution using recurrent generative adversarial network. Proceedings of the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan.10.1109/AVSS.2019.8909900 ↗
10. Super-resolution in magnetic resonance imaging: A review
11. KONet: Towards a Weighted Ensemble Learning Model for Knee Osteoporosis Classification10.1109/ACCESS.2023.3348817 ↗
12. Smith, E., Fujimoto, S., and Meger, D. (2018). Multi-view silhouette and depth decomposition for high resolution 3d object representation. Adv. Neural Inf. Process. Syst., 31.
13. Zhang, K., Danelljan, M., Li, Y., Timofte, R., Liu, J., Tang, J., Wu, G., Zhu, Y., He, X., and Xu, W. (2020, January 23–28). Aim 2020 challenge on efficient super-resolution: Methods and results. Proceedings of the Computer Vision–ECCV 2020 Workshops, Glasgow, UK. Proceedings, Part III 16.
14. Blind image super-resolution: A survey and beyond
15. Liu, J., Tang, J., and Wu, G. (2020, January 23–28). Residual feature distillation network for lightweight image super-resolution. Proceedings of the Computer Vision–ECCV 2020 Workshops, Glasgow, UK. Proceedings, Part III 16.10.1007/978-3-030-67070-2_2 ↗
16. Wang, Y., Wang, L., Yang, J., An, W., and Guo, Y. (2019, January 15–20). Flickr1024: A large-scale dataset for stereo image super-resolution. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Long Beach, CA, USA.10.1109/ICCVW.2019.00478 ↗
17. Ayazoglu, M. (2021, January 20–25). Extremely lightweight quantization robust real-time single-image super resolution for mobile devices. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00280 ↗
18. Fang, J., Lin, H., Chen, X., and Zeng, K. (2022, January 18–24). A hybrid network of cnn and transformer for lightweight image super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00119 ↗
19. Bulat, A., and Tzimiropoulos, G. (2018, January 18–22). Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00019 ↗
20. Bei, Y., Damian, A., Hu, S., Menon, S., Ravi, N., and Rudin, C. (2018, January 18–22). New techniques for preserving global structure and denoising with low information loss in single-image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00132 ↗
21. Zhang, K., Zuo, W., and Zhang, L. (2018, January 18–22). Learning a single convolutional super-resolution network for multiple degradations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00344 ↗
22. Behjati, P., Rodriguez, P., Mehri, A., Hupont, I., Tena, C.F., and Gonzalez, J. (2021, January 11–17). Overnet: Lightweight multi-scale super-resolution with overscaling network. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual.10.1109/WACV48630.2021.00274 ↗
23. Fan, Y., Shi, H., Yu, J., Liu, D., Han, W., Yu, H., Wang, Z., Wang, X., and Huang, T.S. (2017, January 21–26). Balanced two-stage residual networks for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.154 ↗
24. Tai, Y., Yang, J., and Liu, X. (2017, January 21–26). Image super-resolution via deep recursive residual network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.298 ↗
25. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., and Guo, Y. (2021, January 20–25). Exploring sparsity in image super-resolution for efficient inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00488 ↗
26. Chen, R., Qu, Y., Zeng, K., Guo, J., Li, C., and Xie, Y. (2018, January 18–22). Persistent memory residual network for single image super resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00125 ↗
27. Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., and Schroers, C. (2018, January 18–22). A fully progressive approach to single-image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00131 ↗
28. A deep journey into super-resolution: A survey10.1145/3390462 ↗
29. Wang, X., Yu, K., Dong, C., and Loy, C.C. (2018, January 18–22). Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00070 ↗
30. Timofte, R., Rothe, R., and Van Gool, L. (2016, January 27–30). Seven ways to improve example-based single image super resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.206 ↗
31. Sketch-based manga retrieval using manga109 dataset10.1007/s11042-016-4020-z ↗
32. Lightweight image super-resolution based on deep learning: State-of-the-art and future directions10.1016/j.inffus.2023.01.024 ↗
33. Real-world single image super-resolution: A brief review10.1016/j.inffus.2021.09.005 ↗
34. Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 7–14). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings of the Eighth IEEE International Conference on Computer Vision, ICCV 2001, Vancouver, BC, Canada.
35. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., and Lin, L. (2018, January 18–23). Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00113 ↗
36. Image up-sampling using total-variation regularization with a new observation model10.1109/TIP.2005.851684 ↗
37. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
38. Wang, Z., Simoncelli, E.P., and Bovik, A.C. (2003, January 9–12). Multiscale structural similarity for image quality assessment. Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA.
39. Song, D., Xu, C., Jia, X., Chen, Y., Xu, C., and Wang, Y. (2020, January 7–12). Efficient residual dense block search for image super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6877 ↗
40. Zong, Z., Zha, L., Jiang, J., and Liu, X. (2022, January 18–24). Asymmetric information distillation network for lightweight super resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00131 ↗
41. Survey of single image super-resolution reconstruction10.1049/iet-ipr.2019.1438 ↗
42. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
43. Tipping, M., and Bishop, C. (2002). Bayesian image super-resolution. Adv. Neural Inf. Process. Syst., 15.
44. Adaptive rational fractal interpolation function for image super-resolution via local fractal analysis10.1016/j.imavis.2019.02.002 ↗
45. Luo, Z., Li, Y., Yu, L., Wu, Q., Wen, Z., Fan, H., and Liu, S. (2022, January 23–27). Fast nearest convolution for real-time efficient image super-resolution. Proceedings of the Computer Vision–ECCV 2022 Workshops, Tel Aviv, Israel. Proceedings, Part II.10.1007/978-3-031-25063-7_35 ↗
46. Learner, T.A. (2023, May 09). Image Processing—Nearest Neighbour Interpolation. Available online: https://theailearner.com/2018/12/29/image-processing-nearest-neighbour-interpolation/.
47. Khaledyan, D., Amirany, A., Jafari, K., Moaiyeri, M.H., Khuzani, A.Z., and Mashhadi, N. (November, January 29). Low-cost implementation of bilinear and bicubic image interpolation for real-time image super-resolution. Proceedings of the 2020 IEEE Global Humanitarian Technology Conference (GHTC), Seattle, WA, USA.10.1109/GHTC46280.2020.9342625 ↗
48. Cho, H.M., and Choi, K. (2022, January 18–24). Super-Resolution based Video Coding Scheme. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00190 ↗
49. Image super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
50. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
51. Optimal physical preprocessing for example-based super-resolution (Conference Presentation)
52. Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017, January 21–26). Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
53. Lai, W.S., Huang, J.B., Ahuja, N., and Yang, M.H. (2017, January 21–26). Deep laplacian pyramid networks for fast and accurate super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.618 ↗
54. Zhang, K., Li, D., Luo, W., Ren, W., Stenger, B., Liu, W., Li, H., and Yang, M.H. (2021, January 11–17). Benchmarking ultra-high-definition image super-resolution. Proceedings of the IEEE/CVF International Conference on Computer Vision, Virtual.10.1109/ICCV48922.2021.01450 ↗
55. Adaptive importance learning for improving lightweight image super-resolution network10.1007/s11263-019-01253-6 ↗
56. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7–13). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.123 ↗
57. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
58. An information fidelity criterion for image quality assessment using natural scene statistics10.1109/TIP.2005.859389 ↗
59. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
60. Making a “completely blind” image quality analyzer10.1109/LSP.2012.2227726 ↗
61. Yang, C.Y., Ma, C., and Yang, M.H. (2014, January 6–12). Single-image super-resolution: A benchmark. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part IV 13.
62. Red Hat (2024, August 15). What Is a Latency-Sensitive Application?. Available online: https://www.redhat.com/en/topics/edge-computing/latency-sensitive-applications#what-is-a-latency-sensitive-application.
63. Hui, Z., Wang, X., and Gao, X. (2018, January 18–23). Fast and accurate single image super-resolution via information distillation network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00082 ↗
64. Feature-domain adaptive contrastive distillation for efficient single image super-resolution10.1109/ACCESS.2023.3335934 ↗
65. Hui, Z., Gao, X., Yang, Y., and Wang, X. (2019, January 21–25). Lightweight image super-resolution with information multi-distillation network. Proceedings of the 27th Acm International Conference on Multimedia, Nice, France.10.1145/3343031.3351084 ↗
66. Feature similarity rank-based information distillation network for lightweight image superresolution10.1016/j.knosys.2023.110437 ↗
67. TAKDSR: Teacher Assistant Knowledge Distillation Framework for Graphics Image Super-Resolution10.1109/ACCESS.2023.3323273 ↗
68. Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective10.1109/TIP.2024.3434426 ↗
69. Koshelev, I., and Somov, A. (2021, January 13–16). Learned Gradient Descent Performance in Bicubic Super-Resolution Task. Proceedings of the IECON 2021–47th Annual Conference of the IEEE Industrial Electronics Society, Toronto, ON, Canada.10.1109/IECON48115.2021.9589797 ↗
70. A computationally efficient airborne forward-looking super-resolution imaging method based on sparse Bayesian learning
71. Uncertainty quantification in super-resolution guided wave array imaging using a variational Bayesian deep learning approach10.1016/j.ndteint.2022.102753 ↗
72. Super-resolution with sparse arrays: A nonasymptotic analysis of spatiotemporal trade-offs10.1109/TSP.2023.3330670 ↗
73. Edge-priority-extraction network using re-parameterization for real-time super-resolution10.1007/s00371-023-03197-y ↗
74. DDistill-SR: Reparameterized dynamic distillation network for lightweight image super-resolution10.1109/TMM.2022.3219646 ↗
75. Deng, W., Yuan, H., Deng, L., and Lu, Z. (2023, January 17–24). Reparameterized residual feature network for lightweight image super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPRW59228.2023.00172 ↗
76. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
77. Simonyan, K. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
78. Dong, C., Loy, C.C., and Tang, X. (2016, January 11–14). Accelerating the super-resolution convolutional neural network. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part II 14.
79. Kong, F., Li, M., Liu, S., Liu, D., He, J., Bai, Y., Chen, F., and Fu, L. (2022, January 18–24). Residual local feature network for efficient super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00092 ↗
80. Accurate and lightweight image super-resolution with model-guided deep unfolding network10.1109/JSTSP.2020.3037516 ↗
81. Zhou, D., Sun, S., Li, M., Liu, X., and Mi, S. (2024). GAN-based super-resolution for confocal superficial eyelid imaging: Real-time, domain generalization, and noise robustness. Biomed. Signal Process. Control, 89.10.1016/j.bspc.2023.105668 ↗
82. Bhatia, V., and Kumar, Y. (2020). Attaining real-time super-resolution for microscopic images using gan. arXiv.
83. Hong, C., Baik, S., Kim, H., Nah, S., and Lee, K.M. (2022, January 23–27). Cadyq: Content-aware dynamic quantization for image super-resolution. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20071-7_22 ↗
84. A High-Performance Accelerator for Real-Time Super-Resolution on Edge FPGAs10.1145/3678006 ↗
85. Veluri, B., Pernu, C., Saffari, A., Smith, J., Taylor, M., and Gollakota, S. (2023, January 2–6). NeuriCam: Key-Frame Video Super-Resolution and Colorization for IoT Cameras. Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, Madrid, Spain.10.1145/3570361.3592523 ↗
86. Almutairi, R., Bergami, G., and Morgan, G. (2024). Advancements and Challenges in IoT Simulators: A Comprehensive Review. Sensors, 24.10.3390/s24051511 ↗
87. Bouchemal, N., Kallel, S., and Bouchemal, N. (2018, January 27–29). A survey: WSN heterogeneous architecture platform for IoT. Proceedings of the Machine Learning for Networking: First International Conference, MLN 2018, Paris, France. Revised Selected Papers 1.10.1007/978-3-030-19945-6_23 ↗