Super-Resolution Task Inference Acceleration for In-Vehicle Real-Time Video via Edge–End Collaboration
Source abstract
Abstract
As intelligent transportation systems continue to advance, on-board surveillance video has become essential for train safety and intelligent scheduling. However, high-resolution video transmission faces bandwidth limitations, and existing deep learning-based super-resolution models find it difficult to meet real-time requirements due to high computational complexity. To address this, this paper proposes an “edge–end” collaborative multi-terminal task inference framework, which improves inference speed by integrating resources of in-vehicle end devices and edge servers. The framework establishes a real-time-priority mathematical model, uses game theory to solve the problem of minimizing multi-terminal task inference latency, and proposes a multi-terminal task model partitioning strategy and an adaptive adjustment mechanism. It can dynamically partition the model according to device performance and network status, prioritizing real-time performance and minimizing the maximum inference delay. Experimental results show that the dynamic model partitioning mechanism can adaptively determine the optimal partition point, effectively reducing the inference delay of each end device in high-speed mobile and bandwidth-constrained scenarios and providing high-quality video data support for safety monitoring and intelligent analysis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Image and Video Quality Assessment · Sparse and Compressive Sensing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
CRRC Nanjing Puzhen Co., Ltd., Nanjing 210031, China
College of Software Engineering, Southeast University, Nanjing 211189, China
Funding
Guangxi Science and Technology Achievement Transformation Program · Science and Technology Major Special Program of Jiangsu · Natural Science Foundation of Jiangsu Province · CRRC S&T Project
References (55)
Show source-provided references
1. QoE modeling for HTTP adaptive video streaming–a survey and open challenges10.1109/ACCESS.2019.2901778 ↗
2. Barnett, T., Jain, S., Andra, U., and Khurana, T. (2018). Cisco Visual Networking Index (VNI) Complete Forecast Update, 2017–2022. Americas/EMEAR Cisco Knowledge Network (CKN) Presentation, Cisco.
3. Video super-resolution based on deep learning: A comprehensive survey10.1007/s10462-022-10147-y ↗
4. Deep learning for image super-resolution: A survey10.1109/TPAMI.2020.2982166 ↗
5. Deep Learning-based Single-image Super-resolution: A comprehensive review10.1109/ACCESS.2023.3251396 ↗
6. Hui, Z., Wang, X., and Gao, X. (2018, January 18–23). Fast and Accurate Single Image Super-Resolution via Information Distillation Network. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00082 ↗
7. Lee, R., Venieris, S.I., Dudziak, L., Bhattacharya, S., and Lane, N.D. (2019, January 21–25). Mobisr: Efficient on-device super-resolution through heterogeneous mobile processors. Proceedings of the 25th Annual International Conference on Mobile Computing and Networking, Los Cabos, Mexico.10.1145/3300061.3345455 ↗
8. Kong, X., Zhao, H., Qiao, Y., and Dong, C. (2021, January 20–25). Classsr: A general framework to accelerate super-resolution networks by data characteristic. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01184 ↗
9. Supremo: Cloud-assisted low-latency super-resolution in mobile devices10.1109/TMC.2020.3025300 ↗
10. Hui, Z., Gao, X., Yang, Y., and Wang, X. (2019, January 21–25). Lightweight image super-resolution with information multi-distillation network. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351084 ↗
11. Lightweight Super-Resolution Model for Complete Model Copyright Protection10.26599/TST.2023.9010082 ↗
12. Cloud-Assisted Computation Offloading to Support Mobile Services10.1109/TCC.2014.2350471 ↗
13. A Review of Low-End, Middle-End, and High-End IoT Devices10.1109/ACCESS.2018.2879615 ↗
14. Mobile cloud computing for computation offloading: Issues and challenges10.1016/j.aci.2016.11.002 ↗
15. Li, M., Zhang, X., Guo, J., and Li, F. (2023). Cloud–Edge Collaborative Inference with Network Pruning. Electronics, 12.10.3390/electronics12173598 ↗
16. Crime: Input-dependent collaborative inference for recurrent neural networks
17. Kou, Z., Liu, Z., Yang, X., and He, X. (2023, January 26–28). Adaptive Deep Inference Framework for Cloud-Edge Collaboration. Proceedings of the 2023 International Conference on Computers, Information Processing and Advanced Education (CIPAE), Ottawa, ON, Canada.10.1109/CIPAE60493.2023.00014 ↗
18. Li, M., Yang, L., and Huang, J. (2023, January 8–10). A Deep Reinforcement Learning Approach for DNN Collaborative Inference in Mobile Edge Computing. Proceedings of the 2023 10th International Conference on Electrical and Electronics Engineering (ICEEE), Istanbul, Türkiye.10.1109/ICEEE59925.2023.00052 ↗
19. Multi-agent collaborative inference via DNN decoupling: Intermediate feature compression and edge learning10.1109/TMC.2022.3183098 ↗
20. Reinforcement learning based energy-efficient collaborative inference for mobile edge computing10.1109/TCOMM.2022.3229033 ↗
21. LSTM network-based adaptation approach for dynamic integration in intelligent end-edge-cloud systems10.26599/TST.2023.9010086 ↗
22. Resolving power and information10.1364/JOSA.45.000497 ↗
23. Diffraction and resolving power10.1364/JOSA.54.000931 ↗
24. Goodman, J.W. (2005). Introduction to Fourier Optics, Roberts and Company Publishers.
25. Transformer and GAN-based super-resolution reconstruction network for medical images10.26599/TST.2022.9010071 ↗
26. View interpolation for image synthesis
27. Contrast-guided image interpolation10.1109/TIP.2013.2271849 ↗
28. Zhang, X., Ma, S., Zhang, Y., Zhang, L., and Gao, W. (2009). Nonlocal edge-directed interpolation. Pacific-Rim Conference on Multimedia, Springer.10.1007/978-3-642-10467-1_121 ↗
29. Missing intensity interpolation using a kernel PCA-based POCS algorithm and its applications10.1109/TIP.2010.2070072 ↗
30. Liu, J., Xue, R., and Li, Y. (2019, January 10–13). Single image super-resolution reconstruction based on multi-directionality of the edge. Proceedings of the Eleventh International Conference on Digital Image Processing (ICDIP 2019), Guangzhou, China.10.1117/12.2540282 ↗
31. Example-based super-resolution10.1109/38.988747 ↗
32. Yang, J., Wright, J., Huang, T., and Ma, Y. (2008, January 23–28). Image super-resolution as sparse representation of raw image patches. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.
33. Real-time video super-resolution reconstruction using wavelet transforms and sparse representation
34. Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6–12). Learning a deep convolutional network for image super-resolution. Proceedings of the Computer Vision–ECCV 2014, 13th European Conference, Zurich, Switzerland. Proceedings, Part IV 13.
35. Dong, C., Loy, C.C., and Tang, X. (2016, January 11–14). Accelerating the super-resolution convolutional neural network. Proceedings of the Computer Vision–ECCV 2016, 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part II 14.
36. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
37. Zhang, Z., Wang, Z., Lin, Z., and Qi, H. (2019, January 15–20). Image super-resolution by neural texture transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00817 ↗
38. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.181 ↗
39. Tai, Y., Yang, J., and Liu, X. (2017, January 21–26). Image super-resolution via deep recursive residual network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.298 ↗
40. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
41. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
42. Tong, T., Li, G., Liu, X., and Gao, Q. (October, January 22). Image super-resolution using dense skip connections. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.
43. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
44. Wang, L., Guo, Y., Lin, Z., Deng, X., and An, W. (2018, January 2–6). Learning for video super-resolution through HR optical flow estimation. Proceedings of the Computer Vision–ACCV 2018, 14th Asian Conference on Computer Vision, Perth, Australia. Revised Selected Papers, Part I 14.
45. Sajjadi, M.S.M., Vemulapalli, R., and Brown, M. (2018, January 18–23). Frame-recurrent video super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00693 ↗
46. Yan, B., Lin, C., and Tan, W. (February, January 27). Frame and feature-context video super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
47. Chan, K.C., Wang, X., Yu, K., Dong, C., and Loy, C.C. (2021, January 20–25). BasicVSR: The search for essential components in video super-resolution and beyond. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00491 ↗
48. Chan, K.C., Zhou, S., Xu, X., and Loy, C.C. (2022, January 18–24). Investigating tradeoffs in real-world video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00587 ↗
49. Liu, C., Yang, H., Fu, J., and Qian, X. (2022, January 18–24). Learning trajectory-aware transformer for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00560 ↗
50. Video super-resolution via a spatio-temporal alignment network10.1109/TIP.2022.3146625 ↗
51. Chan, K.C., Zhou, S., Xu, X., and Loy, C.C. (2022, January 18–24). BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00588 ↗
52. Edgeduet: Tiling small object detection for edge assisted autonomous mobile vision10.1109/TNET.2022.3223412 ↗
53. Potential games10.1006/game.1996.0044 ↗
54. Li, L., Xu, K., Li, T., Zheng, K., Peng, C., Wang, D., Wang, X., Shen, M., and Mijumbi, R. (2018, January 20–25). A measurement study on multi-path TCP with multiple cellular carriers on high speed rails. Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, Budapest, Hungary.10.1145/3230543.3230556 ↗
55. Raca, D., Quinlan, J.J., Zahran, A.H., and Sreenan, C.J. (2018, January 12–15). Beyond throughput: A 4G LTE dataset with channel and context metrics. Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam, The Netherlands.10.1145/3204949.3208123 ↗