Dense Semantic Forecasting with Multi-Level Feature Warping
Source abstract
Abstract
Anticipation of per-pixel semantics in a future unobserved frame is also known as dense semantic forecasting. State-of-the-art methods are based on single-level regression of a subsampled abstract representation of a recognition model. However, single-level regression cannot account for skip connections from the backbone to the upsampling path. We propose to address this shortcoming by warping shallow features from observed images with upsampled feature flow. Our goal is not straightforward, since warping with coarse feature flow introduces noise into the forecasted features. We therefore base our work on single-frame models that are more resistant to the noise in skip connections. To achieve this, we propose a training procedure that enables recognition models to operate reasonably well with or without skip connections. Validation experiments reveal interesting insights into the influence of particular skip connections on recognition accuracy. Our forecasting method delivers 70.2% mIoU 0.18 s into the future and 58.5% mIoU 0.54 s into the future. These experiments show 0.6 mIoU points of improved accuracy with respect to the baseline and reveal promising directions for future work.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Cell Image Analysis Techniques · Advanced Image Fusion Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, 10000 Zagreb, Croatia
Funding
Rimac Technologies, Croatian Science Foundation · European Regional Development Fund
References (37)
Show source-provided references
1. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., and Savarese, S. (2016, January 27–30). Social LSTM: Human Trajectory Prediction in Crowded Spaces. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.110 ↗
2. Vondrick, C., Pirsiavash, H., and Torralba, A. (2016, January 27–30). Anticipating Visual Representations from Unlabeled Video. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.10.1109/CVPR.2016.18 ↗
3. Yao, Y., Xu, M., Choi, C., Crandall, D.J., Atkins, E.M., and Dariush, B. (2019, January 20–24). Egocentric vision-based future vehicle localization for intelligent driving assistance systems. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794474 ↗
4. Hu, A., Cotter, F., Mohan, N., Gurau, C., and Kendall, A. (2020, January 23–28). Probabilistic Future Prediction for Video Scene Understanding. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58517-4_45 ↗
5. Luc, P., Neverova, N., Couprie, C., Verbeek, J., and LeCun, Y. (2017, January 22–29). Predicting deeper into the future of semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.77 ↗
6. Terwilliger, A., Brazil, G., and Liu, X. (2019, January 7–11). Recurrent Flow-Guided Semantic Forecasting. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.10.1109/WACV.2019.00186 ↗
7. Graber, C., Tsai, G., Firman, M., Brostow, G., and Schwing, A.G. (2021, January 20–25). Panoptic Segmentation Forecasting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01233 ↗
8. Saric, J., Orsic, M., Antunovic, T., Vrazic, S., and Segvic, S. (2020, January 13–19). Warp to the Future: Joint Forecasting of Features and Feature Motion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01066 ↗
9. Lin, Z., Sun, J., Hu, J.F., Yu, Q., Lai, J.H., and Zheng, W.S. (2021, January 10–17). Predictive Feature Learning for Future Segmentation Prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00727 ↗
10. Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., and Belongie, S.J. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
11. Kreso, I., Segvic, S., and Krapac, J. (2017, January 22–29). Ladder-Style DenseNets for Semantic Segmentation of Large Natural Images. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Venice, Italy.10.1109/TITS.2020.2984894 ↗
12. APANet: Auto-Path Aggregation for Future Instance Segmentation Prediction
13. Šarić, J., Oršić, M., Antunović, T., Vražić, S., and Šegvić, S. (2019, January 10–13). Single level feature-to-feature forecasting with deformable convolutions. Proceedings of the German Conference on Pattern Recognition, Dortmund, Germany.10.1007/978-3-030-33676-9_13 ↗
14. Imagenet classification with deep convolutional neural networks
15. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
16. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
17. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid Scene Parsing Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
18. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
19. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
20. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
21. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
22. Cheng, B., Collins, M.D., Zhu, Y., Liu, T., Huang, T.S., Adam, H., and Chen, L.C. (2020, January 13–19). Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01249 ↗
23. Efficient ladder-style densenets for semantic segmentation of large images
24. Chaurasia, A., and Culurciello, E. (2017, January 10–13). Linknet: Exploiting encoder representations for efficient semantic segmentation. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), St. Petersburg, FL, USA.10.1109/VCIP.2017.8305148 ↗
25. Orsic, M., Kreso, I., Bevandic, P., and Segvic, S. (2019, January 15–20). In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01289 ↗
26. Luc, P., Couprie, C., Lecun, Y., and Verbeek, J. (2018, January 8–14). Predicting Future Instance Segmentation by Forecasting Convolutional Features. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01240-3_36 ↗
27. Bhattacharyya, A., Fritz, M., and Schiele, B. (2019, January 6–9). Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
28. Nabavi, S.S., Rochan, M., and Wang, Y. (2018). Future Semantic Segmentation with Convolutional LSTM. arXiv.
29. Šarić, J., Vražić, S., and Šegvić, S. (2021). Dense semantic forecasting in video by joint regression of features and feature motion. IEEE Trans. Neural Netw. Learn. Syst., 1–13.
30. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
33. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
34. Segmenting the future10.1109/LRA.2020.2992184 ↗
35. Vora, S., Mahjourian, R., Pirk, S., and Angelova, A. (2018, January 8–14). Future Semantic Segmentation Using 3D Structure. Proceedings of the ECCV 3D Reconstruction meets Semantics Workshop, Munich, Germany.
36. Chen, X., and Han, Y. (2019, January 8–12). Multi-Timescale Context Encoding for Scene Parsing Prediction. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.10.1109/ICME.2019.00280 ↗
37. Teed, Z., and Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020, Springer.10.1007/978-3-030-58536-5_24 ↗