SNR-Guided Enhancement and Autoregressive Depth Estimation for Single-Photon Camera Imaging
Source abstract
Abstract
Recent advances in deep learning have intensified the need for robust low-light image processing in critical applications like autonomous driving, where single-photon cameras (SPCs) offer high photon sensitivity but produce noisy outputs requiring specialized enhancement. This work addresses this challenge through a unified framework integrating three key components: an SNR-guided adaptive enhancement framework that dynamically processes regions with varying noise levels using spatial-adaptive operations and intelligent feature fusion; a specialized self-attention mechanism optimized for low-light conditions; and a conditional autoregressive generation approach applied to robust depth estimation from enhanced SPC images. Our comprehensive evaluation across multiple datasets demonstrates improved performance over state-of-the-art methods, achieving a PSNR of 24.61 dB on the LOL-v1 dataset and effectively recovering fine-grained textures in depth estimation, particularly in real-world SPC applications, while maintaining computational efficiency. The integrated solution effectively bridges the gap between single-photon sensing and practical computer vision tasks, facilitating more reliable operation in photon-starved environments through its novel combination of adaptive noise processing, attention-based feature enhancement, and generative depth reconstruction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Optical Sensing Technologies · Advanced Vision and Imaging · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Information Engineering, Shanghai Polytechnic University, Shanghai 201209, China
Department of Engineering, Durham University, Durham DH1 3LE, UK
References (69)
Show source-provided references
1. LIME: Low-Light Image Enhancement via Illumination Map Estimation
2. Chen, C., Chen, Q., Xu, J., and Koltun, V. (2018, January 18–22). Learning to See in the Dark. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00347 ↗
3. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. arXiv.
4. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., and Ray, A. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv.
5. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
6. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., and Gallé, M. (2022). BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv.
7. Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., and Lu, Y. (2021). ERNIE 3.0: Large-Scale Knowledge Enhanced Pre-Training for Language Understanding and Generation. arXiv.
8. Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19–24). Pixel Recurrent Neural Networks. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA.
9. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021, January 18–24). Zero-Shot Text-to-Image Generation. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual.
10. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
11. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
12. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
13. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
14. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-End Object Detection with Transformers. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
15. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. (2020, January 12–18). Generative Pretraining From Pixels. Proceedings of the 37th International Conference on Machine Learning (ICML), Vienna, Austria.
16. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
17. Navab, N., Hornegger, J., Wells, W., and Frangi, A. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Springer.10.1007/978-3-319-24553-9 ↗
18. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual.
19. Peebles, W., and Xie, S. (2023, January 2–6). Scalable Diffusion Models with Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00387 ↗
20. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. Advances in Neural Information Processing Systems, Curran Associates, Inc.
21. Ho, J., and Salimans, T. (2021, January 13). Classifier-Free Diffusion Guidance. Proceedings of the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, Virtual.
22. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 18–24). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
23. Cascaded Diffusion Models for High Fidelity Image Generation
24. Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (2016, January 5–10). Conditional Image Generation with PixelCNN Decoders. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain.
25. Esser, P., Rombach, R., and Ommer, B. (2021, January 20–25). Taming Transformers for High-Resolution Image Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01268 ↗
26. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019, January 8–14). Generating Diverse High-Fidelity Images with VQ-VAE-2. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
27. Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.S. (2022, January 18–24). Autoregressive Image Generation Using Residual Quantization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01123 ↗
28. Tian, K., Jiang, Y., Yuan, Z., Peng, X., and Wang, G. (2024, January 10–15). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.10.52202/079017-2694 ↗
29. Yang, T., Zhu, Y., Xie, Y., Zhang, A., Chen, C., and Li, M. (2023, January 1–5). AIM: Adapting Image Models for Efficient Video Action Recognition. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
30. Liu, D., Zhao, S., Zhuo, L., Xu, W., Li, B., and Elhoseiny, M. (2024). Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining. arXiv.
31. Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W.T., and Rubinstein, M. (2022, January 18–24). MaskGIT: Masked Generative Image Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01103 ↗
32. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Minneapolis, MN, USA.
33. Photon-Number-Resolving Megapixel Image Sensor at Room Temperature Without Avalanche Gain10.1364/OPTICA.4.001474 ↗
34. Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model10.1109/TIP.2018.2810539 ↗
35. Retinex Processing for Automatic Image Enhancement10.1117/1.1636183 ↗
36. Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images10.1109/TIP.2018.2794218 ↗
37. Guo, C., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., and Cong, R. (2020, January 13–19). Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00185 ↗
38. EnlightenGAN: Deep Light Enhancement Without Paired Supervision10.1109/TIP.2021.3051462 ↗
39. Wang, R., Zhang, Q., Fu, C.-W., Shen, X., Zheng, W.-S., and Jia, J. (2019, January 15–20). Underexposed Photo Enhancement Using Deep Illumination Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00701 ↗
40. Buades, A., Coll, B., and Morel, J.-M. (2005, January 20–25). A Non-Local Algorithm for Image Denoising. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA.
41. Yang, W., Wang, S., Fang, Y., Wang, Y., and Liu, J. (2020, January 13–19). From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00313 ↗
42. Chen, Y.-S., Wang, Y.-C., Kao, M.-H., and Chuang, Y.-Y. (2018, January 18–23). Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00660 ↗
43. Detectors for Single-Molecule Fluorescence Imaging and Spectroscopy10.1080/09500340600769067 ↗
44. Liu, L., Jia, X., Liu, J., and Tian, Q. (2020, January 13–19). Joint Demosaicing and Denoising with Self Guidance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00231 ↗
45. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 Dataset, California Institute of Technology. Technical Report CNS-TR-2011-001.
46. Guyon, I., Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4–9). Neural Discrete Representation Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
47. Lamb, A., Dumoulin, V., and Courville, A. (2016). Discriminative Regularization for Generative Models. arXiv.
48. Bengio, Y., Léonard, N., and Courville, A. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv.
49. Larsen, A.B.L., Sønderby, S.K., Larochelle, H., and Winther, O. (2016, January 19–24). Autoencoding Beyond Pixels Using a Learned Similarity Metric. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA.
50. Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 11–14). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_43 ↗
51. Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (2016, January 5–10). Generating Images with Perceptual Similarity Metrics Based on Deep Networks. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain.
52. High-Fidelity Generative Image Compression
53. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
54. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
55. Chen, W., Wang, W., Yang, W., and Liu, J. (2018, January 3–6). Deep Retinex Decomposition for Low-Light Enhancement. Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK.
56. Sparse Gradient Regularized Deep Retinex Network for Robust Low-Light Image Enhancement10.1109/TIP.2021.3050850 ↗
57. Chen, C., Chen, Q., Do, M.N., and Koltun, V. (November, January 27). Seeing Motion in the Dark. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
58. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
59. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A ConvNet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
60. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012, January 7–13). Indoor Segmentation and Support Inference from RGBD Images. Proceedings of the European Conference on Computer Vision (ECCV), Florence, Italy.10.1007/978-3-642-33715-4_54 ↗
61. Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., and Gao, W. (2021, January 20–25). Pre-Trained Image Processing Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01212 ↗
62. Luminance-Aware Pyramid Network for Low-Light Image Enhancement10.1109/TMM.2020.3021243 ↗
63. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., and Shao, L. (2020, January 23–28). Learning Enriched Features for Real Image Restoration and Enhancement. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58595-2_30 ↗
64. Liu, R., Ma, L., Zhang, J., Fan, X., and Luo, Z. (2021, January 20–25). Retinex-Inspired Unrolling With Cooperative Prior Architecture Search for Low-Light Image Enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01042 ↗
65. Cai, Y., Bian, H., Lin, J., Wang, H., Timofte, R., and Zhang, Y. (2023, January 1–6). Retinexformer: One-Stage Retinex-Based Transformer for Low-Light Image Enhancement. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01149 ↗
66. SNR-Prior Guided Trajectory-Aware Transformer for Low-Light Video Enhancement10.1109/TCSVT.2023.3299232 ↗
67. Cheng, Z., Wu, Y., Tian, F., Feng, Z., and Li, Y. (2025). MSF-ACA: Low-Light Image Enhancement Network Based on Multi-Scale Feature Fusion and Adaptive Contrast Adjustment. Sensors, 25.10.20944/preprints202506.0758.v1 ↗
68. Jan, A., and Seo, S. (2023). Monocular Depth Estimation Using Res-UNet with an Attention Model. Appl. Sci., 13.10.3390/app13106319 ↗
69. Sun, Y., Ni, M., Zhao, M., Yang, Z., Peng, Y., and Cao, D. (2024). Low-Light Enhancement Method with Dual Branch Feature Fusion and Learnable Regularized Attention. Front. Optoelectron., 17.10.1007/s12200-024-00129-z ↗