EdgeV-SE: Self-Reflective Fine-Tuning Framework for Edge-Deployable Vision-Language Models
Source abstract
Abstract
The deployment of Vision-Language Models (VLMs) in Satellite IoT scenarios is critical for real-time disaster assessment but is often hindered by the substantial memory and compute requirements of state-of-the-art models. While parameter-efficient fine-tuning (PEFT) enables adaptation, with minimal computational overhead, standard supervised methods often fail to ensure robustness and reliability on resource-constrained edge devices. To address this, we propose EdgeV-SE, a self-reflective fine-tuning framework that significantly enhances the performance of VLM without introducing any inference-time overhead. Our framework incorporates an uncertainty-aware self-reflection mechanism with asymmetric dual pathways: a generative linguistic pathway and an auxiliary discriminative visual pathway. By estimating uncertainty from the linguistic pathway using a log-likelihood margin between class verbalizers, EdgeV-SE identifies ambiguous samples and refines its decision boundaries via consistency regularization and cross-pathway mutual learning. Experimental results on hurricane damage assessment demonstrate that our approach improves image classification accuracy, enhances image–text semantic alignment, and achieves superior caption quality. Notably, our work achieves these gains while maintaining practical deployment on a commercial off-the-shelf edge device such as NVIDIA Jetson Orin Nano, preserving the inference latency and memory footprint. Overall, our work contributes a unified self-reflective fine-tuning framework that improves robustness, calibration, and deployability of VLMs on edge devices.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information and Communication Engineering, Dongguk University, Seoul 04620, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation · Dongguk University
References (52)
Show source-provided references
1. Masson-Delmotte, V., Zhai, P., Pirani, A., Connors, S.L., Péan, C., Berger, S., Caud, N., Chen, Y., Goldfarb, L., and Gomis, M.I. (2021). Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, Cambridge University Press.
2. Liu, Z., Jiang, Y., and Rong, J. (2023). Resource Allocation Strategy for Satellite Edge Computing Based on Task Dependency. Appl. Sci., 13.10.3390/app131810027 ↗
3. Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2023, January 10–16). Visual Instruction Tuning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
4. OpenAI (2023). GPT-4V(ision) System Card, OpenAI. Available online: https://cdn.openai.com/papers/GPTV_System_Card.pdf.
5. NVIDIA Corporation (2026, January 11). Jetson Orin Nano Developer Kit User Guide. Available online: https://developer.nvidia.com/embedded/learn/jetson-orin-nano-devkit-user-guide/index.html.
6. Shin, D.-J., and Kim, J.-J. (2022). A Deep Learning Framework Performance Evaluation to Use YOLO in Nvidia Jetson Platform. Appl. Sci., 12.10.3390/app12083734 ↗
7. Retrieve-then-compare mitigates visual hallucination in multi-modal large language models10.20517/ir.2025.13 ↗
8. Multimodal semantic communication system based on graph neural networks10.20517/ir.2025.41 ↗
9. Li, J., Li, D., Xiong, C., and Hoi, S. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv.
10. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.324 ↗
11. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv.
12. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., and Yang, Y. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv.
13. Zhang, Y., Xiang, T., Hospedales, T.M., and Lu, H. (2018, January 18–22). Deep Mutual Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00454 ↗
14. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018, January 18–22). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00286 ↗
15. Seo, H., and Choi, Y.S. (2025). V-PRUNE: Semantic-Aware Patch Pruning Before Tokenization in Vision-Language Model Inference. Appl. Sci., 15.10.3390/app15179463 ↗
16. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
17. Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., and Wei, X. (2023). MobileVLM: A Fast, Strong and Open Vision Language Assistant for Mobile Devices. arXiv.
18. Liu, X., Zhang, Y., Wang, Y., and Wang, Y. (2024). Aligned Vector Quantization for Edge-Cloud Collaborative Vision-Language Models. arXiv.
19. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Virtual.
20. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023, January 10–16). QLoRA: Efficient Finetuning of Quantized LLMs. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
21. A survey on semi-supervised learning10.1007/s10994-019-05855-6 ↗
22. Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., and Li, C.L. (2020, January 6–12). FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
23. Xie, Q., Dai, Z., Hovy, E., Luong, T., and Le, Q. (2020, January 6–12). Unsupervised Data Augmentation for Consistency Training. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
24. Tarvainen, A., and Valpola, H. (2017, January 4–9). Mean Teachers Are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
25. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13–19). Momentum Contrast for Unsupervised Visual Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
26. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning (ICML), Virtual.
27. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (December, January 28). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
28. Wang, X., Wei, J., Schuurmans, D., Wu, Q., Ma, T., and Le, Q. (2023, January 1–5). Self-Consistency Improves Chain of Thought Reasoning in Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
29. Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. (November, January 27). Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
30. Gal, Y., and Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv.
31. Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. arXiv.
32. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017). On Calibration of Modern Neural Networks. arXiv.
33. Shrivastava, A., Gupta, A., and Girshick, R. (2016). Training Region-based Object Detectors with Online Hard Example Mining. arXiv.10.1109/CVPR.2016.89 ↗
34. Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. (2024, January 16–22). Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01316 ↗
35. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR, Virtual.
36. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., and Song, X. (2024, January 10–15). CogVLM: Visual Expert for Pretrained Language Models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
37. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., and Reynolds, M. (December, January 28). Flamingo: A Visual Language Model for Few-Shot Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
38. Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. (2024, January 21–27). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR, Vienna, Austria.
39. Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. (2024, January 16–22). OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01274 ↗
40. Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E. (2023). Woodpecker: Hallucination Correction for Multimodal Large Language Models. arXiv.10.1007/s11432-024-4251-x ↗
41. Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. (2024, January 7–11). Analyzing and Mitigating Object Hallucination in Large Vision-Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
42. Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.T., and Sun, M. (2024, January 16–22). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. Proceedings of the CVPR 2024, Seattle, WA, USA .10.1109/CVPR52733.2024.01310 ↗
43. Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., and He, C. (2023). Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization. arXiv.
44. Vo, A., Nguyen, K.-N., Taesiri, M.R., Dang, V.T., Nguyen, A.T., and Kim, D. (2025). Vision Language Models are Biased. arXiv.
45. Divergence Measures Based on the Shannon Entropy10.1109/18.61115 ↗
46. Mader, K. (2025, November 24). Satellite Images of Hurricane Damage. Available online: https://www.kaggle.com/datasets/kmader/satellite-images-of-hurricane-damage.
47. Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-based Image Description Evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
48. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2020, January 26–30). BERTScore: Evaluating Text Generation with BERT. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
49. Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. (2021, January 7–11). CLIPScore: A Reference-free Evaluation Metric for Image Captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2021.emnlp-main.595 ↗
50. Yuan, W., Pang, R.Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. (2024). Self-Rewarding Language Models. arXiv.
51. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., and Finn, C. (2023, January 10–16). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
52. Aaba, A. (2026, January 02). Wildfire Prediction Dataset (Satellite Images). Kaggle. Derived from Forest Fires—Open Government Portal, Government of Canada. Available online: https://www.kaggle.com/datasets/abdelghaniaaba/wildfire-prediction-dataset.