OBIFlo-SAM: multi-task semantic recognition and segmentation of oracle bone inscription
Source abstract
Abstract
Oracle Bone Inscriptions (OBI), the earliest mature Chinese writing system, exhibit highly abstract pictographic structures and intricate semantic mappings, posing formidable challenges for automated recognition and interpretation. This study introduces OBIFlo-SAM, an efficient, lightweight, and robust cross-modal multi-task framework integrating the pre-trained Florence-2 model with the Segment Anything Model (SAM). Employing a weighted Focal Loss fine-tuning strategy, the framework achieves high-precision semantic generation, text alignment, object detection, and zero-shot segmentation, addressing low-resource scenarios, class imbalance, and complex backgrounds. The OracleVQA dataset, combining multimodal question-answer pairs with detection annotations, provides high-quality training data. Experimental results demonstrate that OBIFlo-SAM significantly outperforms CLIP-based and general-purpose large language models in semantic recognition while excelling in detection and segmentation. This work establishes an intelligent pathway for digital preservation and automated interpretation of ancient scripts, providing foundation for cultural heritage conservation and scholarly research.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Image Processing and 3D Reconstruction · Handwritten Text Recognition Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (37)
Show source-provided references
1. Guo, Z. et al. An improved neural network model based on inception-v3 for oracle bone inscription character recognition. Sci. Program. 2022, 7490363 (2022).
2. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2818–2826 (IEEE, 2016).10.1109/CVPR.2016.308 ↗
3. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS, 2012).
4. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at https://arxiv.org/abs/1409.1556 (2014).
5. Du, B., Liu, G. & Ge, W. Deep self-supervised learning for oracle bone inscriptions features representation. In 2021 IEEE 4th International Conference on Information Systems and Computer Aided Education (ICISCAE), 7–11 (IEEE, 2021).10.1109/ICISCAE52414.2021.9590642 ↗
6. Sun, W. et al. Dual-view oracle bone script recognition system via temporal-spatial psychovisual modulation. In 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 193–198 (IEEE, 2020).10.1109/MIPR49039.2020.00047 ↗
7. Wang, M., Deng, W. & Liu, C.-L. Unsupervised structure-texture separation network for oracle character recognition. IEEE Trans. Image Process. 31, 3137–3150 (2022).10.1109/TIP.2022.3165989 ↗
8. Liu, Z. et al. Oracle character detection based on improved faster r-cnn. In 2021 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), 697–700 (IEEE, 2021).10.1109/ICITBS53129.2021.00175 ↗
9. Meng, L. et al. Oracle bone inscription detector based on ssd. In New Trends in Image Analysis and Processing–ICIAP 2019: ICIAP International Workshops, BioFor, PatReCH, e-BADLE, DeepRetail, and Industrial Session, Revised Selected Papers 20, 126–136 (Springer, 2019).10.1007/978-3-030-30754-7_13 ↗
10. Wang, N., Sun, Q., Jiao, Q. & Ma, J. Oracle bone inscriptions detection in rubbings based on deep learning. In 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), 9, 1671–1674 (IEEE, 2020).10.1109/ITAIC49862.2020.9339132 ↗
11. Gomaa, A. Advanced domain adaptation technique for object detection leveraging semi-automated dataset construction and enhanced yolov8. In 2024 6th novel intelligent and leading emerging sciences conference (NILES), 211–214 (IEEE, 2024).10.1109/NILES63360.2024.10753164 ↗
12. Gomaa, A. & Abdalrazik, A. Novel deep learning domain adaptation approach for object detection using semi-self building dataset and modified yolov4. World Electr. Veh. J. 15, 255 (2024).10.3390/wevj15060255 ↗
13. Gomaa, A. & Saad, O. M. Residual channel-attention (rca) network for remote sensing image scene classification. Multimedia Tools Appl. 84, 33837–33861 (2025).10.1007/s11042-024-20546-8 ↗
14. Gomaa, A., Abdelwahab, M. M. & Abo-Zahhad, M. Efficient vehicle detection and tracking strategy in aerial videos by employing morphological operations and feature points motion analysis. Multimed. Tools Appl. 79, 26023–26043 (2020).10.1007/s11042-020-09242-5 ↗
15. Gomaa, A., Abdelwahab, M. M. & Abo-Zahhad, M. Real-time algorithm for simultaneous vehicle detection and tracking in aerial view videos. In 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS), 222–225 (IEEE, 2018).10.1109/MWSCAS.2018.8624022 ↗
16. Hassan, O. F., Ibrahim, A. F., Gomaa, A., Makhlouf, M. & Hafiz, B. Real-time driver drowsiness detection using transformer architectures: a novel deep learning approach. Sci. Rep. 15, 17493 (2025).10.1038/s41598-025-02111-x ↗
17. Li, K., Batjargal, B. & Maeda, A. A prototypical network-based approach for low-resource font typeface feature extraction and utilization. Data 6, 134 (2021).10.3390/data6120134 ↗
18. Wang, M. & Deng, W. A dataset of oracle characters for benchmarking machine learning algorithms. Sci. Data 11, 87 (2024).10.1038/s41597-024-02933-w ↗
19. Xiao, B. et al. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4818–4829 (IEEE, 2024).10.1109/CVPR52733.2024.00461 ↗
20. Kirillov, A. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015–4026 (IEEE, 2023).
21. Khan, M. T. et al. Fine-tuning vision-language model for automated engineering drawing information extraction. Preprint at https://arxiv.org/abs/2411.03707 (2024).
22. Ro, S., Satwika, S., Gayathri, P. Y., Balsha, M. G. & Ucar, A. Fine-tuning florence2 for enhanced object detection in un-constructed environments: Vision-language model approach. Preprint at https://arxiv.org/abs/2503.04918 (2025).
23. Réby, K., Guilhelm, A. & De Luca, L. Semantic segmentation using foundation models for cultural heritage: An experimental study on notre-dame de paris. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1689–1697 (IEEE, 2023).10.1109/ICCVW60793.2023.00184 ↗
24. WU, G., WANG, Y., LIU, L. & YANG, P. Digital rubbing technology for cultural relics: Integrating image segmentation and style transfer. J. Southwest Univ. Nat. Sci. Ed. 47, 224–235 (2025).
25. Wang, M. & Deng, W. Oracle-mnist: a dataset of oracle characters for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/2205.09442 (2022).
26. Wang, P. et al. An open dataset for oracle bone character recognition and decipherment. Sci. Data 11, 976 (2024).10.1038/s41597-024-03807-x ↗
27. Guan, H. et al. An open dataset for the evolution of oracle bone characters: Evobc. Preprint at https://arxiv.org/abs/2401.12467 (2024).
28. Li, B. et al. Oracle bone inscriptions multi-modal dataset. Preprint at https://arxiv.org/abs/2407.03900 (2024).
29. Guo, J. et al. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10867–10877 (IEEE, 2023).10.1109/CVPR52729.2023.01046 ↗
30. Ding, M. et al. Davit: Dual attention vision transformers. In European Conference on Computer Vision, 74–92 (Springer, 2022).10.1007/978-3-031-20053-3_5 ↗
31. Li, J., Chen, Z., Chen, T., Liu, Z. & Wang, C. Obiformer: A fast attentive denoising framework for oracle bone inscriptions. Displays 89, 103059 (2025).10.1016/j.displa.2025.103059 ↗
32. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2223–2232 (IEEE, 2017).10.1109/ICCV.2017.244 ↗
33. Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PMLR, 2021).
34. Team, G. et al. Gemini: a family of highly capable multimodal models. Preprint at https://arxiv.org/abs/2312.11805 (2023).
35. Achiam, J. et al. Gpt-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).
36. Guan, H. et al. Deciphering oracle bone language with diffusion models. Preprint at https://arxiv.org/abs/2406.00684 (2024).
37. Yang, A. et al. Chinese clip: Contrastive vision-language pretraining in chinese. Preprint at https://arxiv.org/abs/2211.01335 (2022).