CATS: cultural-heritage classification using LLMs and distribute model
Source abstract
Abstract
In this paper, we introduce CATS (Cultural-heritage Advanced Translation Systems) , a generative model applied to creating images for classification and exploring relationships among cultural heritage. We aimed to address the issue where large language models (LLMs) fail to generate appropriate sentences due to the limited training on classical Korean language, and the problem where text-to-image models trained on Korean language do not produce accurate sentences when using Korean words as they are. To solve this problem, a large language model was used to translate historical content containing classical Korean words into English sentences, which were then used as input for the text-to-image generation model. We found that the generation model using the translated English text produced more accurate and consistent images compared to the model using the original Korean text. Consequently, this approach offers highly convenient visual information for users and administrators at a low cost through the use of open-source models. Therefore, we propose the potential of a system that leverages generated images to facilitate the search and extraction of relevant information.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Image Retrieval and Classification Techniques · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (56)
Show source-provided references
1. Pierdicca, R. et al. Point cloud semantic segmentation using a deep learning framework for cultural heritage. Remote Sens. 12.6, 1005 (2020).10.3390/rs12061005 ↗
2. Podell, D. et al. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv Prepr. arXiv 2307, 01952 (2023).
3. Sun, W. & Wang, R. Fully convolutional networks for semantic segmentation of very high resolution remotely sensed images combined with DSM. IEEE Geosci. Remote Sens. Lett. 15, 474–478 (2018).10.1109/LGRS.2018.2795531 ↗
4. Guo, Y. et al. Deep learning for 3d point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 4338–4364 (2020).10.1109/TPAMI.2020.3005434 ↗
5. Tabib, R. A. et al. DeFi: detection and filling of holes in point clouds towards restoration of digitized cultural heritage models. Proceedings of the IEEE/CVF International Conference on Computer Vision. (2023).10.1109/ICCVW60793.2023.00175 ↗
6. Hu, Q. et al. ConvSRGAN: super-resolution inpainting of traditional Chinese paintings Heritage. Science 12, 176 (2024).
7. Gunawan, A. et al. Modernizing old photos using multiple references via photorealistic style transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2023).10.1109/CVPR52729.2023.01199 ↗
8. Zhang, J. et al. “ArchGPT: harnessing large language models for supporting renovation and conservation of traditional architectural heritage.”. Herit. Sci. 12.1, 220 (2024).10.1186/s40494-024-01334-x ↗
9. Rombach, R. et al. High-resolution image synthesis with latent diffusion models.“ Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2022).10.1109/CVPR52688.2022.01042 ↗
10. Devlin, J. et al. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers) (2019).
11. Sinha, K. et al. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arXiv Prepr. arXiv 2104, 06644 (2021).
12. Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach. arXiv Prepr. arXiv 1907, 11692 (2019).
13. Sanh, V. et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv Prepr. arXiv 1910, 01108 (2019).
14. Lan, Z. et al. Albert: A lite bert for self-supervised learning of language representations. arXiv Prepr. arXiv 1909, 11942 (2019).
15. Clark, K. et al. Electra: Pre-training text encoders as discriminators rather than generators. arXiv Prepr. arXiv 2003, 10555 (2020).
16. He, P. et al. Deberta: Decoding-enhanced bert with disentangled attention. arXiv Prepr. arXiv 2006, 03654 (2020).
17. Sarlin, P.-E. et al. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2020).10.1109/CVPR42600.2020.00499 ↗
18. Alec Radford, J. W. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
19. Keskar, N. S. et al. Ctrl: A conditional transformer language model for controllable generation. arXiv Prepr. arXiv 1909, 05858 (2019).
20. Ben Mann, N. et al. Language models are few-shot learners. arXiv Prepr. arXiv 2005, 14165 (2020).
21. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67 (2020).
22. Qiu, X. et al. Pre-trained models for natural language processing: A survey. Sci. China Technol. Sci. 63, 1872–1897 (2020).10.1007/s11431-020-1647-3 ↗
23. Dong, C. et al. A survey of natural language generation. ACM Comput. Surv. 55, 1–38 (2022).10.1145/3554727 ↗
24. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv Prepr. arXiv 2302, 13971 (2023).
25. Su, J. et al. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568, 127063 (2024).10.1016/j.neucom.2023.127063 ↗
26. Taori, R. et al. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford.edu/2023/03/13/alpaca.html 3.6: 7 (2023)
27. Wang, Y. et al. Self-instruct: Aligning language models with self-generated instructions. arXiv Prepr. arXiv 2212, 10560 (2022).
28. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. arXiv Prepr. arXiv 1312, 6114 (2013).
29. Baldi, P. Autoencoders, unsupervised learning, and deep architectures. Proceedings of ICML workshop on unsupervised and transfer learning. JMLR Workshop and Conference Proceedings, (2012).
30. Goodfellow, I. et al. Generative adversarial networks. Commun. ACM 63, 139–144 (2020).10.1145/3422622 ↗
31. Gao, H., Jian, P. & Huang, H. Progan: Network embedding via proximity generative adversarial network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. (2019).10.1145/3292500.3330866 ↗
32. Karras, T., Laine, S. & T. A. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2019).10.1109/CVPR.2019.00453 ↗
33. Karras, T. et al. Analyzing and improving the image quality of stylegan. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2020).10.1109/CVPR42600.2020.00813 ↗
34. Huang, X. & Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE international conference on computer vision. (2017).10.1109/ICCV.2017.167 ↗
35. Zhang, H. et al. Self-attention generative adversarial networks. International conference on machine learning. (PMLR, 2019).
36. Brock, A., Donahue, J. & Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. arXiv Prepr. arXiv 1809, 11096 (2018).
37. Esser, P., Rombach, R. & Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2021).10.1109/CVPR46437.2021.01268 ↗
38. Yu, J. et al. Vector-quantized image modeling with improved vqgan. arXiv Prepr. arXiv 2110, 04627 (2021).
39. Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. Adv. neural Inf. Process. Syst. 33, 6840–6851 (2020).
40. Song, J., Meng, C. & Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020).
41. Ramesh, A. et al. Zero-shot text-to-image generation. International conference on machine learning. (Pmlr, 2021).
42. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C. & Chen, M. Hierarchical text-conditional image generation with CLIP latents. Preprint at http://arxiv.org/abs/2204.06125 (2022).
43. Radford, A. et al. Learning transferable visual models from natural language supervision. International conference on machine learning. (PMLR, 2021).
44. Alayrac, J.-B. et al. Flamingo: a visual language model for few-shot learningAdv. neural Inf. Process. Syst. 35, 23716–23736 (2022).
45. Xu, U. et al. Vlm: Task-agnostic video-language model pre-training for video understanding. arXiv Prepr. arXiv 2105, 09996 (2021).
46. Zhang, L., Rao, A. & Agrawala, M. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision. (2023).10.1109/ICCV51070.2023.00355 ↗
47. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. (Springer International Publishing, 2015).
48. Prompthero. Openjourney. Hugging Face, n.d. Web. 12 July 2024. https://huggingface.co/prompthero/openjourney.
49. Saharia, C. et al. Photorealistic text-to-image diffusion models with deep language understanding. Adv. neural Inf. Process. Syst. 35, 36479–36494 (2022).
50. Hertz, A. et al. Prompt-to-prompt image editing with cross attention control. arXiv Prepr. arXiv 2208, 01626 (2022).
51. Kim, B.-K. et al. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. Workshop on Efficient Systems for Foundation Models@ ICML2023. (2023).
52. Lin, T.-Y. et al. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V 13, pages 740–755 (Springer, 2014).10.1007/978-3-319-10602-1_48 ↗
53. Xuelin Z., Jiawei G., Jiawei, C., Weijia, L. & Liu, B. Two-stream transformer for multi-label image classification. In Proceedings of the 30th ACM International Conference on Multimedia. 3598–3607 (2022).10.1145/3503161.3548343 ↗
54. Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. arXiv Prepr. arXiv 1711, 05101 (2017).
55. Zhang, Z. & Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, (2018).
56. National museum of korea e-museum. https://www.emuseum.go.kr/main.