Automatic Identification and Description of Jewelry Through Computer Vision and Neural Networks for Translators and Interpreters
Source abstract
Abstract
Identifying jewelry pieces presents a significant challenge due to the wide range of styles and designs. Currently, precise descriptions are typically limited to industry experts. However, translators and interpreters often require a comprehensive understanding of these items. In this study, we introduce an innovative approach to automatically identify and describe jewelry using neural networks. This method enables translators and interpreters to quickly access accurate information, aiding in resolving queries and gaining essential knowledge about jewelry. Our model operates at three distinct levels of description, employing computer vision techniques and image captioning to emulate expert analysis of accessories. The key innovation involves generating natural language descriptions of jewelry across three hierarchical levels, capturing nuanced details of each piece. Different image captioning architectures are utilized to detect jewels in images and generate descriptions with varying levels of detail. To demonstrate the effectiveness of our approach in recognizing diverse types of jewelry, we assembled a comprehensive database of accessory images. The evaluation process involved comparing various image captioning architectures, focusing particularly on the encoder–decoder model, crucial for generating descriptive captions. After thorough evaluation, our final model achieved a captioning accuracy exceeding 90%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Human Pose and Action Recognition · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, University of Córdoba, 14071 Córdoba, Spain
Economics, Engineering, Society and Business Organization, University of Tuscia, 01100 Viterbo, Italy
Department of Social Sciences, Translation and Interpreting, University of Córdoba, 14071 Córdoba, Spain
Funding
Regional Ministry of Economy, Knowledge, Enterprises, and University, Government of Andalusia (Spain)
References (44)
Show source-provided references
1. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
2. From Show to Tell: A Survey on Deep Learning-Based Image Captioning10.1109/TPAMI.2022.3148210 ↗
3. Deep Hierarchical Encoder–Decoder Network for Image Captioning10.1109/TMM.2019.2915033 ↗
4. Jackendoff, R. (2002). Foundations of Language: Brain, Meaning, Grammar, Evolution, Oxford University Press.10.1093/acprof:oso/9780198270126.001.0001 ↗
5. Linear grammar as a possible stepping-stone in the evolution of language10.3758/s13423-016-1073-y ↗
6. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y. (2015, January 7–9). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on International Conference on Machine Learning, Lille, France.
7. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
8. Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., and McClosky, D. (2014, January 1). The Stanford CoreNLP Natural Language Processing Toolkit. Proceedings of the Stanford CoreNLP Natural Language Processing Toolkit, Baltimore, MA, USA.10.3115/v1/P14-5010 ↗
9. Bar-Hillel, Y. (1964). Language and Information, Addison-Wesley Pub. Co.
10. Aspects of Language
11. McShane, M., and Nirenburg, S. (2021). Linguistics for the Age of AI, MIT Press.10.7551/mitpress/13618.001.0001 ↗
12. Li, C., Kang, Q., Ge, G., Song, Q., Lu, H., and Cheng, J. (July, January 26). DeepBE: Learning Deep Binary Encoding for Multi-label Classification. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA.10.1109/CVPRW.2016.98 ↗
13. The Specular Exponent as a Criterion for Appearance Quality Assessment of Pearllike Objects by Artificial Vision10.1109/TIE.2011.2146213 ↗
14. Alcalde-Llergo, J.M., Yeguas-Bolívar, E., Zingoni, A., and Fuerte-Jurado, A. (2023, January 25–27). Jewelry Recognition via Encoder-Decoder Models. Proceedings of the 2023 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE), Milano, Italy.10.1109/MetroXRAINE58569.2023.10405609 ↗
15. Yu, J., Li, H., Hao, Y., Zhu, B., Xu, T., and He, X. (November, January 29). CgT-GAN: CLIP-guided Text GAN for Image Captioning. Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, Ottawa, ON, Canada.
16. Shetty, R., Rohrbach, M., Hendricks, L.A., Fritz, M., and Schiele, B. (2017, January 22–29). Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.445 ↗
17. Chaffin, A., Kijak, E., and Claveau, V. (2024). Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning. arXiv.10.1109/ICIP51287.2024.10647426 ↗
18. Shi, H., Li, P., Wang, B., and Wang, Z. (2018, January 17–19). Image captioning based on deep reinforcement learning. Proceedings of the 10th International Conference on Internet Multimedia Computing and Service, ICIMCS’18, Nanjing, China.10.1145/3240876.3240900 ↗
19. Deep Learning Approaches on Image Captioning: A Review10.1145/3617592 ↗
20. Towards generating textual summaries of graphs
21. Grice, H.P. (1975). Logic and Conversation, Academic Press.10.1163/9789004368811_003 ↗
22. Dijk, T.V. (1977). Semantic macro-structures and knowledge frames in discourse comprehension. Cognitive Processes in Comprehension, Psychology Press.
23. Corpora and theories of linguistic performance
24. Levels of description and explanation in meaning construction10.1075/slcs.107.08lev ↗
25. Semantic macro-structures and macro-rules in visual discourse processing Semantic macro-structures and macro-rules in visual discourse processing
26. Guyon, I., Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017), Processing of the 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 4–9 December 2017, Curran Associates, Inc.
27. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3–10). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, New York, NY, USA.10.1145/3442188.3445922 ↗
28. Kazakh jewelry: Problems of translation and creation of a multilingual thesaurus10.32523/2616-678X-2021-135-2-55-63 ↗
29. Targeting Ion Beam Analysis techniques for gold artefacts10.4000/archeosciences.1955 ↗
30. Research on innovative application of silver material in modern jewelry design10.1051/matecconf/201817602013 ↗
31. Ariza-Montes, A. (2006). Plan Estratégico Del Sistema Productivo Local de la Joyería de Córdoba, Universidad de Loyola.
32. Arellano, N., Espinoza, A., Perdomo, D., and Carpio, L. (2025, May 12). Joyería Marthita. DSpace en ESPOL Unidades Académicas Facultad de Ciencias Sociales y Humanísticas. Artículos de Tesis de Grado. Available online: https://www.dspace.espol.edu.ec/handle/123456789/1361?locale=en.
33. Competitividad entre joyerías y tiendas de bisutería fina legalmente constituidas, 2013
34. The effect of celebrity endorsement on consumer behavior: Case of the Lebanese jewelry industry10.1016/j.aebj.2018.11.002 ↗
35. Alvarado Hidalgo, C.A. (2025, May 12). El marketing como factor relevante en la gestión de calidad y plan de mejora en las micro y pequeñas empresas del sector servicio-rubro joyería y perfumería, de la ciudad de Chimbote, 2019. Repositorio Institucional de ULADECH Católica. Trabajo de Investigación. Available online: https://repositorio.uladech.edu.pe/handle/20.500.13032/24863.
36. Liddicoat, A.J., and Curnow, T.J. (2004). Language Descriptions, Wiley.10.1002/9780470757000.ch1 ↗
37. Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. (2019). A Comprehensive Survey on Transfer Learning. arXiv.
38. Wang, Y., Xu, J., and Sun, Y. (March, January 22). End-to-End Transformer Based Model for Image Captioning. Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence. Assoc Advancement Artificial Intelligence, Virtual Event.
39. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
40. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (June, January 27). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
41. Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Int. J. Intell. Sci.
42. Long Short-term Memory10.1162/neco.1997.9.8.1735 ↗
43. Cho, K., Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. ACL Anthol., 103–111.10.3115/v1/W14-4012 ↗
44. A thorough review of models, evaluation metrics, and datasets on image captioning