Isolated German Sign Language Recognition for Classifying Polar Answers Using Landmarks and Lightweight Transformers
Source abstract
Abstract
Sign Languages are the primary communication modality of deaf communities, yet building effective Isolated Sign Language Recognition (ISLR) systems remains difficult under data limitations. In this work, we curated a sub-dataset from the DGS-Korpus focused on recognizing affirmations and negations (polar answers) in German Sign Language (DGS). We designed lightweight transformer models using landmark-based inputs and evaluated them on two tasks: the binary classification of affirmations versus negations (binary semantic recognition) and the multi-class recognition of sign variations expressing positive or negative replies (multi-class gloss recognition). The main contribution of the article, hence, relies on the exploration of models for performing polar answer recognition in DGS and the exploration of differences between performing multi-class or binary class classification. Our best binary model achieved an accuracy of 97.71% using only hand landmarks without Positional Encoding, highlighting the potential of lightweight landmark-based transformers for efficient ISLR in constrained domains.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Hearing Impairment and Communication · Interactive and Immersive Displays
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Chair for Human-Centered Artificial Intelligence, Faculty of Applied Computer Science, University of Augsburg, 86159 Augsburg, Germany
Grupo de Tecnología del Habla y Aprendizaje Automático (THAU Group), Information Processing and Telecommunications Center, E.T.S.I. de Telecomunicación, Universidad Politécnica de Madrid (UPM), 28040 Madrid, Spain
Funding
Bundesministerium für Bildung und Forschung · Spanish Ministry of Education · European Commission · Ministerio de Ciencia, Innovación y Universidades
References (40)
Show source-provided references
1. World Health Organization (2021). World Report on Hearing, World Health Organization.
2. Murray, J. (2013). Deaf Women and Girls Not Able to Enjoy Basic Human Rights, World Federation of the Deaf.
3. Nunnari, F., Avramidis, E., España-Bonet, C., González, M., Hennes, A., and Gebhard, P. (2024, January 20–25). DGS-Fabeln-1: A Multi-Angle Parallel Corpus of Fairy Tales between German Sign Language and German Text. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Turin, Italy.
4. Zhou, H., Zhou, W., Qi, W., Pu, J., and Li, H. (2021, January 20–25). Improving sign language translation with monolingual data by sign back-translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00137 ↗
5. Li, D., Rodriguez, C., Yu, X., and Li, H. (2020, January 1–5). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass, CO, USA.10.1109/WACV45572.2020.9093512 ↗
6. Joze, H.R.V., and Koller, O. (2018). MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. arXiv.
7. Duarte, A.C., Palaskar, S., Ghadiyaram, D., DeHaan, K., Metze, F., Torres, J., and Giró-i-Nieto, X. (2021, January 20–25). How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00276 ↗
8. Sign language transcription conventions for the ECHO project
9. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers10.1016/j.cviu.2015.09.013 ↗
10. Camgöz, N.C., Hadfield, S., Koller, O., Ney, H., and Bowden, R. (2018, January 18–23). Neural Sign Language Translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00812 ↗
11. von Agris, U., and Kraiss, K.F. (2010, January 22–23). Signum database: Video corpus for signer-independent continuous sign language recognition. Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Valletta, Malta.
12. Bernhard, L., Nunnari, F., Unger, A., Bauerdiek, J., Dold, C., Hauck, M., Stricker, A., Baur, T., Heimerl, A., and André, E. (July, January 29). Towards automated sign language production: A pipeline for creating inclusive virtual humans. Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments, Corfu, Greece.
13. Nunnari, F., Bauerdiek, J., Bernhard, L., Espana-Bonet, C., Jäger, C., Unger, A., Waldow, K., Wecker, S., André, E., and Busemann, S. (2021, January 20). AVASAG: A German Sign Language Translation System for Public Services. Proceedings of the Machine Translation Summit XVIII, Digital.
14. Prillwitz, S., Hanke, T., König, S., Konrad, R., Langer, G., and Schwarz, A. (June, January 26). DGS corpus project–development of a corpus based electronic dictionary German Sign Language/German. Proceedings of the Sign-Lang@ LREC 2008, Marrakech, Morocco.
15. Konrad, R., Hanke, T., Langer, G., Blanck, D., Bleicken, J., Hofmann, I., Jeziorski, O., König, L., König, S., and Nishio, R. (2020). MEINE DGS–Annotiert. Öffentliches Korpus der Deutschen Gebärdensprache, 3. Release/MY DGS–Annotated. Public Corpus of German Sign Language, 3rd Release, Universität Hamburg.
16. Konrad, R., Hanke, T., Langer, G., König, S., König, L., Nishio, R., and Regen, A. (2022). Public DGS Corpus: Annotation Conventions/Öffentliches DGS-Korpus: Annotationskonventionen, Universität Hamburg.
17. Bauer, A., Kuder, A., Schulder, M., and Schepens, J. (2024). Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study. PLoS ONE, 19.10.1371/journal.pone.0304040 ↗
18. Nam Pham, D., and Avramidis, E. (2025, January 16). The Importance of Facial Features in Vision-based Sign Language Recognition: Eyes, Mouth or Full Face?. Proceedings of the 9th Workshop on Sign Language Translation and Avatar Technologies (SLTAT2025), Berlin, Germany.10.1145/3742886.3756718 ↗
19. Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. (2021, January 10–17). Multiscale vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00675 ↗
20. Sarhan, N., and Frintrop, S. (2023, January 2–6). Unraveling a decade: A comprehensive survey on isolated sign language recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCVW60793.2023.00345 ↗
21. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
22. Albanie, S., Varol, G., Momeni, L., Afouras, T., Chung, J.S., Fox, N., and Zisserman, A. (2020, January 23–28). BSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing Cues. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58621-8_3 ↗
23. Alishzade, N., and Abdullayeva, G. (2024, January 19–21). [Case Study] Transfer Learning with Inflated 3D CNN for Word-Level Recognition for Azerbaijani Sign Language Dataset. Proceedings of the 10 th World Congress on Electrical Engineering and Computer Systems and Sciences (EECSS’24), Barcelona, Spain.10.11159/mvml24.106 ↗
24. Sarhan, N., and Frintrop, S. (2020, January 25–28). Transfer learning for videos: From action recognition to sign language recognition. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP40778.2020.9191289 ↗
25. Huang, J., and Chouvatut, V. (2024). Video-based sign language recognition via resnet and lstm network. J. Imaging, 10.10.20944/preprints202405.1851.v1 ↗
26. Ronchetti, F., Quiroga, F.M., Estrebou, C., Lanzarini, L., and Rosete, A. (2023). LSA64: An Argentinian sign language dataset. arXiv.
27. Xu, Y., Zhang, J., Zhang, Q., and Tao, D. (December, January 28). ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
28. ViTPose++: Vision Transformer Foundation Model for Generic Body Pose Estimation10.1109/TPAMI.2023.3330016 ↗
29. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M., and Lee, J. (2019, January 17). MediaPipe: A Framework for Perceiving and Processing Reality. Proceedings of the Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019, Long Beach, CA, USA.
30. Varghese, R., and Sambath, M. (2024, January 18–19). YOLOv8: A Novel Object Detection Algorithm with Enhanced Performance and Robustness. Proceedings of the 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India.10.1109/ADICS58448.2024.10533619 ↗
31. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields10.1109/TPAMI.2019.2929257 ↗
32. Ludwig, K., Oksymets, Y., Schön, R., Kienzle, D., and Lienhart, R. (2025, January 11–15). Efficient 2D to full 3D human pose uplifting including joint rotations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2025, Nashville, TN, USA.10.1109/CVPRW67362.2025.00583 ↗
33. Sandoval-Castaneda, M., Li, Y., Brentari, D., Livescu, K., and Shakhnarovich, G. (2023). Self-Supervised Video Transformers for Isolated Sign Language Recognition. arXiv.
34. Luna-Jiménez, C., Gil-Martín, M., Kleinlein, R., San-Segundo, R., and Fernández-Martínez, F. (2023;, January 9–13). Interpreting Sign Language Recognition using Transformers and MediaPipe Landmarks. Proceedings of the 25th International Conference on Multimodal Interaction, Paris, France.10.1145/3577190.3614143 ↗
35. Buribayev, Z., Aouani, M., Zhangabay, Z., Yerkos, A., Abdirazak, Z., and Zhassuzak, M. (2025). Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow. Appl. Sci., 15.10.3390/app15105685 ↗
36. Boháček, M., and Hrúz, M. (2022, January 4–8). Sign Pose-Based Transformer for Word-Level Sign Language Recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, Waikoloa, HI, USA.10.1109/WACVW54805.2022.00024 ↗
37. Luna-Jiménez, C., Eing, L., Aicher, A., Nunnari, F., and André, E. (2025, January 13–17). Lightweight Transformers for Isolated Sign Language Recognition. Proceedings of the 27th International Conference on Multimodal Interaction, Canberra, Australia. ICMI ’25.10.1145/3716553.3750772 ↗
38. Pu, M., Lim, M.K., and Chong, C.Y. (November, January 28). Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, VIC, Australia. MM ’24.
39. Ma, J., Ma, L., Ruan, W., Chen, H., and Feng, J. (2022, January 24–26). A Wushu Posture Recognition System Based on MediaPipe. Proceedings of the 2022 2nd International Conference on Information Technology and Contemporary Sports (TCS), Guangzhou, China.10.1109/TCS56119.2022.9918744 ↗
40. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA. NIPS’17.