Action-JointsLLM: A Novel Transformer-Based Framework for Efficient Fine-Grained Action Detection
Source abstract
Abstract
Fine-grained action detection remains a challenging and actively studied problem. While previous methods predominantly rely on convolutional neural networks (CNNs), their limited modeling capacity and high computational cost restrict their effectiveness in fine-grained settings. To overcome these challenges, we propose Action-JointsLLM, a novel framework that leverages joints extracted from visual data and large language models (LLMs) for accurate and efficient fine-grained action detection. Specifically, we introduce a Joint2Text module that transforms sequences of human joints into natural language descriptions, enabling direct compatibility with LLMs. Additionally, we design an attn-pooling mechanism to identify the contributions of each joint and time step across the motion sequence, enabling more accurate action analysis. Our approach demonstrates strong generalization across different LLM backbones and achieves state-of-the-art performance on the PD-WALK dataset, using Parkinson’s disease detection as a representative fine-grained action recognition task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Multimodal Machine Learning Applications · Human Motion and Animation
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou 310027, China
Zhejiang Key Laboratory of Intelligent Sensing Technology and Advanced Medical Instrument, Hangzhou 310027, China
Funding
“Leading Goose + X” Science and Technology Program of Zhejiang Province of China
References (52)
Show source-provided references
1. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Li, F.-F. (2014). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2014.223 ↗
2. Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015). Beyond short snippets: Deep networks for video classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2015.7299101 ↗
3. Simonyan, K., and Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems, Curran Associates, Inc.
4. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016). Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2016.213 ↗
5. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, IEEE.10.1109/ICCV.2015.510 ↗
6. Carreira, J., and Zisserman, A. (2017). Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2017.502 ↗
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Curran Associates, Inc.
8. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
9. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV48922.2021.00676 ↗
10. Visual instruction tuning
11. Bordes, F., Pang, R.Y., Ajay, A., Li, A.C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., and Jayaraman, B. (2024). An introduction to vision-language modeling. arXiv.
12. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv.
13. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., and Tang, J. (2025). Qwen2.5-vl technical report. arXiv.
14. Yan, S., Xiong, Y., and Lin, D. (2018). Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press.10.1609/aaai.v32i1.12328 ↗
15. He, Y., Yang, T., Yang, C., and Zhou, H. (2022). Integrated Equipment for Parkinson’s Disease Early Detection Using Graph Convolution Network. Electronics, 11.10.3390/electronics11071154 ↗
16. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019). Actional-structural graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2019.00371 ↗
17. Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2017.143 ↗
18. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2019.00584 ↗
19. Language models are few-shot learners
20. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
21. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
22. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv.
23. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.
24. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The llama 3 herd of models. arXiv.
25. Team, Q. (2024). Qwen2 technical report. arXiv.
26. Meta, A. (2026, January 08). The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation. Available online: https://ai.meta.com/blog/llama-4-multimodal-intelligence/.
27. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., and Lv, C. (2025). Qwen3 technical report. arXiv.
28. Toma, A., Lawler, P.R., Ba, J., Krishnan, R.G., Rubin, B.B., and Wang, B. (2023). Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. arXiv.
29. Wang, G., Yang, G., Du, Z., Fan, L., and Li, X. (2023). ClinicalGPT: Large language models finetuned with diverse medical data and comprehensive evaluation. arXiv.
30. Yu, H., Cheng, T., Cheng, Y., and Feng, R. (2025). Finemedlm-o1: Enhancing the medical reasoning ability of llm from supervised fine-tuning to test-time training. arXiv.
31. Towards conversational diagnostic artificial intelligence10.1038/s41586-025-08866-7 ↗
32. Xia, H., Yang, Z., Wang, Y., Tracy, R., Zhao, Y., Huang, D., Chen, Z., Zhu, Y., Wang, Y.f., and Shen, W. (2024). Sportqa: A benchmark for sports understanding in large language models. arXiv.10.18653/v1/2024.naacl-long.283 ↗
33. Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., and Zhu, R. (2025). Video understanding with large language models: A survey. IEEE Trans. Circuits Syst. Video Technol., early access.10.1109/TCSVT.2025.3566695 ↗
34. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2023). Efficient streaming language models with attention sinks. arXiv.
35. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. Proceedings of the 40th International Conference on Machine Learning, JMLR.org.
36. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics.
37. Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.10.18653/v1/D19-1410 ↗
38. Wang, M., Xing, J., and Liu, Y. (2021). Actionclip: A new paradigm for video action recognition. arXiv.
39. Zhang, H., Li, X., and Bing, L. (2023). Video-llama: An instruction-tuned audio–visual language model for video understanding. arXiv.10.18653/v1/2023.emnlp-demo.49 ↗
40. Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., and Zhao, D. (2024). Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv.
41. Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., and Wang, Y. (2023). MotionBERT: A Unified Perspective on Learning Human Motion Representations. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51070.2023.01385 ↗
42. World Health Organization (2022). Parkinson disease: A public health approach. Technical Brief.
43. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft COCO: Common objects in context. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer. Proceedings, Part V 13.
44. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the International Conference on Learning Representations, Virtual.
45. Chen, F., Datta, G., Kundu, S., and Beerel, P.A. (2023). Self-Attentive Pooling for Efficient Deep Learning. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), IEEE.10.1109/WACV56688.2023.00396 ↗
46. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016). NTU RGB+D: A large scale dataset for 3D human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2016.115 ↗
47. Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. (2023). Token Merging: Your ViT But Faster. arXiv.
48. Girdhar, R., and Ramanan, D. (2017). Attentional pooling for action recognition. Advances in Neural Information Processing Systems, Curran Associates, Inc.
49. Tokenlearner: Adaptive space-time tokenization for videos
50. Dynamicvit: Efficient vision transformers with dynamic token sparsification
51. Kim, K., Lyu, S., Mantri, S., and Dunn, T.W. TULIP: Multi-camera 3D Precision Assessment of Parkinson’s Disease. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024, IEEE.10.1109/CVPR52733.2024.02128 ↗
52. Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The kinetics human action video dataset. arXiv.