journal article·2025-11-28Open access

Research on an Automatic Classification Method for Art Film Scenes Based on Image and Audio Deep Features

Zhaojun An, Heinz D. Fill

Applied Sciences· Volume 15· Issue 23

Abstract

This paper addresses the challenging task of automatic scene classification in art films, a genre characterized by symbolic visuals, asynchronous audio, and non-linear storytelling. We propose Styloformer, a multimodal transformer architecture designed to integrate visual, auditory, textual, and curatorial signals into a unified representation space. The model combines cross-modal attention, stylistic clustering, influence prediction, and canonicality estimation to handle the semantic and historical complexity of art cinema. Additionally, we introduce a novel module called Historiographic Navigation, which embeds ontological priors and temporal logic to support interpretive reasoning. Evaluated on multiple benchmarks, Styloformer achieves state-of-the-art performance, including 91.85% accuracy and 94.31% AUC on the MovieNet dataset—outperforming baselines such as CLIP and ViT. Ablation studies further demonstrate the importance of each architectural component. Unlike general-purpose video models, our system is tailored to the aesthetic and narrative structure of art films, making it suitable for applications in digital curation and computational film analysis. Styloformer represents a scalable and interpretable approach to understanding artistic media, bridging machine learning with art historical reasoning.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Aesthetic Perception and Analysis · Generative Adversarial Networks and Image Synthesis · Music and Audio Processing

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Faculty of Philosophy and Letters, University of Buenos Aires, Buenos Aires 102957, Argentina

School of Computer Science, Cornell University, Ithaca, NY 14853, USA

References (35)

Show source-provided references
1. Artificial neural networks and deep learning in the visual arts: A review10.1007/s00521-020-05565-4 ↗
2. Spectral graph wavelet transform-based feature representation for automated classification of emotions from EEG signal10.1109/JSEN.2023.3330090 ↗
3. A design of movie script generation based on natural language processing by optimized ensemble deep learning with heuristic algorithm10.1016/j.datak.2023.102150 ↗
4. Improving transfer learning for movie trailer genre classification using a dual image and video transformer10.1016/j.ipm.2023.103343 ↗
5. Video scenes segmentation based on multimodal genre prediction10.1016/j.procs.2020.08.002 ↗
6. State-of-the-art and future challenges in video scene detection: A survey10.1007/s00530-013-0306-4 ↗
7. Simões, G.S., Wehrmann, J., Barros, R.C., and Ruiz, D.D. (2016, January 24–29). Movie genre classification with convolutional neural networks. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.10.1109/IJCNN.2016.7727207 ↗
8. State of the “art”: A taxonomy of artistic stylization techniques for images and video10.1109/TVCG.2012.160 ↗
9. Shot scale distribution in art films10.1007/s11042-016-3339-9 ↗
10. Multimodal deep representation learning for video classification10.1007/s11280-018-0548-3 ↗
11. Representation learning for mammography mass lesion classification with convolutional neural networks10.1016/j.cmpb.2015.12.014 ↗
12. Ditsanthia, E., Pipanmaekaporn, L., and Kamonsantiroj, S. (2018, January 12–14). Video representation learning for cctv-based violence detection. Proceedings of the 2018 3rd Technology Innovation Management and Engineering Science International Conference (TIMES-iCON), Bangkok, Thailand.10.1109/TIMES-iCON.2018.8621751 ↗
13. CAM-RNN: Co-attention model based RNN for video captioning10.1109/TIP.2019.2916757 ↗
14. The co-attention model for tiny activity analysis10.1016/j.neucom.2012.06.040 ↗
15. Zhang, Y., Wang, Z.R., and Du, J. (2019, January 14–19). Deep fusion: An attention guided factorized bilinear pooling for audio-video emotion recognition. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.10.1109/IJCNN.2019.8851942 ↗
16. A survey of open-world person re-identification10.1109/TCSVT.2019.2898940 ↗
17. On the use of computable features for film classification10.1109/TCSVT.2004.839993 ↗
18. Taxonomy of directing semantics for film shot classification10.1109/TCSVT.2009.2022705 ↗
19. Kipp, M., and Martin, J.C. (2009, January 10–12). Gesture and emotion: Can basic gestural form features discriminate emotions?. Proceedings of the 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, The Netherlands.10.1109/ACII.2009.5349544 ↗
20. Extracting semantics from audio-visual content: The final frontier in multimedia retrieval10.1109/TNN.2002.1021881 ↗
21. Sun, J., Wu, X., Yan, S., Cheong, L.F., Chua, T.S., and Li, J. (2009, January 20–25). Hierarchical spatio-temporal context modeling for action recognition. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.
22. A generalized temporal context model for classifying image collections10.1007/s00530-005-0202-7 ↗
23. Multimodal video indexing: A review of the state-of-the-art10.1023/B:MTAP.0000046380.27575.a5 ↗
24. Techniques for movie content analysis and skimming: Tutorial and overview on video abstraction techniques10.1109/MSP.2006.1621451 ↗
25. Texture mixing and texture movie synthesis using statistical learning
26. Huang, Q., Xiong, Y., Rao, A., Wang, J., and Lin, D. (2020, January 23–28). Movienet: A holistic dataset for movie understanding. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part IV 16.
27. Phon-Amnuaisuk, S., Hadi, S., and Omar, S. (2020, January 18–22). Exploring Spatiotemporal Features for Activity Classifications in Films. Proceedings of the Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand. Proceedings, Part IV 27.10.1007/978-3-030-63820-7_47 ↗
28. Liu, C., Shmilovici, A., and Last, M. (2022, January 23–27). MND: A New Dataset and Benchmark of Movie Scenes Classified by Their Narrative Function. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-25069-9_39 ↗
29. Wilkinghoff, K., and Cornaggia-Urrigshardt, A. (2024, January 14–19). TACos: Learning temporally structured embeddings for few-shot keyword spotting with dynamic time warping. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445814 ↗
30. PANDA Phase One: PANDA collaboration10.1140/epja/s10050-021-00475-y ↗
31. Steel ball surface inspection using modified DRAEM and machine vision10.1007/s10845-024-02370-x ↗
32. Aggelis, D.G., and Shiotani, T. (2021). Parameters based AE analysis. Acoustic Emission Testing: Basics for Research–Applications in Engineering, Springer.10.1007/978-3-030-67936-1_4 ↗
33. Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review10.1007/s12559-024-10285-1 ↗
34. Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang, J. (October, January 29). Long-clip: Unlocking the long-text capability of clip. Proceedings of the European Conference on Computer Vision, Milan, Italy.
35. Touvron, H., Cord, M., and Jégou, H. (2022, January 23–27). Deit iii: Revenge of the vit. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20053-3_30 ↗