AMSA: Adaptive Multi-Channel Image Sentiment Analysis Network with Focal Loss

Given the importance of sentiment analysis in diverse environments, various methods are used for image sentiment analysis, including contextual sentiment analysis that utilizes character and scene relationships. However, most existing works employ character faces in conjunction with context, yet lack the capacity to analyze the emotions of characters in unconstrained environments, such as when their faces are obscured or blurred. Accordingly, this article presents the Adaptive Multi-Channel Sentiment Analysis Network (AMSA), a contextual image sentiment analysis framework, which consists of three channels: body, face, and context. AMSA employs Multi-task Cascaded Convolutional Networks (MTCNN) to detect faces within body frames; if detected, facial features are extracted and fused with body and context information for emotion recognition. If not, the model leverages body and context features alone. Meanwhile, to address class imbalance in the EMOTIC dataset, Focal Loss is introduced to improve classification performance, especially for minority emotion categories. Experimental results have shown that certain sentiment categories with lower representation in the dataset demonstrate leading classification accuracy, the AMSA yields a 2.53% increase compared with state-of-the-art methods.