Relevant Visual Semantic Context-Aware Attention-Based Dialog
Source abstract
Abstract
The existing dataset for visual dialog comprises multiple rounds of questions and a diverse range of image contents. However, it faces challenges in overcoming visual semantic limitations, particularly in obtaining sufficient context from visual and textual aspects of images. This paper proposes a new visual dialog dataset called Diverse History-Dialog (DS-Dialog) to address the visual semantic limitations faced by the existing dataset. DS-Dialog groups relevant histories based on their respective Microsoft Common Objects in Context (MSCOCO) image categories and consolidates them for each image. Specifically, each MSCOCO image category consists of top relevant histories extracted based on their semantic relationships between the original image caption and historical context. These relevant histories are consolidated for each image, and DS-Dialog enhances the current dataset by adding new context-aware relevant history to provide more visual semantic context for each image. The new dataset is generated through several stages, including image semantic feature extraction, keyphrase extraction, relevant question extraction, and relevant history dialog generation. The DS-Dialog dataset contains about 2.6 million question-answer pairs, where 1.3 million pairs correspond to existing VisDial’s question-answer pairs, and the remaining 1.3 million pairs include a maximum of 5 image features for each VisDial image, with each image comprising 10-round relevant question-answer pairs. Moreover, a novel adaptive relevant history selection is proposed to resolve missing visual semantic information for each image. DS-Dialog is used to benchmark the performance of previous visual dialog models and achieves better performance than previous models. Specifically, the proposed DS-Dialog model achieves an 8% higher mean reciprocal rank (MRR), 11% higher R@1%, 6% higher R@5%, 5% higher R@10%, and 8% higher normalized discounted cumulative gain (NDCG) compared to LF. DS-Dialog also achieves approximately 1 point improvement on R@k, mean, MRR, and NDCG compared to the original RVA, and 2 points improvement compared to LF and DualVD. These results demonstrates the importance of the relevant semantic historical context in enhancing the visual semantic relationship between textual and visual representations of the images and questions.
Source: openalex_inverted_index · Quality: candidate unverified · Source record ↗
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (64)
Show source-provided references
1. Image retrieval by dense caption reasoning
2. Optimized artificial neural network techniques to improve cybersecurity of higher education institution10.32604/cmc.2022.026477 ↗
3. Deep feature fusion model for sentence semantic matching10.32604/cmc.2019.06045 ↗
4. Microsoft COCO: Common objects in context
5. Faster R-CNN: Towards real-time object detection with region proposal networks
6. VQA: Visual question answering
7. VIVO: Visual vocabulary pre-training for novel object captioning
8. Oscar: Object-semantics aligned pre-training for vision-language tasks
9. NOCAPS: Novel object captioning at scale
10. News image captioning based on text summarization using image as query
11. Phrase-based image caption generator with hierarchical LSTM network10.1016/j.neucom.2018.12.026 ↗
12. Densecap: Fully convolutional localization networks for dense captioning
13. Show, attend and tell: Neural image caption generation with visual attention
14. Image captioning with semantic attention
15. Knowing when to look: Adaptive attention via a visual sentinel for image captioning
16. Image captioning using DenseNet network and adaptive attention
17. Multi-attention generative adversarial network for image captioning10.1016/j.neucom.2019.12.073 ↗
18. Co-attention network with question type for visual question answering10.1109/ACCESS.2019.2908035 ↗
19. Residual self-attention for visual question answering
20. Generative attention model with adversarial self-learning for visual question answering
21. Semantic BI-embedded GRU for fill-in-the-blank image question answering
22. Image captioning and visual question answering based on attributes and external knowledge10.1109/TPAMI.2017.2708709 ↗
23. Auer · The Semantic web · 2007 · 722
24. Bottom-up and top-down attention for image captioning and visual question answering
25. Stacked attention networks for image question answering
26. Visual dialog
27. Learning cooperative visual dialog agents with deep reinforcement learning
28. Improving generative visual dialog by answering diverse questions
29. Guesswhat?! Visual object discovery through multi-modal dialogue
30. Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model
31. Visual dialog with targeted objects
32. Attention is all you need
33. Visual reference resolution using attention memory for visual dialog
34. Textual-visual reference-aware attention network for visual dialog10.1109/TIP.2020.2992888 ↗
35. Recursive visual attention in visual dialog
36. Image-question-answer synergistic network for visual dialog
37. Visual coreference resolution in visual dialog using neural module networks
38. Making history matter: History-advantage sequence training for visual dialog
39. Multi-modal fusion with multi-level attention for visual dialog10.1016/j.ipm.2019.102152 ↗
40. DuaLVD: An adaptive dual encoding model for deep visual understanding in visual dialogue
41. Recurrent attention network with reinforced generator for visual dialog10.1145/3390891 ↗
42. Deep learning-based text classification: A comprehensive review10.1145/3439726 ↗
43. A position-aware transformer for image captioning
44. Efficient image captioning based on vision transformer models10.32604/cmc.2022.029313 ↗
45. Dual attention network for scene segmentation
46. J. Devlin, M. W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv, pp. 1810.04805, 2018.
47. TANDA: Transfer and adapt pre-trained transformer models for answer sentence selection
48. How to fine-tune bert for text classification?
49. X. Liu, P. He, W. Chen and J. Gao, “Multi-task deep neural networks for natural language understanding,” arXiv preprint arXiv:1901.11504, 2019.10.18653/v1/P19-1441 ↗
50. Semantics-aware BERT for language understanding
51. A comparative assessment of state-of-the-art methods for multilingual unsupervised keyphrase extraction
52. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv, pp. 1908.10084, 2019.10.18653/v1/D19-1410 ↗
53. L. H. Li, M. Yatskar, D. Yin and C. J. Hsieh, “VisualBERT: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019.
54. Y. Wang, S. Joty, M. R. Lyu, I. King, C. Xiong et al., “VD-BERT: A unified vision and dialog transformer with bert,” arXiv preprint arXiv:2004.13278, 2020.10.18653/v1/2020.emnlp-main.269 ↗
55. VU-BERT: A unified framework for visual dialog
56. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
57. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline
58. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma et al., “ALBERT: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942, 2019.
59. V. Sanh, L. Debut, J. Chaumond and T. Wolf, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019.
60. SpanBERT: Improving pre-training by representing and predicting spans10.1162/tacl_a_00300 ↗
61. M. Grootendorst, “KeyBERT: Minimal keyword extraction with bert,” [Online]. Available: https://maartengr.github.io/KeyBERT/index.html
62. S. Kottur, J. M. Moura, D. Parikh, D. Batra and M. Rohrbach, “Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog,” arXiv:1903.03166, 2019.
63. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning
64. Glove: Global vectors for word representation