An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
Source abstract
Abstract
Visual question answering (VQA) is a task that generates or predicts an answer to a question in human language about visual images. VQA is an active field combining two AI branches: NLP and computer vision. VQA in the medical field is still at an early stage, and it needs vast efforts and exploration to reach practical usage. This paper proposes two models that are utilized in the latest vision and NLP transformers that outperform the SOTA and have not yet been utilized in medical VQA. The ELECTRA-base transformer is used for textual feature extraction, whereas SWIN is used for visual feature extraction. In the SOTA medical VQA, selecting the model is based on the model that achieves the highest validation accuracy or the last model in training. The first proposed model, the best-value-based model, is selected based on the highest validation accuracy. The second model, the greedy-soup-based model, uses a greedy soup technique based on the fusion of multiple fine-tuned models to set the model parameters. The greedy soup selects the model parameters by fusing the model parameters that have significant performance on the validation accuracy in training. The greedy-soup-based model outperforms the best-value-based model, and both proposed models outperform the SOTA, which has an accuracy of 83.49%. The greedy-soup-based model is optimized with batch size and learning rate. During the optimization, seven extra models exceed the SOTA accuracy. The best model trained with a learning rate of 1.0×10−4 and batch size 16 achieves an accuracy of 87.41%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer and Information Sciences, King Saud University, P.O. Box 2614, Riyadh 13312, Saudi Arabia
Computer Engineering Department, Hadhramout University, Al Mukalla 10587, Yemen
King Abdulaziz City for Science and Technology, Riyadh 11442, Saudi Arabia
Funding
Deanship of Scientific Research at King Saud University through the initiative of DSR Graduate Students Research Support
References (83)
Show source-provided references
1. Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. (2016, January 27–28). Visual7w: Grounded question answering in images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.540 ↗
2. Abacha, A.B., Hasan, S.A., Datla, V.V., Liu, J., Demner-Fushman, D., and Müller, H. (2019, January 9–12). VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019. In proceeding of Working Notes of CLEF 2019, Lugano, Switzerland.
3. Abacha, A.B., Datla, V.V., Hasan, S.A., Demner-Fushman, D., and Müller, H. (2020, January 22–25). Overview of the VQA-Med Task at ImageCLEF 2020: Visual Question Answering and Generation in the Medical Domain. Proceedings of the CLEF 2020—Conference and Labs of the Evaluation Forum, Thessaloniki, Greece.
4. Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., and Wu, X.M. (2021, January 13–16). SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering. Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Nice, France.10.1109/ISBI48211.2021.9434010 ↗
5. Tascon-Morales, S., Márquez-Neila, P., and Sznitman, R. (2022). Medical Image Computing and Computer Assisted Intervention—MICCAI 2022, Proceedings of the 25th International Conference, Singapore, 18–22 September 2022, Springer. Part VIII.
6. Image question answering: A visual semantic embedding model and a new dataset
7. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7–13). VQA: Visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.279 ↗
8. Malinowski, M., Rohrbach, M., and Fritz, M. (2015, January 7–13). Ask your neurons: A neural-based approach to answering questions about images. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.9 ↗
9. Jiang, A., Wang, F., Porikli, F., and Li, Y. (2015). Compositional memory for visual question answering. arXiv.
10. Chen, K., Wang, J., Chen, L.C., Gao, H., Xu, W., and Nevatia, R. (2015). ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering. arXiv.
11. Ilievski, I., Yan, S., and Feng, J. (2016). A focused dynamic attention model for visual question answering. arXiv.
12. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016, January 27–28). Neural module networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.12 ↗
13. Song, J., Zeng, P., Gao, L., and Shen, H.T. (2022). From pixels to objects: Cubic visual attention for visual question answering. arXiv.
14. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016). Learning to compose neural networks for question answering. arXiv.10.18653/v1/N16-1181 ↗
15. Xiong, C., Merity, S., and Socher, R. (2016, January 20–22). Dynamic memory networks for visual and textual question answering. Proceedings of the International Conference on Machine Learning, PMLR, New York City, NY, USA.
16. Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., and Socher, R. (2016, January 20–22). Ask me anything: Dynamic memory networks for natural language processing. Proceedings of the International Conference on Machine Learning, PMLR, New York City, NY, USA.
17. Noh, H., and Han, B. (2016). Training recurrent answering units with joint loss minimization for VQA. arXiv.
18. Structured two-stream attention network for video question answering
19. Wang, P., Wu, Q., Shen, C., Hengel, A.v.d., and Dick, A. (2015). Explicit knowledge-based reasoning for visual question answering. arXiv.
20. FVQA: Fact-based visual question answering10.1109/TPAMI.2017.2754246 ↗
21. Wu, Q., Wang, P., Shen, C., Dick, A., and Van Den Hengel, A. (2016, January 27–28). Ask me anything: Free-form visual question answering based on knowledge from external sources. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.500 ↗
22. Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
23. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–28). Deep Residual Learning for Image Recognition. Proceedings of the CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
24. Nguyen, B.D., Do, T.T., Nguyen, B.X., Do, T., Tjiputra, E., and Tran, Q.D. Overcoming Data Limitation in Medical Visual Question Answering. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention.
25. Do, T., Nguyen, B.X., Tjiputra, E., Tran, M., Tran, Q.D., and Nguyen, A. (2021). Multiple Meta-model Quantifying for Medical Visual Question Answering. arXiv.10.1007/978-3-030-87240-3_7 ↗
26. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
27. Bidirectional recurrent neural networks10.1109/78.650093 ↗
28. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
29. Peng, Y., Liu, F., and Rosen, M.P. (2018, January 10–14). UMass at ImageCLEF Medical Visual Question Answering (Med-VQA) 2018 Task. Proceedings of the CEUR Workshop, Avignon, France.
30. Hierarchical question-image co-attention for visual question answering
31. Shi, Y., Furlanello, T., Zha, S., and Anandkumar, A. (2018, January 8–14). Question Type Guided Attention in Visual Question Answering. Proceedings of the ECCV 2018, Munich, Germany.10.1007/978-3-030-01225-0_10 ↗
32. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., and Chang, K.W. (2019). VisualBERT: A simple and performant baseline for vision and language. arXiv.
33. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
34. Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. (2020, January 23–28). Uniter: Universal image-text representation learning. Proceedings of the Computer Vision—ECCV 2020, 16th European Conference, Glasgow, UK.10.1007/978-3-030-58577-8_7 ↗
35. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Virtual only.
36. Cong, F., Xu, S., Guo, L., and Tian, Y. (2022, January 10–14). Caption-Aware Medical VQA via Semantic Focusing and Progressive Cross-Modality Comprehension. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Portugal.10.1145/3503161.3548122 ↗
37. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
38. Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., and Kornblith, S. (2022, January 17–23). Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MA, USA.
39. Lowe, D.G. (1999, January 20–27). Object recognition from local scale-invariant features. Proceedings of the Piscataway, NJ, USA of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece.10.1109/ICCV.1999.790410 ↗
40. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
41. Lienhart, R., and Maydt, J. (2002, January 22–25). An extended set of Haar-like features for rapid object detection. Proceedings of the IEEE International Conference on Image Processing, Rochester, NY, USA.
42. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
43. Information fusion in visual question answering: A Survey10.1016/j.inffus.2019.03.005 ↗
44. Abacha, A.B., Gayen, S., Lau, J.J., Rajaraman, S., and Demner-Fushman, D. NLM at ImageCLEF 2018 Visual Question Answering in the Medical Domain; In Proceedings of Working Notes of CLEF 2018, Avignon, France, 10–14 September 2018.
45. Verma, H., and Ramachandran, S. (2020, January 22–25). HARENDRAKV at VQA-Med 2020: Sequential VQA with Attention for Medical Visual Question Answering. Proceedings of the Working Notes of CLEF 2018, Thessaloniki, Greece.
46. Bounaama, R., and Abderrahim, M.E.A. (2019, January 9–12). Tlemcen University at ImageCLEF 2019 Visual Question Answering Task. Proceedings of the Working Notes of CLEF 2018, Thessaloniki, Lugano, Switzerland.
47. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 5–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
48. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016, January 1–5). Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TC, USA.10.18653/v1/D16-1044 ↗
49. Kim, J.H., On, K.W., Lim, W., Kim, J., Ha, J.W., and Zhang, B.T. (2017, January 24–26). Hadamard Product for Low-rank Bilinear Pooling. Proceedings of the 5th International Conference on Learning Representations, ICLR Toulon, France.
50. Ben-Younes, H., Cadene, R., Cord, M., and Thome, N. (2017, January 22–29). MUTAN: Multimodal Tucker Fusion for Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.285 ↗
51. Medical knowledge-based network for Patient-oriented Visual Question Answering10.1016/j.ipm.2022.103241 ↗
52. Haridas, H.T., Fouda, M.M., Fadlullah, Z.M., Mahmoud, M., ElHalawany, B.M., and Guizani, M. (2022, January 15–18). MED-GPVS: A Deep Learning-Based Joint Biomedical Image Classification and Visual Question Answering System for Precision e-Health. Proceedings of the ICC 2022—IEEE International Conference on Communications, Seoul, Republic of Korea.10.1109/ICC45855.2022.9839076 ↗
53. Kovaleva, O., Shivade, C., Kashyap, S., Kanjaria, K., Wu, J., Ballah, D., Coy, A., Karargyris, A., Guo, Y., and Beymer, D.B. (2020, January 9). Towards Visual Dialog for Radiology. Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, Online.10.18653/v1/2020.bionlp-1.6 ↗
54. Liao, Z., Wu, Q., Shen, C., van den Hengel, A., and Verjans, J. (2020, January 22–25). AIML at VQA-Med 2020: Knowledge Inference via a Skeleton-based Sentence Mapping Approach for Medical Domain Visual Question Answering. Proceedings of the Working Notes of CLEF 2020, Thessaloniki, Greece.
55. Gong, H., Huang, R., Chen, G., and Li, G. (2021, January 21–24). SYSU-Hcp at VQA-MED 2021: A data-centric model with efficient training methodology for medical visual question answering. Proceedings of the Working Notes of CLEF 2021, Bucharest, Romania.
56. Wang, H., Pan, H., Zhang, K., He, S., and Chen, C. (2022, January 10–13). M2FNet: Multi-granularity Feature Fusion Network for Medical Visual Question Answering. Proceedings of the PRICAI 2022: Trends in Artificial Intelligence, 19th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2022, Shanghai, China. Part II.10.1007/978-3-031-20865-2_11 ↗
57. Medical visual question answering based on question-type reasoning and semantic space constraint10.1016/j.artmed.2022.102346 ↗
58. Visual question answering: A state-of-the-art review10.1007/s10462-020-09832-7 ↗
59. He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P. (2020). PathVQA: 30.000+ questions for medical visual question answering. arXiv.10.36227/techrxiv.13127537.v1 ↗
60. Allaouzi, I., Benamrou, B., Benamrou, M., and Ahmed, M.B. (2018, January 10–14). Deep Neural Networks and Decision Tree Classifier for Visual Question Answering in the Medical Domain. Proceedings of the Working Notes of CLEF 2018, Avignon, France.
61. Zhou, Y., Kang, X., and Ren, F. (2018, January 10–14). Employing Inception-Resnet-v2 and Bi-LSTM for Medical Domain Visual Question Answering. Proceedings of the Working Notes of CLEF 2018, Avignon, France.
62. Talafha, B., and Al-Ayyoub, M. (2018, January 10–14). JUST at VQA-Med: A VGG-Seq2Seq Model. Proceedings of the Working Notes of CLEF 2018, Avignon, France.
63. A Question-Centric Model for Visual Question Answering in Medical Imaging10.1109/TMI.2020.2978284 ↗
64. Skip-Thought Vectors
65. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q.V. (2019, January 8–14). XLNet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings of the 33rd Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
66. Eslami, S., de Melo, G., and Meinel, C. (2021, January 21–24). Teams at VQA-MED 2021: BBN-orchestra for long-tailed medical visual question answering. Proceedings of the Working Notes of CLEF 2021, Bucharest, Romania.
67. Schilling, R., Messina, P., Parra, D., and Lobel, H. (2021, January 21–24). Puc Chile team at VQA-MED 2021: Approaching VQA as a classfication task via fine-tuning a pretrained CNN. Proceedings of the Working Notes of CLEF 2021, Bucharest, Romania.
68. Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering10.1109/TNNLS.2018.2817340 ↗
69. Ask Your Neurons: A Deep Learning Approach to Visual Question Answering10.1007/s11263-017-1038-2 ↗
70. Kuniaki, S., Andrew, S., Yoshitaka, U., and Tatsuya, H. (2017, January 10–14). Dualnet: Domain-invariant network for visual question answering. Proceedings of the the IEEE International Conference on Multimedia and Expo (ICME) 2017, Hong Kong.
71. Noh, H., Seo, P.H., and Han, B. (2016, January 27–30). Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction. Proceedings of the CVPR, Las Vegas, NV, USA.10.1109/CVPR.2016.11 ↗
72. Multimodal residual learning for visual QA
73. Cross-modal multistep fusion network with co-attention for visual question answering10.1109/ACCESS.2018.2844789 ↗
74. Bai, Y., Fu, J., Zhao, T., and Mei, T. (2018, January 8–14). Deep Attention Neural Tensor Network for Visual Question Answering. Proceedings of the ECCV 2018, Munich, Germany.10.1007/978-3-030-01258-8_2 ↗
75. Narasimhan, M., and Schwing, A.G. (2018, January 8–14). Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering. Proceedings of the ECCV 2018, Munich, Germany.10.1007/978-3-030-01237-3_28 ↗
76. Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., and Zhuang, Y. (2020, January 14–19). Counterfactual Samples Synthesizing for Robust Visual Question Answering. Proceedings of the Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.01081 ↗
77. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv.
78. A general framework for parallel distributed processing
79. Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16–21). Rectifier nonlinearities improve neural network acoustic models. Proceedings of the ICML, Atlanta, GA, USA.
80. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
81. Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, F. (2018). Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data, 3.10.3390/data3030025 ↗
82. TeleOphta: Machine learning and image processing methods for teleophthalmology10.1016/j.irbm.2013.01.010 ↗
83. Selvaraju, R.R., Tendulkar, P., Parikh, D., Horvitz, E., Ribeiro, M.T., Nushi, B., and Kamar, E. (2020, January 14–19). Squinting at VQA models: Introspecting vqa models with sub-questions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01002 ↗