Chatbot-Based Natural Language Interfaces for Data Visualisation: A Scoping Review
Source abstract
Abstract
Rapid growth in the generation of data from various sources has made data visualisation a valuable tool for analysing data. However, visual analysis can be a challenging task, not only due to intricate dashboards but also when dealing with complex and multidimensional data. In this context, advances in Natural Language Processing technologies have led to the development of Visualisation-oriented Natural Language Interfaces (V-NLIs). In this paper, we carry out a scoping review that analyses synergies between the fields of Data Visualisation and Natural Language Interaction. Specifically, we focus on chatbot-based V-NLI approaches and explore and discuss three research questions. The first two research questions focus on studying how chatbot-based V-NLIs contribute to interactions with the Data and Visual Spaces of the visualisation pipeline, while the third seeks to know how chatbot-based V-NLIs enhance users’ interaction with visualisations. Our findings show that the works in the literature put a strong focus on exploring tabular data with basic visualisations, with visual mapping primarily reliant on fixed layouts. Moreover, V-NLIs provide users with restricted guidance strategies, and few of them support high-level and follow-up queries. We identify challenges and possible research opportunities for the V-NLI community such as supporting high-level queries with complex data, integrating V-NLIs with more advanced systems such as Augmented Reality (AR) or Virtual Reality (VR), particularly for advanced visualisations, expanding guidance strategies beyond current limitations, adopting intelligent visual mapping techniques, and incorporating more sophisticated interaction methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Visualization and Analytics · Semantic Web and Ontologies · Data Management and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Departament de Matemàtiques i Informàtica, Universitat de Barcelona (UB), 08007 Barcelona, Spain
Institut of Complex Systems (UBICS), Universitat de Barcelona (UB), 08007 Barcelona, Spain
Institut de Matemàtica UB (IMUB), Universitat de Barcelona (UB), 08007 Barcelona, Spain
Funding
SGR project · Ministerio de Ciencia e Innovación (Spain) · MCIN/AEI/10.13039/501100011033
References (100+)
Show source-provided references
1. Information visualisation methods and techniques: State-of-the-art and future directions
2. Kerren, A., Purchase, H.C., and Ward, M.O. (2014). Multivariate Network Visualization, Springer.10.1007/978-3-319-06793-3 ↗
3. Visforum: A visual analysis system for exploring user groups in online forums10.1145/3162075 ↗
4. Freeman, T.C., Horsewell, S., Patir, A., Harling-Lee, J., Regan, T., Shih, B.B., Prendergast, J., Hume, D.A., and Angus, T. (2022). Graphia: A platform for the graph-based visualisation and analysis of high dimensional data. PLoS Comput. Biol., 18.10.1371/journal.pcbi.1010310 ↗
5. Zheng, B., and Sadlo, F. (2021, January 19–21). On the visualization of hierarchical multivariate data. Proceedings of the 2021 IEEE 14th Pacific Visualization Symposium, Tianjin, China.10.1109/PacificVis52677.2021.00026 ↗
6. Stoiber, C., Rind, A., Grassinger, F., Gutounig, R., Goldgruber, E., Sedlmair, M., Emrich, Š., and Aigner, W. (2019, January 3–7). Netflower: Dynamic network visualization for data journalists. Proceedings of the Computer Graphics Forum, Porto, Portugal.10.1111/cgf.13721 ↗
7. Sondag, M., Meulemans, W., Schulz, C., Verbeek, K., Weiskopf, D., and Speckmann, B. (2020, January 3–5). Uncertainty treemaps. Proceedings of the 2020 IEEE Pacific Visualization Symposium (PacificVis), Tianjin, China.10.1109/PacificVis48177.2020.7614 ↗
8. Bastian, M., Heymann, S., and Jacomy, M. (2009, January 17–20). Gephi: An open source software for exploring and manipulating networks. Proceedings of the International AAAI Conference on Web and Social Media, San Jose, CA, USA.10.1609/icwsm.v3i1.13937 ↗
9. IoT data visualization for business intelligence in corporate finance10.1016/j.ipm.2021.102736 ↗
10. Toward a deeper understanding of the role of interaction in information visualization10.1109/TVCG.2007.70515 ↗
11. Srinivasan, A., and Stasko, J. (2017, January 12–16). Natural language interfaces for data analysis with visualization: Considering what has and could be asked. Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers, Barcelona, Spain.
12. Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., and Wang, J. (2021). Towards natural language interfaces for data visualization: A survey. arXiv.
13. Analysis methods in neural language processing: A survey10.1162/tacl_a_00254 ↗
14. It’s Good to Talk: A Comparison of Using Voice Versus Screen-Based Interactions for Agent-Assisted Tasks10.1145/3484221 ↗
15. Orko: Facilitating multimodal interaction for visual exploration and analysis of networks10.1109/TVCG.2017.2745219 ↗
16. Murillo-Morales, T., and Miesenberger, K. (2020, January 9–11). Audial: A natural language interface to make statistical charts accessible to blind persons. Proceedings of the Computers Helping People with Special Needs: 17th International Conference, ICCHP 2020, Lecco, Italy. Proceedings, Part I 17.10.1007/978-3-030-58796-3_44 ↗
17. OpenAI (2023, May 08). ChatGPT. Available online: https://openai.com/models/.
18. OpenAI (2023, May 08). DALL-E: A Generative Model for Diverse and Creative Images. Available online: https://openai.com/dall-e-2/.
19. How to ask what to say?: Strategies for evaluating natural language interfaces for data visualization10.1109/MCG.2020.2986902 ↗
20. A multi-modal natural language interface to an information visualization environment10.1023/A:1011368926479 ↗
21. McTear, M. (2020). Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots, Morgan & Claypool Publishers.10.1007/978-3-031-02176-3 ↗
22. Hoque, E., Kavehzadeh, P., and Masry, A. (2022). Chart Question Answering: State of the Art and Future Directions. arXiv.10.1111/cgf.14573 ↗
23. Card, M. (1999). Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann.
24. Shneiderman, B. (2003). The Craft of Information Visualization, Elsevier.
25. Schulz, H.J., and Schumann, H. (2006, January 5–7). Visualizing graphs-a generalized view. Proceedings of the Tenth International Conference on Information Visualisation (IV’06), London, UK.
26. Making data visualization more efficient and effective: A survey10.1007/s00778-019-00588-3 ↗
27. Hanrahan, P. (2006, January 27–29). Vizql: A language for query, analysis and visualization. Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, Chicago, IL, USA.10.1145/1142473.1142560 ↗
28. Natural Language to visualization by neural machine translation10.1109/TVCG.2021.3114848 ↗
29. Reactive vega: A streaming dataflow architecture for declarative interactive visualization10.1109/TVCG.2015.2467091 ↗
30. Visualisation of hierarchical multivariate data: Categorisation and case study on hate speech10.1177/14738716221120509 ↗
31. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments10.1073/pnas.1807180116 ↗
32. Data and information visualization methods, and interactive mechanisms: A survey
33. Polaris: A system for query, analysis, and visualization of multidimensional relational databases10.1109/2945.981851 ↗
34. Convis: A visual text analytic system for exploring blog conversations10.1111/cgf.12378 ↗
35. Microsoft Power BI (2023, May 08). Available online: https://powerbi.microsoft.com/es-es/.
36. Chabot, C., Stolte, C., and Hanrahan, P. (2003). Tableau software. Tableau Softw., 6, Available online: https://www.tableau.com/.
37. Show me: Automatic presentation for visual analysis10.1109/TVCG.2007.70594 ↗
38. Setlur, V., Tory, M., and Djalali, A. (2019, January 16–20). Inferencing underspecified natural language utterances in visual analysis. Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA.10.1145/3301275.3302270 ↗
39. Applying pragmatics principles for interaction with visual analytics10.1109/TVCG.2017.2744684 ↗
40. Srinivasan, A., and Setlur, V. (2021, January 10–14). Snowy: Recommending Utterances for Conversational Visual Analysis. Proceedings of the The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA.10.1145/3472749.3474792 ↗
41. Wang, C., Feng, Y., Bodik, R., Dillig, I., Cheung, A., and Ko, A.J. (2021, January 8–13). Falx: Synthesis-Powered Visualization Authoring. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445249 ↗
42. Hu, K., Bakker, M.A., Li, S., Kraska, T., and Hidalgo, C. (2019, January 4–9). VizML: A Machine Learning Approach to Visualization Recommendation. Proceedings of the 2019 Conference on Human Factors in Computing Systems (CHI), Glasgow, UK.10.1145/3290605.3300358 ↗
43. Borgo, R., Kehrer, J., Chung, D.H., Maguire, E., Laramee, R.S., Hauser, H., Ward, M., and Chen, M. (2013, January 6–10). Glyph-based Visualization: Foundations, Design Guidelines, Techniques and Applications. Proceedings of the Eurographics (State of the Art Reports), Girona, Spain.
44. A survey on automatic infographics and visualization recommendations10.1016/j.visinf.2020.07.002 ↗
45. Infocolorizer: Interactive recommendation of color palettes for infographics10.1109/TVCG.2021.3085327 ↗
46. Wang, Q., Chen, Z., Wang, Y., and Qu, H. (2020). Applying machine learning advances to data visualization: A survey on ml4vis. arXiv.
47. Stasko, J., and Zhang, E. (2000, January 9–10). Focus+ context display and navigation techniques for enhancing radial, space-filling hierarchy visualizations. Proceedings of the IEEE Symposium on Information Visualization 2000, INFOVIS 2000 Proceedings, Salt Lake City, UT, USA.
48. Scalable visual data exploration of large data sets via multiresolution
49. What is interaction for data visualization?10.1109/TVCG.2019.2934283 ↗
50. Amar, R., Eagan, J., and Stasko, J. (2005, January 23–25). Low-level components of analytic activity in information visualization. Proceedings of the IEEE Symposium on Information Visualization, 2005—INFOVIS 2005, Minneapolis, MN, USA.
51. Chatbots: History, technology, and applications
52. Interacting with educational chatbots: A systematic review10.1007/s10639-022-11177-3 ↗
53. Health care chatbots are here to help10.1109/MPULS.2019.2911816 ↗
54. Thomas, N. (, January 21–24). An e-business chatbot using AIML and LSA. Proceedings of the 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Jaipur, India.10.1109/ICACCI.2016.7732476 ↗
55. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries10.1109/TVCG.2020.3030378 ↗
56. Liu, C., Han, Y., Jiang, R., and Yuan, X. (2021, January 19–21). Advisor: Automatic visualization answer for natural-language question on tabular data. Proceedings of the 2021 IEEE 14th Pacific Visualization Symposium (PacificVis), Tianjin, China.10.1109/PacificVis52677.2021.00010 ↗
57. Embodied conversational interface agents10.1145/332051.332075 ↗
58. Enhancing sentient embodied conversational agents with machine learning10.1016/j.patrec.2019.11.035 ↗
59. Ma, Z., Dou, Z., Zhu, Y., Zhong, H., and Wen, J.R. (2021, January 11–15). One chatbot per person: Creating personalized chatbots based on implicit user profiles. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada.10.1145/3404835.3462828 ↗
60. Natural language processing: State of the art, current trends and challenges10.1007/s11042-022-13428-4 ↗
61. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation10.1613/jair.5477 ↗
62. Gao, T., Dontcheva, M., Adar, E., Liu, Z., and Karahalios, K.G. (2015, January 11–15). Datatone: Managing ambiguity in natural language interfaces for data visualization. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, Charlotte, NC, USA.10.1145/2807442.2807478 ↗
63. Beck, S., Frank, S., Hakamian, A., Merino, L., and van Hoorn, A. (2021, January 27–28). TransVis: Using Visualizations and Chatbots for Supporting Transient Behavior in Microservice Systems. Proceedings of the 2021 Working Conference on Software Visualization (VISSOFT), Luxembourg City, Luxembourg.10.1109/VISSOFT52517.2021.00016 ↗
64. Shi, D., Guo, Y., Guo, M., Wu, Y., Chen, Q., and Cao, N. (2021). Talk2Data: High-Level Question Decomposition for Data-Oriented Question and Answering. arXiv.
65. How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges10.1145/3534965 ↗
66. Steinmetz, N., Senthil-Kumar, B., and Sattler, K.U. (2021, January 23–26). Conversational Question Answering Using a Shift of Context. Proceedings of the EDBT/ICDT Workshops, Nicosia, Cyprus.
67. Tory, M., and Setlur, V. (2019, January 20–25). Do what i mean, not what i say! design considerations for supporting intent and context in analytical conversation. Proceedings of the 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), Vancouver, BC, Canada.10.1109/VAST47406.2019.8986918 ↗
68. Setlur, V., Battersby, S.E., Tory, M., Gossweiler, R., and Chang, A.X. (2016, January 16–19). Eviza: A natural language interface for visual analysis. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan.10.1145/2984511.2984588 ↗
69. Siddiqui, N., and Hoque, E. (2020, January 7–11). ConVisQA: A Natural Language Interface for Visually Exploring Online Conversations. Proceedings of the 2020 24th International Conference Information Visualisation (IV), Melbourne, Australia.10.1109/IV51561.2020.00077 ↗
70. FlowNL: Asking the Flow Data in Natural Languages
71. Hearst, M., and Tory, M. (2019, January 20–25). Would you like a chart with that? Incorporating visualizations into conversational interfaces. Proceedings of the 2019 IEEE Visualization Conference (VIS), Vancouver, BC, Canada.10.1109/VISUAL.2019.8933766 ↗
72. Otmazgin, S., Cattan, A., and Goldberg, Y. (2022). F-COREF: Fast, Accurate and Easy to Use Coreference Resolution. arXiv.
73. Huang, Y., Wang, Y., and Tam, Y.C. (2021). Uniter-based situated coreference resolution with rich multimodal input. arXiv.
74. More than telling a story: Transforming data into visually shared stories10.1109/MCG.2015.99 ↗
75. Narrative visualization: Telling stories with data10.1109/TVCG.2010.179 ↗
76. Access to health care, reproductive health and disability: A large scale survey in Sierra Leone10.1016/j.socscimed.2011.08.040 ↗
77. Guidance for conducting systematic scoping reviews
78. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation10.7326/M18-0850 ↗
79. Constructing a search strategy and searching for evidence10.1097/01.NAJ.0000446779.99522.f6 ↗
80. Bieliauskas, S., and Schreiber, A. (2017, January 18–19). A conversational user interface for software visualization. Proceedings of the 2017 IEEE Working Conference on Software Visualization (Vissoft), Shanghai, China.10.1109/VISSOFT.2017.21 ↗
81. John, R.J.L., Potti, N., and Patel, J.M. (2017, January 8–11). Ava: From Data to Insights through Conversations. Proceedings of the CIDR, Chaminade, CA, USA.
82. Lee, D.J.L., Quamar, A., Kandogan, E., and Özcan, F. (2021, January 20–25). Boomerang: Proactive insight-based recommendations for guiding conversational data analysis. Proceedings of the 2021 International Conference on Management of Data, Virtual Event, China.10.1145/3448016.3452748 ↗
83. Maddigan, P., and Susnjak, T. (2023). Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models. arXiv.10.1109/ACCESS.2023.3274199 ↗
84. Kim, Y.H., Lee, B., Srinivasan, A., and Choe, E.K. (2021, January 8–13). Data@ hand: Fostering visual exploration of personal data on smartphones leveraging speech and touch interaction. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445421 ↗
85. Interweaving multimodal interaction with flexible unit visualizations for data exploration10.1109/TVCG.2020.2978050 ↗
86. Zhi, Q., and Metoyer, R. (2020, January 25–30). Gamebot: A visualization-augmented chatbot for sports game. Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3334480.3382794 ↗
87. GeCoAgent: A conversational agent for empowering genomic data extraction and analysis
88. Srinivasan, A., Lee, B., Henry Riche, N., Drucker, S.M., and Hinckley, K. (2020, January 25–30). InChorus: Designing consistent multimodal interactions for data visualization on tablet devices. Proceedings of the 2020 CHI conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376782 ↗
89. Fast, E., Chen, B., Mendelsohn, J., Bassen, J., and Bernstein, M.S. (2018, January 21–26). Iris: A conversational agent for complex tasks. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3174047 ↗
90. Chowdhury, I., Moeid, A., Hoque, E., Kabir, M.A., Hossain, M.S., and Islam, M.M. (2020, January 7–11). MIVA: Multimodal interactions for facilitating visual analysis with multiple coordinated views. Proceedings of the 2020 24th International Conference Information Visualisation (IV), Melbourne, Australia.10.1109/IV51561.2020.00124 ↗
91. Ruoff, M., Myers, B.A., and Maedche, A. (May, January 29). ONYX-User Interfaces for Assisting in Interactive Task Learning for Natural Language Interfaces of Data Visualization Tools. Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts, New Orleans, LA, USA.
92. Kassel, J.F., and Rohs, M. (2018, January 21–26). Valletto: A multimodal interface for ubiquitous visual analytics. Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3170427.3188445 ↗
93. Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J.R., Bethard, S., and McClosky, D. (2014, January 23–24). The Stanford CoreNLP natural language processing toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, MD, USA.10.3115/v1/P14-5010 ↗
94. Bird, S. (2006, January 17–18). NLTK: The natural language toolkit. Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, Sydney, Australia.10.3115/1225403.1225421 ↗
95. Bush, N., Wallace, R., Ringate, T., Taylor, A., and Baer, J. (2023, May 08). Artificial Intelligence Markup Language (AIML) Version 1.0.1. Alice Foundation Work Draft. Available online: http://www.aiml.foundation/.
96. Parr, T. (2013). The Definitive ANTLR 4 Reference, Torrossa.
97. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing
98. AppleInc. (2023, May 08). Speech|Apple Developer. Available online: https://developer.apple.com/documentation/speech.
99. Microsoft (2023, May 08). Cognitive Speech Services|Microsoft Azure. Available online: https://azure.microsoft.com/en-us/products/cognitive-services/speech-services/.
100. Kelly, S. (2023, May 08). Compromise. Available online: http://compromise.cool/.