The AI of Oz: A Conceptual Framework for Democratizing Generative AI in Live-Prototyping User Studies
Source abstract
Abstract
Commonly used methods in User-Centered Design (UCD) can face challenges in incorporating user feedback during early design stages, often resulting in extended iteration cycles. To address this, we explore the following question: “How can generative artificial intelligence (AI) be utilized to enable prototyping within user studies to facilitate immediate user feedback integration and validation?” We introduce a conceptual framework for live-prototyping, where designers modify AI-generated components of a prototype in real time through a separate control interface during user testing. This approach invites more immediate interaction between feedback and design decisions. To explore our concept, we engaged in a case study with experienced prototyping practitioners, examining how real-time prototyping might shape design processes. Participants highlighted the framework’s potential to support spontaneous insight generation and enhance collaborative dynamics. However, they also highlighted important considerations, including the need for a certain level of AI knowledge and challenges around planning and reliability. By integrating generative AI into the UCD process, our conceptual framework contributes to ongoing conversations around evolving user-centered methodologies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Ethics and Social Impacts of AI · Innovative Human-Technology Interaction · Persona Design and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence and Human Interfaces, Faculty of Digital and Analytical Sciences, Paris Lodron University Salzburg, Jakob-Haringer-Str. 8, 5020 Salzburg, Austria
Funding
Paris Lodron University of Salzburg
References (51)
Show source-provided references
1. User-Centered Design
2. The state of user-centered design practice10.1145/1047671.1047677 ↗
3. A Working Framework for the User-Centered Design Approach and a Survey of the Available Methods
4. Adapting usability investigations for agile user-centered design
5. Nissinen, T. (2015). User Experience Prototyping: A Literature Review. [Bachelor’s Thesis, University of Oulu].
6. Takaffoli, M., Li, S., and Mäkelä, V. (2024, January 1–5). Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?. Proceedings of the 2024 ACM Designing Interactive Systems Conference, Copenhagen, Denmark.10.1145/3643834.3660720 ↗
7. Uusitalo, S., Salovaara, A., Jokela, T., and Salmimaa, M. (2024, January 1–5). “Clay to Play With”: Generative AI Tools in UX and Industrial Design Practice. Proceedings of the 2024 ACM Designing Interactive Systems Conference, Copenhagen, Denmark.10.1145/3643834.3661624 ↗
8. Petridis, S., Terry, M., and Cai, C.J. (2023, January 23–28). PromptInfuser: Bringing User Interface Mock-ups to Life with Large Language Models. Proceedings of the Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany.10.1145/3544549.3585628 ↗
9. Zhang, H., Chen, P., Xie, X., Lin, C., Liu, L., Li, Z., You, W., and Sun, L. (2024, January 13–16). ProtoDreamer: A Mixed-prototype Tool Combining Physical Model and Generative AI to Support Conceptual Design. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, Pittsburgh, PA, USA.10.1145/3654777.3676399 ↗
10. Lin, P.Y., Andersen, K., Schmidt, R., Schoenmakers, S., Hofmeyer, H., Pauwels, P., and Ijsselsteijn, W. (2024, January 1–5). Text-to-Image AI as a Catalyst for Semantic Convergence in Creative Collaborations. Proceedings of the 2024 ACM Designing Interactive Systems Conference, Copenhagen, Denmark.10.1145/3643834.3661543 ↗
11. Vaithilingam, P., Arawjo, I., and Glassman, E.L. (2024, January 1–5). Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation. Proceedings of the 2024 ACM Designing Interactive Systems Conference, Copenhagen, Denmark.10.1145/3643834.3661525 ↗
12. Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., and Cai, C.J. (May, January 29). PromptMaker: Prompt-based Prototyping with Large Language Models. Proceedings of the Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA.10.1145/3491101.3503564 ↗
13. Ray, A. (2024, January 9). Smart Design Evolution with GenAI and 3D Printing. Proceedings of the 2024 IEEE Integrated STEM Education Conference (ISEC), Princeton, NJ, USA.10.1109/ISEC61299.2024.10665095 ↗
14. Siricharoien, W.V. (2024). Elevating User-Centered Design with AI: A Comprehensive Exploration using the AI-UCD Algorithm Framework. Eai Endorsed Trans. Context-Aware Syst. Appl., 10.10.4108/eetcasa.4211 ↗
15. Wizard of Oz Prototyping: When and How?
16. On using intelligent computer-assisted language learning in real-life foreign language teaching and learning10.1017/S0958344010000261 ↗
17. Vredenburg, K., Mao, J.Y., Smith, P.W., and Carey, T. (2002, January 20–25). A survey of user-centered design practice. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, MN, USA.10.1145/503376.503460 ↗
18. UI/UX for Generative AI: Taxonomy, Trend, and Challenge10.1109/ACCESS.2024.3502628 ↗
19. Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A.E., Sarkar, A., Sellen, A., and Rintel, S. (2024, January 11–16). The Metacognitive Demands and Opportunities of Generative AI. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3613904.3642902 ↗
20. The Transformation of the Design Thinking Process with AI Intervention: Focusing on Generative Artificial Intelligence and Large Language Models10.46248/kidrs.2024.2.25 ↗
21. Kamnerddee, C., Putjorn, P., and Intarasirisawat, J. (2024, January 14–15). AI-Driven Design Thinking: A Comparative Study of Human-Created and AI-Generated UI Prototypes for Mobile Applications. Proceedings of the 2024 8th International Conference on Information Technology (InCIT), Chonburi, Thailand.10.1109/InCIT63192.2024.10810565 ↗
22. Challenges and Responsibilities in the UX Design of Text-to-Image AI Models: A Discussion Through a Comparative Heuristics Evaluation10.46467/TdD40.2024.156-175 ↗
23. Accelerating Innovation with Generative AI: AI-Augmented Digital Prototyping and Innovation Methods10.1109/EMR.2023.3272799 ↗
24. Weisz, J.D., He, J., Muller, M., Hoefer, G., Miles, R., and Geyer, W. (2024, January 11–16). Design Principles for Generative AI Applications. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3613904.3642466 ↗
25. DesignFusion: Integrating Generative Models for Conceptual Design Enrichment10.1115/1.4065487 ↗
26. Subramonyam, H., Seifert, C., and Adar, E. (2021, January 14–17). ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. Proceedings of the 26th International Conference on Intelligent User Interfaces, Santa Fe, NM, USA.10.1145/3397481.3450640 ↗
27. Simpson, J., Richardson, M., and Richards, D. (2022, January 5–8). A Wizard or a Fool? Initial Assessment of a Wizard of Oz Agent Supporting Collaborative Virtual Environments. Proceedings of the 10th International Conference on Human-Agent Interaction, Christchurch, New Zealand.10.1145/3527188.3563930 ↗
28. Shin, A., Oh, J., and Lee, J. (2019, January 11–14). Apprentice of Oz: Human in the Loop System for Conversational Robot Wizard of Oz. Proceedings of the 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Republic of Korea.10.1109/HRI.2019.8673205 ↗
29. IRWoZ: Constructing an Industrial Robot Wizard-of-OZ Dialoguing Dataset10.1109/ACCESS.2023.3259325 ↗
30. Zheng, C., Wu, Y., Shi, C., Ma, S., Luo, J., and Ma, X. (2023, January 23–28). Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany.10.1145/3544548.3581131 ↗
31. Vero: An accessible method for studying human–AI teamwork10.1016/j.chb.2022.107606 ↗
32. Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P., and Bernstein, M.S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv.10.1145/3586183.3606763 ↗
33. Barthet, M., Khalifa, A., Liapis, A., and Yannakakis, G. (2022, January 5–8). Generative Personas That Behave and Experience Like Humans. Proceedings of the 17th International Conference on the Foundations of Digital Games, Athens, Greece.10.1145/3555858.3555879 ↗
34. Dermouche, S., and Pelachaud, C. (2019, January 14–18). Generative Model of Agent’s Behaviors in Human-Agent Interaction. Proceedings of the 2019 International Conference on Multimodal Interaction, Suzhou, China.10.1145/3340555.3353758 ↗
35. Do we want less automation?10.1126/science.adh9429 ↗
36. Yang, H., Yue, S., and He, Y. (2023). Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions. arXiv.
37. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning10.61969/jai.1337500 ↗
38. Santiago, J.M., Parayno, R.L., Deja, J.A., and Samson, B.P.V. (2023). Rolling the Dice: Imagining Generative AI as a Dungeons & Dragons Storytelling Companion. arXiv.
39. Developing a toolkit for prototyping machine learning-empowered products: The design and evaluation of ml-rapid
40. Yang, Q., Steinfeld, A., Rosé, C., and Zimmerman, J. (2020, January 25–30). Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376301 ↗
41. Survey of Hallucination in Natural Language Generation10.1145/3571730 ↗
42. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3–10). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event.10.1145/3442188.3445922 ↗
43. How do people react to ChatGPT’s unpredictable behavior? Anthropomorphism, uncanniness, and fear of AI: A qualitative study on individuals’ perceptions and understandings of LLMs’ nonsensical hallucinations10.1016/j.ijhcs.2025.103471 ↗
44. Dove, G., Halskov, K., Forlizzi, J., and Zimmerman, J. (2017, January 6–11). UX Design Innovation: 2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI EA 2017. Proceedings of the CHI 2017—2017 ACM SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, USA.10.1145/3025453.3025739 ↗
45. Generative AI enhances individual creativity but reduces the collective diversity of novel content10.1126/sciadv.adn5290 ↗
46. Chen, T., Hirota, Y., Otani, M., Garcia, N., and Nakashima, Y. (2024, January 11–15). Would Deep Generative Models Amplify Bias in Future Models?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR52733.2024.01030 ↗
47. OpenAI (2023). GPT-4 Technical Report. arXiv.
48. Cook, V. (2013). Second Language Learning and Language Teaching, Routledge.10.4324/9780203770511 ↗
49. Subramonyam, H., Thakkar, D., Dieber, J., and Sinha, A. (2024). Content-Centric Prototyping of Generative AI Applications: Emerging Approaches and Challenges in Collaborative Software Teams. arXiv.10.1145/3706598.3713166 ↗
50. Sahebnasi, M.J., Farrokhimaleki, M., Wang, N., Zhao, R., and Maurer, F. (2024, January 3–7). Exploring the Potential of Generative AI in Prototyping XR Applications. Proceedings of the Workshop on Prototyping and Developing Real-World Applications of Extended Reality at the 17th International Conference on Advanced Visual Interfaces, Genoa, Italy.
51. AI as a co-creator and a design material: Transforming the design process10.1016/j.destud.2025.101303 ↗