Coordination of Speaking Opportunities in Virtual Reality: Analyzing Interaction Dynamics and Context-Aware Strategies
Source abstract
Abstract
This study explores the factors influencing turn-taking coordination in virtual reality (VR) environments, with a focus on identifying key interaction dynamics that affect the ease of gaining speaking opportunities. By analyzing VR interaction data through logistic regression and clustering, we identify significant variables impacting turn-taking success and categorize typical interaction states that present unique coordination challenges. The findings reveal that features related to interaction proactivity, individual status, and communication quality significantly impact turn-taking outcomes. Furthermore, clustering analysis identifies five primary interaction contexts: high competition, intense interaction, prolonged single turn, high-status role, and low activity, each with unique turn-taking coordination requirements. This work provides insights into enhancing turn-taking support systems in VR, emphasizing contextually adaptive feedback to reduce speaking overlap and turn-taking failures, thereby improving overall interaction flow in immersive environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Automated Systems · Speech and dialogue systems · Cognitive Science and Mapping
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan
Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan
Funding
JSPS KAKENHI
References (79)
Show source-provided references
1. Jerald, J. (2015). The VR Book: Human-Centered Design for Virtual Reality, Morgan & Claypool.10.1145/2792790 ↗
2. Yassien, A., ElAgroudy, P., Makled, E., and Abdennadher, S. (2020, January 25–29). A design space for social presence in VR. Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, Tallinn, Estonia.10.1145/3419249.3420112 ↗
3. Easily applicable social virtual reality and social presence in online higher education during the covid-19 pandemic: A qualitative study
4. Steinicke, F., Lehmann-Willenbrock, N., and Meinecke, A.L. (November, January 30). A first pilot study to compare virtual group meetings using video conferences and (immersive) virtual reality. Proceedings of the 2020 ACM Symposium on Spatial User Interaction, Virtual Event.10.1145/3385959.3422699 ↗
5. Yoshimura, A., and Borst, C.W. (2020). Remote Instruction in Virtual Reality: A Study of Students Attending Class Remotely from Home with VR Headsets. Mensch und Computer 2020—Workshopband, Gesellschaft für Informatik e.V.
6. Tanenbaum, T.J., Hartoonian, N., and Bryan, J. (2020, January 25–30). “How do I make this thing smile?” An Inventory of Expressive Nonverbal Communication in Commercial Social Virtual Reality Platforms. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376606 ↗
7. Williamson, J.R., O’Hagan, J., Guerra-Gomez, J.A., Williamson, J.H., Cesar, P., and Shamma, D.A. (May, January 29). Digital proxemics: Designing social and collaborative interaction in virtual environments. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA.10.1145/3491102.3517594 ↗
8. “Talking without a Voice” Understanding Non-verbal Communication in Social Virtual Reality10.1145/3415246 ↗
9. Ishii, R., Kumano, S., and Otsuka, K. (2017, January 17–20). Prediction of next-utterance timing using head movement in multi-party meetings. Proceedings of the 5th International Conference on Human Agent Interaction, Bielefeld, Germany.10.1145/3125739.3125765 ↗
10. Moustafa, F., and Steed, A. (December, January 28). A longitudinal study of small group interaction in social virtual reality. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, Tokyo, Japan.
11. Williamson, J., Li, J., Vinayagamoorthy, V., Shamma, D.A., and Cesar, P. (2021, January 8–13). Proxemics and social interactions in an instrumented virtual reality workshop. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445729 ↗
12. Ishii, R., Kumano, S., and Otsuka, K. (2016, January 12–16). Analyzing mouth-opening transition pattern for predicting next speaker in multi-party meetings. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.10.1145/2993148.2993189 ↗
13. Lee, M.C., Trinh, M., and Deng, Z. (2023, January 9–13). Multimodal Turn Analysis and Prediction for Multi-party Conversations. Proceedings of the 25th International Conference on Multimodal Interaction, Paris, France.10.1145/3577190.3614139 ↗
14. Li, L., Molhoek, J., and Zhou, J. (2024). Inferring Intentions to Speak Using Accelerometer Data In-the-Wild. arXiv.
15. Chen, J., Gu, C., Zhang, J., Liu, Z., and Konomi, S. (2024). Sensing the Intentions to Speak in VR Group Discussions. Sensors, 24.10.3390/s24020362 ↗
16. A simplest systematics for the organization of turn-taking for conversation10.1353/lan.1974.0010 ↗
17. Some signals and rules for taking speaking turns in conversations10.1037/h0033031 ↗
18. Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns
19. Using prosody to predict the end of sentences in English and French: Normal and brain-damaged subjects10.1080/016909696387231 ↗
20. Projecting the end of a speaker’s turn: A cognitive cornerstone of conversation10.1353/lan.2006.0130 ↗
21. Novick, D.G., Hansen, B., and Ward, K. (1996, January 3–6). Coordinating turn-taking with gaze. Proceedings of the Fourth International Conference on Spoken Language Processing, ICSLP’96, Philadelphia, PA, USA.10.1109/ICSLP.1996.608001 ↗
22. Streeck, J., and Hartge, U. (1992). Previews: Gestures at the transition place. The Contextualization of Language, John Benjamins Publishing Company.10.1075/pbns.22.10str ↗
23. Gaze and mutual gaze10.1017/S0007125000073980 ↗
24. Jokinen, K., Nishida, M., and Yamamoto, S. (2010, January 7). On eye-gaze and turn-taking. Proceedings of the 2010 Workshop on Eye Gaze in Intelligent Human Machine Interaction, Hong Kong, China.10.1145/2002333.2002352 ↗
25. Gaze and turn-taking behavior in casual conversational interactions10.1145/2499474.2499481 ↗
26. Petukhova, V., and Bunt, H. (2009, January 24–26). Who’s next? Speaker-selection mechanisms in multiparty dialogue. Proceedings of the Workshop on the Semantics and Pragmatics of Dialogue, Stockholm, Sweden.
27. Whose turn is it anyway? Latency and the organization of turn-taking in video-mediated interaction10.1016/j.pragma.2020.11.005 ↗
28. Akselrad, D., DeVeaux, C., Han, E., Miller, M.R., and Bailenson, J.N. (2023, January 14–18). Body crumple, sound intrusion, and embodiment violation: Toward a framework for miscommunication in VR. Proceedings of the Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, Minneapolis, MN, USA.10.1145/3584931.3606968 ↗
29. Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., and Sasse, M.A. (2003, January 5–10). The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA.10.1145/642700.642703 ↗
30. Vertegaal, R., Van der Veer, G., and Vons, H. (2000, January 15–17). Effects of gaze on multiparty mediated communication. Proceedings of the Graphics Interface, Montréal, QC, Canada.
31. Vertegaal, R., Weevers, I., Sohn, C., and Cheung, C. (2003, January 5–10). Gaze-2: Conveying eye contact in group video conferencing using eye-controlled camera direction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA.10.1145/642700.642702 ↗
32. Conversations over video conferences: An evaluation of the spoken aspects of video-mediated communication10.1207/s15327051hci0804_4 ↗
33. Whittaker, S. (2003). Theories and methods in mediated communication: Steve whittaker. Handbook of Discourse Processes, Routledge.
34. McVeigh-Schultz, J., Kolesnichenko, A., and Isbister, K. (2019, January 4–9). Shaping pro-social interaction in VR: An emerging design framework. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300794 ↗
35. Evidence for a collective intelligence factor in the performance of human groups10.1126/science.1193147 ↗
36. Meetings at work: Perceived effectiveness and recommended improvements10.1016/j.jbusres.2015.02.015 ↗
37. Kocsis, D.J., de Vreede, G.J., and Briggs, R.O. (2015). Designing and Executing Effective Meetings with Codified Best Facilitation Practices. The Cambridge Handbook of Meeting Science, Cambridge University Press.10.1017/CBO9781107589735.021 ↗
38. Argelaguet, F., McMahan, R., and Sugimoto, M. (2020). Evaluation and Comparison of Desktop Viewing and Headset Viewing of Remote Lectures in VR with Mozilla Hubs. Proceedings of the ICAT-EGVE 2020—International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, The Eurographics Association.
39. Hu, E., Grønbæk, J.E.S., Houck, A., and Heo, S. (, January 23–28). Openmic: Utilizing proxemic metaphors for conversational floor transitions in multiparty video meetings. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany.10.1145/3544548.3581013 ↗
40. He, Z., Wang, K., Feng, B.Y., Du, R., and Perlin, K. (2021, January 10–14). Gazechat: Enhancing virtual conferences with gaze-aware 3d photos. Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event.10.1145/3472749.3474785 ↗
41. Realistic facial expression reconstruction for VR HMD users10.1109/TMM.2019.2933338 ↗
42. Kurzweg, M., Reinhardt, J., Nabok, W., and Wolf, K. (2021). Using body language of avatars in vr meetings as communication status cue. Proceedings of Mensch und Computer 2021, Association for Computing Machinery.10.1145/3473856.3473865 ↗
43. Li, J.V., Kreminski, M., Fernandes, S.M., Osborne, A., McVeigh-Schultz, J., and Isbister, K. (May, January 29). Conversation balance: A shared vr visualization to support turn-taking in meetings. Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts, New Orleans, LA, USA.
44. Using respiration to predict who will speak next and when in multiparty meetings10.1145/2946838 ↗
45. Prediction of who will be the next speaker and when using gaze behavior in multiparty meetings10.1145/2946838 ↗
46. Mizuno, S., Hojo, N., Kobashikawa, S., and Masumura, R. (2023, January 4–10). Next-speaker prediction based on non-verbal information in multi-party video conversation. Proceedings of the ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10094679 ↗
47. Predicting Who Will Be the Next Speaker and When in Multi-party Meetings10.53829/ntr201507ra1 ↗
48. Wang, P., Han, E., Queiroz, A., DeVeaux, C., and Bailenson, J.N. (2024). Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Reality. arXiv.
49. Gu, C., Chen, J., Zhang, J., Yang, T., Liu, Z., and Konomi, S. (2024). Detecting Leadership Opportunities in Group Discussions Using Off-the-Shelf VR Headsets. Sensors, 24.10.3390/s24082534 ↗
50. Spatial perception of ceiling height and type variation in immersive virtual environments10.1016/j.buildenv.2019.106285 ↗
51. Immersive virtual environments versus physical built environments: A benchmarking study for building design and user-built environment explorations10.1016/j.autcon.2015.03.020 ↗
52. Restorative Effects of Virtual Nature Settings10.1089/cyber.2009.0308 ↗
53. Synchrony within triads using virtual reality
54. The effects of avatars’ gender and appearance on social behavior in virtual worlds10.4101/jvwr.v2i5.779 ↗
55. Wu, S., Xu, L., Dai, Z., and Pan, Y. (2023). Factors affecting avatar customization behavior in virtual environments. Electronics, 12.10.20944/preprints202304.1220.v1 ↗
56. (2024, December 17). ThoughtCo. Two Truths and a Lie: How to Play. Available online: https://www.thoughtco.com/2-truths-lie-idea-list-1-31144.
57. Zwaagstra, L. (1997). Group Dynamics and Initiative Activities with Outdoor Programs. Back to the Basics: Proceedings of the International Conference on Outdoor Recreation and Education, Association of Outdoor Recreation and Education.
58. The effect of using different kinds of ice-breakers on upperintermediate language learners’ speaking ability
59. An analysis of turn-taking and backchannels based on prosodic and syntactic features in Japanese map task dialogs10.1177/002383099804100404 ↗
60. Maynard, S.K. (1989). Japanese Conversation: Self-Contextualization Through Structure and Interactional Management, Ablex Publishing Corporation.
61. When gaze-selected next speakers do not take the turn10.1016/j.pragma.2018.05.016 ↗
62. Schegloff, E.A. (2001). Accounts of conduct in interaction: Interruption, overlap, and turn-taking. Handbook of Sociological Theory, Springer.10.1007/0-387-36274-6_15 ↗
63. Hilton, K. (2016, January 8–12). The Perception of Overlapping Speech: Effects of Speaker Prosody and Listener Attitudes. Proceedings of the Interspeech, San Francisco, CA, USA.10.21437/Interspeech.2016-1456 ↗
64. Cafaro, A., Glas, N., and Pelachaud, C. (2016, January 9–13). The effects of interrupting behavior on interpersonal attitude and engagement in dyadic interactions. Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore.
65. Participation in Heterogeneous and Homogeneous Groups: A Theoretical Integration10.1086/229742 ↗
66. Models of participation in status-differentiated groups10.2307/2786983 ↗
67. Marking the turn: Obligation, engagement, and alienation in group discussions10.1177/0190272510371456 ↗
68. Hall, E.T. (1966). The Hidden Dimension, Doubleday & Co.. [1st ed.].
69. Distance matters: Physical space and social impact10.1177/0146167295218002 ↗
70. Isn’t that spatial?: Distance and communication in a 2-D virtual environment
71. Interpersonal distance in the SARS-CoV-2 crisis10.1177/0018720820956858 ↗
72. An interactive table for supporting participation balance in face-to-face collaborative learning10.1109/TLT.2010.18 ↗
73. Kim, J., Truong, K.P., Charisi, V., Zaga, C., Lohse, M., Heylen, D., and Evers, V. (2015, January 6–10). Vocal turn-taking patterns in groups of children performing collaborative tasks: An exploratory study. Proceedings of the 16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2015, Dresden, Germany.10.21437/Interspeech.2015-380 ↗
74. Speaking turns in face-to-face discussions10.1037/0022-3514.60.5.675 ↗
75. Dimensions of group process: Amount and structure of vocal interaction10.1016/S0065-2601(08)60413-X ↗
76. Duncan, S., and Fiske, D.W. (1977). Face-to-Face Interaction: Research, Methods, and Theory, Routledge. [1st ed.].
77. Dominance as expressed and inferred through speaking time: A meta-analysis
78. Aran, O., and Gatica-Perez, D. (2013, January 9–13). One of a kind: Inferring personality impressions in meetings. Proceedings of the 15th ACM on International Conference on Multimodal Interaction, Sydney, Australia.10.1145/2522848.2522859 ↗
79. Lepri, B., Subramanian, R., Kalimeri, K., Staiano, J., Pianesi, F., and Sebe, N. (2010, January 8–10). Employing social gaze and speaking activity for automatic determination of the extraversion trait. Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, Beijing, China.10.1145/1891903.1891913 ↗