Human Annotated Dialogues Dataset for Natural Conversational Agents
Source abstract
Abstract
Conversational agents are gaining huge popularity in industrial applications such as digital assistants, chatbots, and particularly systems for natural language understanding (NLU). However, a major drawback is the unavailability of a common metric to evaluate the replies against human judgement for conversational agents. In this paper, we develop a benchmark dataset with human annotations and diverse replies that can be used to develop such metric for conversational agents. The paper introduces a high-quality human annotated movie dialogue dataset, HUMOD, that is developed from the Cornell movie dialogues dataset. This new dataset comprises 28,500 human responses from 9500 multi-turn dialogue history-reply pairs. Human responses include: (i) ratings of the dialogue reply in relevance to the dialogue history; and (ii) unique dialogue replies for each dialogue history from the users. Such unique dialogue replies enable researchers in evaluating their models against six unique human responses for each given history. Detailed analysis on how dialogues are structured and human perception on dialogue score in comparison with existing models are also presented.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Multimodal Machine Learning Applications · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
AIT Austrian Institute of Technology, 2700 Wiener Neustadt, Austria
CentraleSupélec, Université de Lorraine, CNRS, LORIA, F-57000 Metz, France
Holzinger Group, HCI-KDD, Institute for Medical Informatics/Statistics, Medical University Graz, 8036 Graz, Austria
FH Joanneum Gesellschaft mbH, 8020 Graz, Austria
Université de Lorraine, CNRS, LIEC, F-57000 Metz, France
Funding
H2020 Marie Skłodowska-Curie Actions
References (41)
Show source-provided references
1. Petukhova, V., Gropp, M., Klakow, D., Schmidt, A., Eigner, G., Topf, M., Srb, S., Motlicek, P., Potard, B., and Dines, J. (2014). The DBOX Corpus Collection of Spoken Human-Human and Human-Machine Dialogues, European Language Resources Association (ELRA). Technical Report.
2. Henderson, M., Thomson, B., and Williams, J.D. (2014, January 18–20). The second dialog state tracking challenge. Proceedings of the SIGDIAL 2014 Conference, Philadelphia, PA, USA.10.3115/v1/W14-4337 ↗
3. Williams, J., Raux, A., Ramachandran, D., and Black, A. (2013, January 22–24). The dialog state tracking challenge. Proceedings of the SIGDIAL 2013 Conference, Metz, France.
4. Lowe, R., Pow, N., Serban, I., and Pineau, J. (2015, January 2–4). The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. Proceedings of the SIGDIAL 2015 Conference, Prague, Czech Republic.10.18653/v1/W15-4640 ↗
5. Artstein, R., Gandhe, S., Gerten, J., Leuski, A., and Traum, D. (2009). Semi-formal evaluation of conversational characters. Languages: From formal to natural, Springer.10.1007/978-3-642-01748-3_2 ↗
6. Holzinger, A. (2002). User-Centered Interface Design for disabled and elderly people: First experiences with designing a patient communication system (PACOSY). Computer Helping People with Special Needs, ICCHP 2002, Lecture Notes in Computer Science (LNCS 2398), Springer.10.1007/3-540-45491-8_8 ↗
7. Danescu-Niculescu-Mizil, C., and Lee, L. (2011, January 23). Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, Portland, OR, USA.
8. Forchini, P. (2009, January 20–23). Spontaneity reloaded: American face-to-face and movie conversation compared. Proceedings of the Corpus Linguistics Conference 2009 (CL2009), Liverpool, UK.
9. Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., and Pineau, J. (2016, January 13–15). On the evaluation of dialogue systems with next utterance classification. Proceedings of the SIGDIAL 2016 Conference, Los Angeles, CA, USA.10.18653/v1/W16-3634 ↗
10. Bordes, A., Boureau, Y.L., and Weston, J. (2017, January 24–26). Learning end-to-end goal-oriented dialog. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
11. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016, January 12–17). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), San Diego, CA, USA.10.18653/v1/N16-1174 ↗
12. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. (Long and Short Papers).
13. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association For Computational Linguistics (ACL), Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
14. Lin, C.Y. (2004). Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, Association for Computational Linguistics.
15. Banerjee, S., and Lavie, A. (2005, January 9). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
16. A survey on dialogue systems: Recent advances and new frontiers10.1145/3166054.3166058 ↗
17. PEGASUS: A spoken dialogue interface for online air travel planning10.1016/0167-6393(94)90083-3 ↗
18. Raux, A., Langner, B., Bohus, D., Black, A.W., and Eskenazi, M. (2005, January 4–8). Let’s Go Public! Taking a spoken dialog system to the real world. Proceedings of the Ninth European Conference on Speech Communication and Technology, Lisbon, Portugal.10.21437/Interspeech.2005-399 ↗
19. Allen, J.F., Miller, B.W., Ringger, E.K., and Sikorski, T. (1996, January 23–28). A robust system for natural spoken dialogue. Proceedings of the 34th annual meeting on Association for Computational Linguistics (ACL), Santa Cruz, CA, USA.10.3115/981863.981872 ↗
20. Dodge, J., Gane, A., Zhang, X., Bordes, A., Chopra, S., Miller, A., Szlam, A., and Weston, J. (2016, January 2–4). Evaluating prerequisite qualities for learning end-to-end dialog systems. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
21. Ritter, A., Cherry, C., and Dolan, W.B. (2011, January 27–31). Data-driven response generation in social media. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), Edinburgh, UK.
22. Wang, H., Lu, Z., Li, H., and Chen, E. (2013, January 18–21). A dataset for research on short-text conversations. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), Washington, DC, USA.
23. Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. (2018). Personalizing Dialogue Agents: I have a dog, do you have pets too?. arXiv.10.18653/v1/P18-1205 ↗
24. Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. (December, January 27). DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP), Taipei, Taiwan.
25. Tiedemann, J. (2012, January 21–22). Parallel Data, Tools and Interfaces in OPUS. Proceedings of the International Conference on Language Resources and Evaluation (LREC), Istanbul, Turkey.
26. Banchs, R.E. (2012, January 8–14). Movie-DiC: A movie dialogue corpus for research and development. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), Jeju Island, Korea.
27. Serban, I.V., Sordoni, A., Bengio, Y., Courville, A.C., and Pineau, J. (2016, January 12–17). Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence AAAI, Phoenix, AZ, USA.10.1609/aaai.v30i1.9883 ↗
28. Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., and Pineau, J. (2016). How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv.10.18653/v1/D16-1230 ↗
29. Lowe, R., Noseworthy, M., Serban, I.V., Angelard-Gontier, N., Bengio, Y., and Pineau, J. (August, January 30). Towards an automatic Turing test: Learning to evaluate dialogue responses. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, BC, Canada.
30. Li, X.L., Liu, B., and Ng, S.K. (2010, January 9–11). Negative training data can be harmful to text classification. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cambridge, MA, USA.
31. Li, X.L., and Liu, B. (2005, January 3–7). Learning from positive and unlabeled examples with different data distributions. Proceedings of the European Conference on Machine Learning (ECML). Springer, Porto, Portugal.10.1007/11564096_24 ↗
32. Merdivan, E., Loghmani, M.R., and Geist, M. (2017, January 4–9). Reconstruct & Crush Network. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
33. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit10.1037/h0026256 ↗
34. Framewise phoneme classification with bidirectional LSTM and other neural network architectures10.1016/j.neunet.2005.06.042 ↗
35. Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
36. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
37. Sharma, S., El Asri, L., Schulz, H., and Zumer, J. (2017). Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation. arXiv.
38. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
39. Holzinger, A., Kieseberg, P., Weippl, E., and Tjoa, A.M. (2018). Current Advances, Trends and Challenges of Machine Learning and Knowledge Extraction: From Machine Learning to Explainable AI. Springer Lecture Notes in Computer Science LNCS 11015, Springer.10.1007/978-3-319-99740-7_1 ↗
40. Augmenting Statistical Data Dissemination by Short Quantified Sentences of Natural Language10.2478/jos-2018-0048 ↗
41. Holzinger, A., Kickmeier-Rust, M., and Mueller, H. (2019). KANDINSKY Patterns as IQ-Test for machine learning. Springer Lecture Notes LNCS 11713, Springer Nature.10.1007/978-3-030-29726-8_1 ↗