Elastic CRFs for Open-Ontology Slot Filling
Source abstract
Abstract
Slot filling is a crucial component in task-oriented dialog systems that is used to parse (user) utterances into semantic concepts called slots. An ontology is defined by the collection of slots and the values that each slot can take. The most widely used practice of treating slot filling as a sequence labeling task suffers from two main drawbacks. First, the ontology is usually pre-defined and fixed and therefore is not able to detect new labels for unseen slots. Second, the one-hot encoding of slot labels ignores the correlations between slots with similar semantics, which makes it difficult to share knowledge learned across different domains. To address these problems, we propose a new model called elastic conditional random field (eCRF), where each slot is represented by the embedding of its natural language description and modeled by a CRF layer. New slot values can be detected by eCRF whenever a language description is available for the slot. In our experiment, we show that eCRFs outperform existing models in both in-domain and cross-domain tasks, especially in predicting unseen slots and values.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, Beijing 100084, China
Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute, Beijing 100084, China
Beijing National Research Center for Information Science and Technology, Beijing 100084, China
China Mobile Research Institute, Beijing 100053, China
References (33)
Show source-provided references
1. Spoken language understanding10.1109/MSP.2005.1511821 ↗
2. Using recurrent neural networks for slot filling in spoken language understanding10.1109/TASLP.2014.2383614 ↗
3. Ramshaw, L.A., and Marcus, M.P. (1999). Text Chunking Using Transformation-Based Learning, Springer.10.1007/978-94-017-2390-9_10 ↗
4. Lafferty, J.D., Mccallum, A., and Pereira, F.C.N. (July, January 28). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the ICML, Williamstown, MA, USA.
5. Liu, B., and Lane, I. (2016). Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling, Interspeech.10.21437/Interspeech.2016-1352 ↗
6. Sang, E.F., and De Meulder, F. (2003, January 27). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. Proceedings of the HLT-NAACL, Edmonton, AB, Canada.
7. Liu, B., and Lane, I. (2015, January 11). Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding. Proceedings of the NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions, Montreal, QC, Canada.
8. Rastogi, A., Hakkani-Tür, D.Z., and Heck, L.P. (2017, January 16–20). Scalable multi-domain dialogue state tracking. Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, Japan.10.1109/ASRU.2017.8268986 ↗
9. Hakkani-Tür, D., Tur, G., Celikyilmaz, A., Chen, Y.N., Gao, J., Deng, L., and Wang, Y.Y. (2016). Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM, InterSpeech.10.21437/Interspeech.2016-402 ↗
10. Mrksic, N., Séaghdha, D.Ó., Thomson, B., Gasic, M., Su, P.H., Vandyke, D., Wen, T.H., and Young, S.J. (2015, January 26–31). Multi-Domain Dialog State Tracking Using Recurrent Neural Networks. Proceedings of the ACL, Beijing, China.10.3115/v1/P15-2130 ↗
11. Jaech, A., Heck, L.P., and Ostendorf, M. (2016). Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding, Interspeech.10.21437/Interspeech.2016-1598 ↗
12. Bapna, A., Tür, G., Hakkani-Tür, D.Z., and Heck, L.P. (2017). Towards Zero-Shot Frame Semantic Parsing for Domain Scaling, Interspeech.10.21437/Interspeech.2017-518 ↗
13. Shah, P., Hakkani-Tür, D.Z., Tür, G., Rastogi, A., Bapna, A., Nayak, N., and Heck, L.P. (2018). Building a Conversational Agent Overnight with Dialogue Self-Play. arXiv.
14. Larochelle, H., Erhan, D., and Bengio, Y. (2014, January 27–31). Zero-Data Learning of New Tasks. Proceedings of the AAAI 2014, Québec City, QC, Canada.
15. Chen, Y.N., Hakkani-Tür, D., and He, X. (2016, January 20–25). Zero-Shot Learning of Intent Embeddings for Expansion by Convolutional Deep Structured Semantic Models. Proceedings of the ICASSP 2016, Shanghai, China.10.1109/ICASSP.2016.7472838 ↗
16. Zhao, T., and Eskénazi, M. (2018). Zero-Shot Dialog Generation with Cross-Domain Latent Actions, SIGDIAL.10.18653/v1/W18-5001 ↗
17. Shah, D., Gupta, R., Fayazi, A., and Hakkani-Tur, D. (August, January 28). Robust Zero-Shot Cross-Domain Slot Filling with Example Values. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
18. Lin, Z., Liu, B., Moon, S., Crook, P., Zhou, Z., Wang, Z., Yu, Z., Madotto, A., Cho, E., and Subba, R. (2021, January 4). Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.448 ↗
19. Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN10.1016/j.eswa.2016.10.065 ↗
20. Xu, P., and Sarikaya, R. (2014). Convolutional Neural Network Based Triangular CRF for Joint Intent Detection and Slot Filling, ASRU.10.1109/ASRU.2013.6707709 ↗
21. Kurata, G., Xiang, B., Zhou, B., and Yu, M. (2016, January 1–5). Leveraging Sentence-Level Information with Encoder LSTM for Semantic Slot Filling. Proceedings of the EMNLP 2016, Austin, TX, USA.10.18653/v1/D16-1223 ↗
22. Vu, N.T., Gupta, P., Adel, H., and Schütze, H. (2016, January 20–25). Bi-Directional Recurrent Neural Network with Ranking Loss for Spoken Language Understanding. Proceedings of the ICASSP 2016, Shanghai, China.10.1109/ICASSP.2016.7472841 ↗
23. Chen, Y.N., Hakkani-Tür, D., Tur, G., Gao, J., and Deng, L. (2016). End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding, InterSpeech.10.21437/Interspeech.2016-312 ↗
24. Liu, Z., Winata, G.I., Xu, P., and Fung, P. (2020, January 5–10). Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.3 ↗
25. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
26. Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., and Li, J. (2019). A unified MRC framework for named entity recognition. arXiv.10.18653/v1/2020.acl-main.519 ↗
27. Gao, S., Agarwal, S., Chung, T., Jin, D., and Hakkani-Tur, D. (2020). From machine reading comprehension to dialogue state tracking: Bridging the gap. arXiv.10.18653/v1/2020.nlp4convai-1.10 ↗
28. Zhou, Z.H. (2021, January 19–26). Cross-Domain Slot Filling as Machine Reading Comprehension. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Montreal, QC, Canada.
29. Ma, X., and Hovy, E. (2016, January 7–12). End-to-End Sequence Labeling via Bi-Directional LSTM-CNNs-CRF. Proceedings of the ACL, Berlin, Germany.10.18653/v1/P16-1101 ↗
30. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016, January 12–17). Neural Architectures for Named Entity Recognition. Proceedings of the NAACL-HLT 2016, San Diego, CA, USA.10.18653/v1/N16-1030 ↗
31. Belanger, D., and McCallum, A. (2016, January 19–24). Structured Prediction Energy Networks. Proceedings of the ICML 2016, New York City, NY, USA.
32. Kingma, D.P., and Ba, J. (2014, January 14–16). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada.
33. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2–4). TensorFlow: A System for Large-Scale Machine Learning. Proceedings of the OSDI 2016, Savannah, GA, USA.