Common Pitfalls and Recommendations for Use of Machine Learning in Depression Severity Estimation: DAIC-WOZ Study
Source abstract
Abstract
The DAIC-WOZ dataset is a widely used benchmark for the task of depression severity estimation from multimodal behavioral data. Yet the reliability, reproducibility, and methodological rigor of published machine learning models remain uncertain. In this systematic review, we examined all works published through September 2025 that mention the DAIC-WOZ dataset and report mean absolute error as an evaluation metric. Our search identified 536 papers, of which 414 remained after deduplication. Following title and abstract screening, 132 records were selected for full-text review. After applying eligibility criteria, 66 papers were included in the quality assessment stage. Of these, only five met minimal reproducibility standards (such as clear data partitioning, model description, and training protocol documentation) and were included in this review. We found that published models suffer from poor documentation and methodology, and, inter alia, identified subject leakage as a critical methodological flaw. To illustrate its impact, we conducted experiments on the DAIC-WOZ dataset, comparing the performance of the model trained with and without subject leakage. Our results indicate that leakage produces significant overestimation of the validation performance; however, our evidence is limited to the audio, text, and combined modalities of the DAIC-WOZ dataset. Without leakage, the model consistently performed worse than a simple mean predictor. Aside from poor methodological rigor, we found that the predictive accuracy of the included models is poor: reported MAEs on DAIC-WOZ are of the same magnitude as the dataset’s own PHQ-8 variability, and are comparable to or larger than the variability typically observed in general population samples. We conclude with specific recommendations aimed at improving the methodology, reproducibility, and documentation of manuscripts. Code for our experiments is publicly available.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Mental Health via Writing · Digital Mental Health Interventions · Emotion and Mood Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Independent Researcher, 50-370 Wroclaw, Poland
Faculty of Information and Communication Technology, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland
Funding
Wroclaw University of Science and Technology
References (88)
Show source-provided references
1. World Health Organization (2025, September 26). Depression: Let’s Talk Says WHO, as Depression Tops List of Causes of Ill Health. Available online: https://www.who.int/news/item/30-03-2017–depression-let-s-talk-says-who-as-depression-tops-list-of-causes-of-ill-health.
2. Recognition and management of depression in general practice: Consensus statement10.1136/bmj.305.6863.1198 ↗
3. Editorial: What is depression?10.1111/jcpp.12844 ↗
4. American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders, American Psychiatric Publishing. [5th ed.].10.1176/appi.books.9780890425596 ↗
5. Yi, L., Xie, G., Li, Z., Li, X., Zhang, Y., Wu, K., Shao, G., Lv, B., Jing, H., and Zhang, C. (2023). Automatic depression diagnosis through hybrid EEG and near-infrared spectroscopy features using support vector machine. Front. Neurosci., 17.10.3389/fnins.2023.1205931 ↗
6. Automated Depression Detection from Text and Audio: A Systematic Review10.1109/JBHI.2025.3570900 ↗
7. Machine Learning-Based Prediction of Depressive Disorders via Various Data Modalities: A Survey10.1109/JAS.2025.125393 ↗
8. A systematic review on automated clinical depression diagnosis10.1038/s44184-023-00040-z ↗
9. Gratch, J., Artstein, R., Lucas, G., Stratou, G., Scherer, S., Nazarian, A., Wood, R., Boberg, J., DeVault, D., and Marsella, S. (2014, January 26–31). The Distress Analysis Interview Corpus of human and computer interviews. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
10. Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., and Messner, E.M. (2019, January 21). AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, New York, NY, USA.10.1145/3347320.3357688 ↗
11. PRISMA 202010.1016/j.jclinepi.2021.04.008 ↗
12. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers10.1148/ryai.2020200029 ↗
13. PointTransform Networks for automatic depression level prediction via facial keypoints10.1016/j.knosys.2024.111951 ↗
14. A multimodal fusion model with multi-level attention mechanism for depression detection10.1016/j.bspc.2022.104561 ↗
15. Atta, A., El Sayad, D., Ezzat, D., Amin, S., and El Gamal, M. (2024, January 19–21). Speech-Based Depression Detection System Optimized Using Particle Swarm Optimization. Proceedings of the 2024 6th Novel Intelligent and Leading Emerging Sciences Conference (NILES), Giza, Egypt.10.1109/NILES63360.2024.10753199 ↗
16. Shu, T., Zhang, F., and Sun, X. (2023, January 4–6). Gaze Behavior based Depression Severity Estimation. Proceedings of the 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML), Urumqi, China.10.1109/PRML59573.2023.10348319 ↗
17. Firoz, N., Beresteneva, O.G., and Aksyonov, S.V. (2023, January 10–16). Enhancing Depression Detection: Employing Autoencoders and Linguistic Feature Analysis with BERT and LSTM Model. Proceedings of the 2023 International Russian Automation Conference (RusAutoCon), Sochi, Russia.10.1109/RusAutoCon58002.2023.10272795 ↗
18. Williamson, J.R., Godoy, E., Cha, M., Schwarzentruber, A., Khorrami, P., Gwon, Y., Kung, H.T., Dagli, C., and Quatieri, T.F. (2016, January 16). Detecting Depression using Vocal, Facial and Semantic Communication Cues. Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, Amsterdam, The Netherlands.10.1145/2988257.2988263 ↗
19. Rethinking Inconsistent Context and Imbalanced Regression in Depression Severity Prediction10.1109/TAFFC.2024.3405584 ↗
20. Feng, K., and Chaspari, T. (2024, January 10–13). Robust and Explainable Depression Identification from Speech Using Vowel-Based Ensemble Learning Approaches. Proceedings of the 2024 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Houston, Texas, USA.10.1109/BHI62660.2024.10913638 ↗
21. Firoz, N. (2023, January 10–16). Detecting Depression from Text: A Gender Based Comparative Approach Using Machine Learning and BERT Embeddings. Proceedings of the 20th All-Russian Conference of Student Research Incubators, Tomsk, Russia.
22. Additive Cross-Modal Attention Network (ACMA) for Depression Detection Based on Audio and Textual Features10.1109/ACCESS.2024.3362233 ↗
23. Facial Depression Estimation via Multi-Cue Contrastive Learning10.1109/TCSVT.2025.3533543 ↗
24. Dinkel, H., Wu, M., and Yu, K. (2020). Text-based depression detection on sparse data. arXiv.
25. Zhao, Z., and Wang, K. (2022, January 11–15). Unaligned Multimodal Sequences for Depression Assessment From Speech. Proceedings of the 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK.10.1109/EMBC48229.2022.9871556 ↗
26. Firoz, N., Beresteneva, O.G., Vladimirovich, A.S., and Tahsin, M.S. (2023). Enhancing depression detection through advanced text analysis: Integrating BERT, autoencoder, and LSTM models. Res. Sq. Platf. LLC.10.21203/rs.3.rs-2782391/v1 ↗
27. Guangyao, S., Shenghui, Z., Bochao, Z., and Yubo, A. (2022, January 30–31). Multimodal depression detection using a deep feature fusion network. Proceedings of the Third International Conference on Computer Science and Communication Technology, ICCSCT 2022, Beijing, China.
28. Depression Scale Dictionary Decomposition Framework for Multimodal Automatic Depression Level Prediction10.1109/TCSVT.2025.3533480 ↗
29. Zheng, W., Yan, L., Gou, C., and Wang, F.Y. (2020, January 6–10). Graph Attention Model Embedded with Multi-Modal Knowledge for Depression Detection. Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME 2020), London, United Kingdom.10.1109/ICME46284.2020.9102872 ↗
30. D-ResNet-PVKELM: Deep neural network and paragraph vector based kernel extreme machine learning model for multimodal depression analysis10.1007/s11042-023-14351-y ↗
31. DepressionMLP: A Multi-Layer Perceptron Architecture for Automatic Depression Level Prediction via Facial Keypoints and Action Units10.1109/TCSVT.2024.3382334 ↗
32. Depression detection methods based on multimodal fusion of voice and text10.1038/s41598-025-03524-4 ↗
33. Hong, J., Lee, J., Choi, D., and Jung, J. (2025, January 18–20). LEFORMER: Liquid Enhanced Multimodal Learning for Depression Severity Estimation. Proceedings of the 2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS 2025), Madrid, Spain.10.1109/CBMS65348.2025.00091 ↗
34. Kumar, P., Misra, S., Shao, Z., Zhu, B., Raman, B., and Li, X. (March, January 28). Multimodal Interpretable Depression Analysis Using Visual, Physiological, Audio and Textual Data. Proceedings of the 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025), Tucson, Arizona, USA.10.1109/WACV61041.2025.00518 ↗
35. Multi-level spatiotemporal graph attention fusion for multimodal depression detection10.1016/j.bspc.2025.108123 ↗
36. Zhao, Z., Bao, Z., Zhang, Z., Cummins, N., Wang, H., and Schuller, B. (2020, January 4–8). Hierarchical Attention Transfer Networks for Depression Assessment from Speech. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9053207 ↗
37. RADIANCE: Reliable and interpretable depression detection from speech using transformer10.1016/j.compbiomed.2024.109325 ↗
38. Lu, J., Liu, B., Lian, Z., Cai, C., Tao, J., and Zhao, Z. (2022, January 11–14). Prediction of Depression Severity Based on Transformer Encoder and CNN Model. Proceedings of the 2022 13th International Symposium on Chinese Spoken Language Processing (ISCSLP), Singapore.10.1109/ISCSLP57327.2022.10038064 ↗
39. Text-guided multimodal depression detection via cross-modal feature reconstruction and decomposition10.1016/j.inffus.2024.102861 ↗
40. Recognition of Audio Depression Based on Convolutional Neural Network and Generative Antagonism Network Model10.1109/ACCESS.2020.2998532 ↗
41. Tang, J., Guo, Q., Sun, W., and Shang, Y. (2025, January 5–7). A Layered Multi-Expert Framework for Long-Context Mental Health Assessments. Proceedings of the 2025 IEEE Conference on Artificial Intelligence (CAI), Santa Clara, CA, USA.10.1109/CAI64502.2025.00080 ↗
42. Wei, P.C., Peng, K., Roitberg, A., Yang, K., Zhang, J., and Stiefelhagen, R. (2023, January 23–27). Multi-modal Depression Estimation Based on Sub-attentional Fusion. Proceedings of the Computer Vision—ECCV 2022 Workshops, Tel Aviv, Israel.10.1007/978-3-031-25075-0_42 ↗
43. Depression detection through transformers-based emotion recognition in multivariate time series facial data10.11591/ijai.v14.i2.pp1302-1310 ↗
44. Chen, M., Xiao, X., Zhang, B., Liu, X., and Lu, R. (2022, January 21–25). Neural Architecture Searching for Facial Attributes-based Depression Recognition. Proceedings of the 26th International Conference on Pattern Recognition (ICPR), Montreal, Quebec, Canada.10.1109/ICPR56361.2022.9956537 ↗
45. Yang, L., Jiang, D., He, L., Pei, E., Oveneke, M.C., and Sahli, H. (2016, January 16). Decision Tree Based Depression Classification from Audio Video and Language Information. Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, Amsterdam, The Netherlands.10.1145/2988257.2988269 ↗
46. Lin, L., Chen, X., Shen, Y., and Zhang, L. (2020). Towards automatic depression detection: A BiLSTM/1D CNN-based model. Appl. Sci., 10.10.3390/app10238701 ↗
47. Rasipuram, S., Bhat, J.H., Maitra, A., Shaw, B., and Saha, S. (July, January 30). Multimodal Depression Detection Using Task-oriented Transformer-based Embedding. Proceedings of the 2022 IEEE Symposium on Computers and Communications (ISCC), Rhodes Island, Greece.10.1109/ISCC55528.2022.9913044 ↗
48. Lau, C., Chan, W.Y., and Zhu, X. (November, January 31). Improving Depression Assessment with Multi-Task Learning from Speech and Text Information. Proceedings of the 55th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.
49. Rohanian, M., Hough, J., and Purver, M. (2019, January 15–19). Detecting Depression with Word-Level Multimodal Fusion. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-2283 ↗
50. Prakrankamanant, P., Watanabe, S., and Chuangsuwanich, E. (2025). Explainable Depression Detection using Masked Hard Instance Mining. arXiv.10.21437/Interspeech.2025-1181 ↗
51. Hu, J., Wang, A., Xie, Q., Ma, H., Li, Z., and Guo, D. (2025). Agentmental: An interactive multi-agent framework for explainable and adaptive mental health assessment. arXiv.
52. TTFNet: Temporal-Frequency Features Fusion Network for Speech Based Automatic Depression Recognition and Assessment10.1109/JBHI.2025.3574864 ↗
53. Multilevel depression status detection based on fine-grained prompt learning10.1016/j.patrec.2024.01.005 ↗
54. Sun, B., Zhang, Y., He, J., Yu, L., Xu, Q., Li, D., and Wang, Z. (2017, January 23). A Random Forest Regression Method With Selected-Text Feature For Depression Assessment. Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, New York, NY, USA.10.1145/3133944.3133951 ↗
55. An automatic diagnostic network using skew-robust adversarial discriminative domain adaptation to evaluate the severity of depression10.1016/j.cmpb.2019.01.006 ↗
56. Zhang, Y., Hu, W., and Wu, Q. (2021, January 14–15). Autoencoder Based on Cepstrum Separation to Detect Depression from Speech. Proceedings of the 3rd International Conference on Information Technologies and Electrical Engineering, New York, NY, USA.10.1145/3452940.3453038 ↗
57. A Multimodal approach for detection and assessment of depression using text, audio and video10.1007/s43657-023-00152-8 ↗
58. Niu, M., Chen, K., Chen, Q., and Yang, L. (2021, January 6–11). HCAG: A Hierarchical Context-Aware Graph Attention Model for Depression Detection. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, Ontario, Canada.10.1109/ICASSP39728.2021.9413486 ↗
59. Yang, L., Sahli, H., Xia, X., Pei, E., Oveneke, M.C., and Jiang, D. (2017, January 23). Hybrid Depression Classification and Estimation from Audio Video and Text Information. Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, New York, NY, USA.10.1145/3133944.3133950 ↗
60. Ishimaru, M., Okada, Y., Uchiyama, R., Horiguchi, R., and Toyoshima, I. (2023). A new regression model for depression severity prediction based on correlation among audio features using a graph convolutional neural network. Diagnostics, 13.10.3390/diagnostics13040727 ↗
61. Tang, J., and Shang, Y. (2024, January 28–31). Advancing Mental Health Pre-Screening: A New Custom GPT for Psychological Distress Assessment. Proceedings of the 2024 IEEE 6th International Conference on Cognitive Machine Intelligence (CogMI), Washington, DC, USA.10.1109/CogMI62246.2024.00030 ↗
62. Oureshi, S.A., Dias, G., Saha, S., and Hasanuzzaman, M. (2021, January 18–22). Gender-Aware Estimation of Depression Severity Level in a Multimodal Setting. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534330 ↗
63. Improving Depression Level Estimation by Concurrently Learning Emotion Intensity10.1109/MCI.2020.2998234 ↗
64. Parallel Multiscale Bridge Fusion Network for Audio–Visual Automatic Depression Assessment10.1109/TCSS.2024.3416029 ↗
65. Du, Z., Li, W., Huang, D., and Wang, Y. (2019, January 14–18). Encoding Visual Behaviors with Attentive Temporal Convolution for Depression Prediction. Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France.10.1109/FG.2019.8756584 ↗
66. Song, S., Shen, L., and Valstar, M. (2018, January 15–19). Human Behaviour-Based Automatic Depression Analysis Using Hand-Crafted Statistics and Deep Learned Spectral Features. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00032 ↗
67. Feature Augmenting Networks for Improving Depression Severity Estimation From Speech Signals10.1109/ACCESS.2020.2970496 ↗
68. Gong, Y., and Poellabauer, C. (2017, January 23). Topic Modeling Based Multi-modal Depression Detection. Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, New York, NY, USA.10.1145/3133944.3133945 ↗
69. Li, Y., Shao, S., Milling, M., and Schuller, B.W. (2025). Large language models for depression recognition in spoken language integrating psychological knowledge. Front. Comput. Sci., 7.10.3389/fcomp.2025.1629725 ↗
70. Se-GCaTCT: Correlation based optimized self-guided cross-attention temporal convolutional transformer for depression detection with effective optimization strategy10.1016/j.bspc.2025.108561 ↗
71. SIMMA: Multimodal Automatic Depression Detection via Spatiotemporal Ensemble and Cross-Modal Alignment10.1109/TCSS.2025.3542986 ↗
72. Improving depression prediction using a novel feature selection algorithm coupled with context-aware analysis10.1016/j.jad.2021.09.001 ↗
73. Spatial–Temporal Feature Network for Speech-Based Depression Recognition10.1109/TCDS.2023.3273614 ↗
74. Towards automatic text-based estimation of depression through symptom prediction10.1186/s40708-023-00185-9 ↗
75. Hong, S., Cohn, A., and Hogg, D. (2022, January 25–29). Using graph representation learning with schema encoders to measure the severity of depressive symptoms. Proceedings of the Tenth International Conference on Learning Representations, Online. Available online: https://eprints.whiterose.ac.uk/id/eprint/186629/.
76. Kang, A., Chen, J.Y., Lee-Youngzie, Z., and Fu, S. (2024). Synthetic Data Generation with LLM for Improved Depression Prediction. arXiv.
77. Syed, Z.S., Sidorov, K., and Marshall, D. (2017, January 23). Depression Severity Prediction Based on Biomarkers of Psychomotor Retardation. Proceedings of the AVEC ’17: 7th Annual Workshop on Audio/Visual Emotion Challenge, New York, NY.10.1145/3133944.3133947 ↗
78. Rathi, S., Kaur, B., and Agrawal, R.K. (2019, January 17–20). Enhanced Depression Detection from Facial Cues Using Univariate Feature Selection Techniques. Proceedings of the Pattern Recognition and Machine Intelligence, Kolkata, India.10.1007/978-3-030-34869-4_3 ↗
79. The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation10.7717/peerj-cs.623 ↗
80. Burdisso, S., Reyes-Ramírez, E., Villatoro-tello, E., Sánchez-Vega, F., Lopez Monroy, A., and Motlicek, P. (2024, January 21). DAIC-WOZ: On the Validity of Using the Therapist’s prompts in Automatic Depression Detection from Clinical Interviews. Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico.10.18653/v1/2024.clinicalnlp-1.8 ↗
81. Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. (2014, January 4–9). COVAREP—A collaborative voice analysis repository for speech technologies. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6853739 ↗
82. Cross-Sectional Reference Data From 29 European Countries for 6 Frequently Used Depression Measures10.1001/jamanetworkopen.2025.17394 ↗
83. Speech-Based Depression Assessment: A Comprehensive Survey10.1109/TAFFC.2024.3521327 ↗
84. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
85. Shin, S., Kim, J., Yu, Y., Lee, S., and Lee, K. (2021). Self-supervised transfer learning from natural images for sound classification. Appl. Sci., 11.10.3390/app11073043 ↗
86. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
87. Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., and Pantic, M. (2013, January 21). AVEC 2013: The continuous audio/visual emotion and depression recognition challenge. Proceedings of the AVEC ’13: 3rd ACM International Workshop on Audio/Visual Emotion Challenge, New York, NY, USA.10.1145/2512530.2512533 ↗
88. Tasnim, M., Ehghaghi, M., Diep, B., and Novikova, J. (2023). DEPAC: A Corpus for Depression and Anxiety Detection from Speech. arXiv.10.18653/v1/2022.clpsych-1.1 ↗