The effectiveness of a sentence completion test for depression screening using large language models

Abstract

Depressive symptoms pose a significant mental health challenge globally, including in Thailand. The aim of this study was to assess the effectiveness of a sentence completion test for depression using large language models (LLMs). To improve objectivity and reduce bias in assessments, this study detects and classifies the trends in depression, modernizing the screening process with a newly developed depression sentence completion test. This research examines the four key areas: 1) family, 2) society, 3) health, and 4) self-concept among 373 participants, aged 20 to 40. Additionally, four models were applied to test: 1) LLAMA 3.1-8B, 2) Gemma2-9B, 3) Qwen2-7B, and 4) Typhoon1.5-7B. The result revealed that health, self-concept, and DIFF (Difference) were strongly related to sentiment levels with strong positive values of 0.48, 0.49, and 0.54, respectively, which might mean that they are significant indicators of depression risk. Family and society had positive but lower values of 0.27 and 0.19, respectively. Statistical validation confirms model reliability with a 0.78 lower bound accuracy (p ≤ .05). In the evaluation of all Thai-compatible LLMs, random forest models consistently performed better than decision tree classifiers in the classification of depression risk. LLaMA3.1 and Gemma2 produced the highest sensitivity. Ethical problems have to be considered when using LLMs in mental health. Embedding diverse populations and dynamic updating to sample data in future studies will assure greater accuracy and generalization across different demographic groups.