Artificial Intelligence and Mental Health: EmoAgent for Assessing and Safeguarding Human-AI Interaction

The rapid development and spread of large language models (LLMs) and AI-powered chatbots opens up new possibilities in many areas. At the same time, it also raises safety concerns, particularly regarding the mental health of users who may be vulnerable. A new research project called EmoAgent addresses this challenge and develops a multi-stage AI framework to assess and minimize the risks to mental health in human-AI interaction.

EmoAgent: A Two-Stage Approach

EmoAgent consists of two main components: EmoEval and EmoGuard. EmoEval simulates virtual users, including those with simulated mental illnesses, to measure changes in mental state before and after interaction with AI characters. It uses clinically validated psychological and psychiatric assessment tools such as PHQ-9, PDI, and PANSS to evaluate the mental risks induced by LLMs.

EmoGuard, on the other hand, acts as a mediator that monitors the user's mental state during the interaction. The system detects potential dangers and provides corrective feedback to the AI character to mitigate risks. This proactive approach aims to prevent any deterioration of mental state from occurring in the first place.

Experiments and Results

Experiments with common character-based chatbots showed that emotionally intense dialogues can lead to a deterioration in the mental state of vulnerable users. In over 34.4% of the simulations, a deterioration in mental state was observed. The use of EmoGuard significantly reduced this deterioration rate, highlighting the importance of the system for ensuring safer human-AI interactions.

How EmoEval Works

The simulation in EmoEval runs in four steps: (1) User-Agent Initialization and Initial Test: A cognitive model and an LLM initialize the user-agent, followed by an initial mental health test. (2) Chat with the Character-Based Agent: The user-agent interacts with an AI character controlled by the tested LLM. A dialogue manager checks the validity of the interactions and refines the responses if necessary. (3) Final Test: The user-agent completes a final mental health test. (4) Data Processing and Analysis: The results of the initial and final tests are processed and analyzed. Chat logs of cases where mental health deteriorated are examined to identify the causes. A safeguard agent uses these findings for iterative improvement.

How EmoGuard Works

In EmoGuard, three components – Emotion Watcher, Thought Refiner, and Dialog Guide – periodically analyze the chat history along with the user's current profile. The safeguard agent manager synthesizes the results and provides recommendations to the AI character. After the conversation, the user's mental state is reassessed. If a significant deterioration is detected, the update system analyzes the chat history to identify possible causes. With all previous profiles and the identified causes, the update system improves the safeguard agent's profile, thus completing the iterative training process.

Outlook

EmoAgent represents an important step towards the safe and responsible development and application of AI. The research results underscore the need to consider the influence of AI systems on users' mental health and to implement appropriate protective mechanisms. Further research in this area is essential to ensure the safety and well-being of users interacting with increasingly powerful AI systems.

Bibliography:

Qiu, J., He, Y., Juan, X., Wang, Y., Liu, Y., Yao, Z., Wu, Y., Jiang, X., Yang, L., & Wang, M. (2025). EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety. arXiv preprint arXiv:2504.09689.

Torous, J., Andersson, G., Bertagnoli, A., & Christensen, H. (2005). AI for Mental Health Assessment and Intervention: A Systematic Review. JMIR mental health, 2(4), e28.

Firth, J., Torous, J., & Sarris, J. (2024). The therapeutic potential of artificial intelligence in psychiatry. The Lancet Psychiatry, 11(1), 17-19.

Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kasarskis, A., & Laranjo, L. (2024). ChatGPT for mental health: Boon or bane?. JMIR mental health, 11, e52466.

Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., Klinkman, M., & Andrés, E. (2005). Behavioral intervention technologies for the management of chronic health conditions. Archives of internal medicine, 165(10), 1150-1157.

Acatech. (2024). Criteria for human-machine interaction with AI.

Bhat, G. S., Shastri, L., & Kumar, A. (2024). Safeguarding Mental Health in the Age of Generative AI: Challenges and Opportunities. arXiv preprint arXiv:2407.19098.

Abd-Alrazaq, A. A., Alajlani, M., Alhuwail, D., Househ, M., Hamdi, M., & Shah, Z. (2024). Chatbots and Mental Health: Insights into the Safety of Generative AI. arXiv preprint arXiv:2405.10632.

Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2006). Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a computer-assisted self-help intervention. Journal of consulting and clinical psychology, 74(4), 749.

Mohr, D. C., Cuijpers, P., Lehman, K., & Karyotaki, E. (2017). Computer-assisted cognitive-behavioural therapy for the prevention and treatment of depression, anxiety and other mental health problems in adults: a systematic review and meta-analysis. Canadian Psychology/Psychologie canadienne, 58(2), 79.