Oversight Is Psychiatry’s Next AI Challenge

By Manu Suresh Sharma, Chief Medical Informatics Officer, Behavioral Health Network, Hartford Healthcare. Assistant Clinical Professor of Psychiatry, Yale School of Medicine

The first wave of AI in psychiatry focused on questions like, “Could the technology work? Was it safe enough to use? Would clinicians adopt it? Would patients trust it? Would health systems pay for it?” While these questions remain relevant, we have gotten better at answering them. In recent years, health systems have widely adopted AI tools such as ambient documentation, patient portals, wellness apps, and chatbots that handle distress calls at any hour. The conversation has now shifted to, or should shift to, tougher questions about what happens after these systems are deployed. Who monitors them? How do we recognize when they fail?  

Mental health chatbots provide a valuable perspective on this broader challenge. Though they are not the only AI application in psychiatry, they highlight the stakes involved. Current evidence points to clear benefits of mental health chatbots. These tools are tireless, constantly available, and free of the social tensions often found in physical clinics. For many users, they effectively reduce barriers to access in a landscape plagued by ongoing, systemic gaps. However, this very accessibility and the belief that digital tools carry less stigma than human services require more careful examination. The traits that make chatbots appealing also allow them to occupy sensitive, vulnerable spaces in a patient’s life.  

Organizations are aware of these vulnerabilities and often adopt a “human in the loop” approach. The assumption they make is that adding a person to the workflow completely removes risk. However, this phrase often acts more like a sedative than a real solution, as the human in the loop can often be found napping.

To create an accountable system, oversight needs to begin long before the first patient sends a message. First, a health system must define the chatbot’s role with clear precision. Is it providing psychoeducation, coaching, triage, crisis support, or something resembling therapy? What topics is it not allowed to address? When should it stop giving advice and connect the user with a human? Second, we must define failure in the clearest clinical terms. Obvious failures such as overlooked suicidal intent are easy to define. However, it is important to clearly outline more subtle failures, such as validating delusional thoughts, inappropriate treatment suggestions, or overstating confidentiality. If these boundaries and definitions are not clearly outlined and converted into scripts, prompts, escalation rules, and audit criteria, no one will know what to monitor after launch.

AI should not enter psychiatry faster than our ability to notice when it fails.

After deployment, oversight must shift from theoretical safety to actual safety. Real users often ramble, joke, disguise their true feelings, contradict themselves, return multiple times, and can become attached to systems that respond warmly. In psychiatry, the nature of harm can build over time. It may not be evident in a single encounter but can manifest gradually across multiple interactions over weeks or months.  

Therefore, monitoring cannot be passive but must take the form of ongoing clinical reviews, random checks of transcripts, automated alerts for high-risk language, clinician evaluations of borderline cases, patient feedback reviews and equity checks across language and literacy levels.

AI can assist the clinical team in this area. A second AI system, serving as an “AI judge,” can analyze chatbot conversations at scale and flag unusual patterns for human review. When used thoughtfully, these judges can make monitoring practical by allowing clinicians to focus on the riskiest interactions instead of manually reviewing thousands of transcripts.  

But automated oversight may create a hall of mirrors. AI judges are not infallible. They can also overlook subtle clinical nuances, show bias, over- or under-identify issues, or underestimate risk in language groups they handle poorly. Thus, their role should be limited to triage. It is crucial for health systems using AI judges to validate them against human clinician reviews, monitor their sensitivity for high-risk failures in an ongoing manner, and audit cases of disagreement. Recalibrating AI judges whenever the underlying chatbot or patient population changes is also equally important.  

The escalation pathway is as critical as the chatbot’s responses. For example, if a user shows suicidal intent, who will get notified? Will there be a warm handoff to a crisis hotline or a care manager? How quickly will the case be reviewed? What happens outside of business hours? When the chatbot detects risk but directs the patient to a vague list of resources, it has not addressed the safety issue; it has merely automated a dead end.  

Finally, we should not assume we can eliminate risk completely. Human care can also fail. Clinicians can miss warning signs, and health systems can lose track of referrals, letting patients fall through the cracks. The point is not that humans are flawless and machines are dangerous. The argument is that chatbots and other AI tools can significantly change the scale and speed of failure.  

Therefore, in psychiatry and medicine overall, the next phase of AI should be judged less by how widely it is adopted and more by whether we can create accountable structures around these tools. The key questions we need to answer are, “Did it fail safely? Did anyone notice? Was there a response? Did the monitoring system learn?”  

Technology will definitely shape the future of psychiatry. But that future should not be left solely to technology. If chatbots and other AI tools are going to be present with patients during vulnerable moments, oversight can not be seen just as red tape but has to be conceptualized as clinical care itself.