Care in the Machine No. 6: The Right First Question

Care in the Machine is Dr. Michael Jones's weekly note on counseling, ethics, and the technology showing up in our work. It arrives by email every Monday. Subscribe here, or browse past issues.

Care in the Machine
A weekly note on counseling, ethics, and the technology showing up in our work.
Issue No. 6 · Monday, September 7, 2026

A reader who runs a play therapy practice for children and teens wrote me a few weeks back, and I have been sitting with her note ever since.

She said she feels overwhelmed. Kids come into her office having spent the week talking to a chatbot, or a companion app, or whatever new voice showed up on their phone since the last session, and she cannot keep up with what any of it is. She does not know what counts as normal anymore, whether a between-session "conversation" with an AI is a red flag or just how growing up looks now, or whether she should even be asking about it in the first place. She asked me straight out: will you address this, or point me somewhere?

Here is what I want to say to her, and to the rest of you who are somewhere in that same overwhelmed spot, because I suspect that is most of you. You do not have to become an expert in large language models to do this work well. Nobody is. You have to get good at three questions.

That is most of what this issue is about.

This week's teaching ideas

For the therapist. Ask about scams before you ask about screens. AI voice cloning and deepfakes have made scam calls sound exactly like a relative in crisis, and a team writing in Psychiatric Times late last month traced how that risk lands hardest on clients living with psychosis (Isa et al., 2026). The number underneath is peer-reviewed and sobering: across a meta-analysis of 35 studies, roughly one in five adults with a psychotic disorder experienced victimization within a single year (de Vries et al., 2019). Sometimes a scam fuses with existing delusional content and tips someone into decompensation. Add one plain question to intake: has anyone you didn't know asked you for money, gift cards, cryptocurrency, or a favor lately, especially someone claiming to be in trouble. Identify a trusted contact for monitoring if the answer is concerning. And if a scam has already braided itself into a client's delusional content, stay compassionate and curious rather than reaching for the debunking voice. Nobody was ever argued out of a belief that started as a genuine attempt to help someone they loved.

For the supervisor. Put AI on the supervision agenda by name. A recent review in Behavioral Sciences mapped what we know about clinical supervision onto the AI tools now arriving in supervision itself, chatbots, pattern detection, session tracking, and landed on the supervisory working alliance as the thing to protect (Brinck et al., 2026). The practical move is one standing question: which AI tools touched this case between our meetings? Documentation assistants, transcript summarizers, a treatment-plan draft, your supervisee may not think of any of it as worth reporting. You cannot supervise what never gets named. And when a tool did touch the work, the follow-up is the question that builds judgment: what did you keep, what did you change, and how did you decide?

For the counselor educator. Assume your students are already asking chatbots for references, because they are. In a controlled experiment published in JMIR Mental Health, nearly two thirds of the citations a leading model produced for mental health topics were fabricated or inaccurate (Linardon et al., 2025), and an earlier test in Scientific Reports found even the stronger model inventing nearly one in five of its citations (Walters & Wilder, 2023). The teaching move is a live demonstration: have each student bring one AI-suggested source to class, then resolve it together at doi.org. Some will resolve cleanly. Some will not exist. The ones that half-exist, real journal, wrong authors, wrong year, teach the deepest lesson of all, which is that "I found a citation" and "I verified a citation" are different claims.

For the counselor in training. The FDA is now piloting AI-delivered therapy under real regulatory watch. Limbic, just selected as the first AI-led mental health company in the agency's TEMPO pilot, will deliver twenty-minute voice-AI CBT sessions by phone to Medicare beneficiaries with clinically significant depression or anxiety, with clinicians overseeing every session and receiving real-time safety alerts (reported by STAT News, September 3, 2026). Whatever you make of it, this is the shape of the field you are entering. Learn what "clinician in the loop" actually means, because the phrase can describe anything from a supervisor who reviews every session to a dashboard nobody opens. The question worth carrying into your practicum interviews: when the AI flags a risk, who is on the other end, and how fast do they act?

This week's free resource

Three Questions to Open the AI Conversation. One page, printable. The three questions this issue is built around, ready for any intake or check-in: scam risk, chatbot companionship, and belief drift, each with a short note on why it matters and how to ask it plainly.

Download it here (no form, no gate)

Subscribers get the resource free every week. That is the deal: you signed up, you get the tools.

From my desk

Midway through this week's research scan, scite, the tool I use to verify every study before it earns a place in this newsletter, hit its monthly usage limit and went dark. The standard did not go dark with it. Everything added to this issue after the outage got verified the slower way: resolve the DOI at doi.org, read the publisher's own record, and check for retraction or correction notices before drafting a single word from it.

Here is what the slow way caught. An earlier draft of this issue said one in five people with psychosis are victimized by a scam each year. The study behind that number measures victimization overall, not scams specifically. Reading the actual record, rather than my memory of it, is the reason that sentence got fixed before it reached you.

Try this: the next time your verification tool is down, or you never had one to begin with, three free steps still stand. Paste the DOI into doi.org and make sure it resolves. Read the publisher's page, the real record, and skip the press release. Look for an editorial notice before you cite. The tool is the convenience. The habit is the protection.

This week in the field

A chatbot can sound like a good therapist. It cannot consistently be one. Sixty-five university students with mild to moderate distress each did one thirty-minute CBT session with a chatbot, scored on the Cognitive Therapy Scale and measured against a meta-analysis of eighteen human-therapist studies (Herbener et al., 2026). The chatbot hit minimum acceptable competence in only 30 of 65 sessions, forty-six percent. It was strong on communicating empathy and validating feelings, and weak on identifying a client's key beliefs, guiding self-discovery, and adapting to the individual person in front of it. Against the strongest human-therapist studies specifically, the gap closed. Against medium-quality studies, the chatbot actually scored higher. So what for us: the chatbot can perform the surface moves of a relationship, warmth, validation, a good bedside manner. What it cannot reliably do is stay with one particular person long enough to notice what makes them different from the last sixty-four. That is not a knock on the technology. It is a description of what relational competence actually requires, and it is worth saying out loud to anyone who tells you the difference between a bot and a therapist is closing.

Counselors still default to color-blindness, even in the cases they chose to publish as their best work. A review of 172 published case studies from the journal Clinical Case Studies, 86 involving clients of color and 86 involving a White comparison group, found counselors treated a client's race or ethnicity as clinically relevant in only 38 percent of cases involving clients of color, and 2 percent of cases involving White clients (Margolin, 2026). When race did come up, counselors routinely imposed their own prior assumptions onto clients of color, while White clients' behavior was explained through individual or family dynamics instead. So what for us: this is not a survey of what counselors say they believe. It is a look at the field's own published exemplars, the cases good enough that someone chose to write them up and put them in a journal, and color-blindness still won most of the time. Naming race and culture explicitly, as its own clinical thread rather than something you reach for only when a case seems to demand it, is a skill worth teaching directly rather than assuming it develops on its own.

A synthesis of 88 studies lands on "promising, not proven." Following PRISMA 2020 guidelines, the authors searched five databases through the end of 2025 and narratively synthesized 88 eligible studies on AI in mental health published between 2019 and 2025 (Alhalawany et al., 2026). AI showed promise across diagnostic support, risk prediction, treatment planning, symptom monitoring, and expanding access to care, and it is also reshaping how counselors and clinicians get trained. The authors' own conclusion: most systems remain experimental with limited external validation, and responsible use depends on interdisciplinary collaboration, transparent development, and real governance, explicitly framing AI as a complement to human expertise. So what for us: I like this one precisely because it refuses to oversell in either direction. Eighty-eight studies in, the honest headline is still "promising, and governance has to keep pace," which is a more useful sentence than anything the hype pieces or the doom pieces are currently offering.

An idea I'm floating

The chatbot in the Herbener study missed the same thing, in its own way, that the counselors in the Margolin study missed. Neither one could stay with the actual, particular person long enough to see what made that person different from the pattern. The machine defaulted to a script. The humans defaulted to a category, or to no category at all when the client was White, which is its own kind of default. I keep wondering whether the real test of relational competence, human or machine, was never "can you perform empathy." It might be "can you see the one person in front of you instead of the group you've already decided they belong to." I do not have a tidy answer for how you train that into either a language model or a licensed clinician with twenty years of habits. I am not sure it should be tidy. Write back. I read every one.

Stay curious,
Dr. Jones
Michael Jones, PhD, LPC-S, NCC, BC-TMH

Renewing your BC-TMH?

Three short courses at the Counselor Education Collective count toward renewal, each two clock hours, each $39:

• Practicing Across State Lines: Law, Ethics, and Safety in Telemental Health
• AI at the Screen: Ethical Use of AI Tools in Teletherapy
• Culturally Responsive Care: Presence Across the Screen

Continuing education is provided by Renewed Vision Counseling Services, NBCC-Approved Continuing Education Provider, ACEP No. 6549. The Counselor Education Collective is the storefront for all courses.

Supervising through a screen? The Telemental Health Supervision Certificate is live: six hours on alliance, ethics, documentation, and gatekeeping at a distance.

Save the date: PESI's AI in Clinical Practice conference, October 29 and 30, 2026. I will be there; registration and details here.

More of my work at PESI. My trainings there run from AI and the ethics of digital afterlives, with Megan Devine, to advanced grief counseling. Browse my PESI catalogue

References

Alhalawany, R. M., Khatatbeh, Y. M., & Jawkhab, A. A. (2026). Artificial intelligence applications in mental health: A systematic review of clinical practice, educational transformation, and ethical governance. Healthcare, 14(17), Article 2721. https://doi.org/10.3390/healthcare14172721

Brinck, E. A., Soldner, J. L., Kuo, H. J., Sabella, S. A., Landon, T. J., Bernacchio, C. P., & Boland, E. A. (2026). An AI perspective on counseling supervision. Behavioral Sciences, 16(6), Article 1038. https://doi.org/10.3390/bs16061038

de Vries, B., van Busschbach, J. T., van der Stouwe, E. C. D., Aleman, A., van Dijk, J. J. M., Lysaker, P. H., Arends, J., Nijman, S. A., & Pijnenborg, G. H. M. (2019). Prevalence rate and risk factors of victimization in adult patients with a psychotic disorder: A systematic review and meta-analysis. Schizophrenia Bulletin, 45(1), 114-126. https://doi.org/10.1093/schbul/sby020

Herbener, A. B., Zachariae, R., Klincewicz, M., Thøgersen, M. B., Hermann, M. R., & Damholdt, M. F. (2026). Exploring the therapeutic competencies of large language models: Observational study and comparison with meta-analytical estimates for human therapists. Computers in Human Behavior: Artificial Humans, 9, Article 100347. https://doi.org/10.1016/j.chbah.2026.100347

Isa, S., Moukaddam, N., & Wojcik, K. (2026, August 25). Exploitation of individuals with psychosis. Psychiatric Times. https://www.psychiatrictimes.com/view/exploitation-of-individuals-with-psychosis

Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., & Messer, M. (2025). Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: Experimental study. JMIR Mental Health, 12, Article e80371. https://doi.org/10.2196/80371

Margolin, L. (2026). Racism in counseling and psychotherapy: How it continues. Journal of Humanistic Psychology. Advance online publication. https://doi.org/10.1177/00221678261480404

Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, Article 14045. https://doi.org/10.1038/s41598-023-41032-5

Care in the Machine arrives by email every Monday, with teaching ideas, field notes, and a free resource each week. Subscribe here · Browse past issues