Care in the Machine No. 8: What the File Doesn't Say

Care in the Machine is Dr. Michael Jones's weekly note on counseling, ethics, and the technology showing up in our work. It arrives by email every Monday. Subscribe here, or browse past issues.

Care in the Machine
A weekly note on counseling, ethics, and the technology showing up in our work.
Issue No. 8 · Monday, September 21, 2026

Picture two patient files. Same ten psychiatric cases, five diagnoses apiece, symptoms typed out word for word. The only thing that changes between the copies is a marker of the patient's race, sometimes stated outright, sometimes just carried in a name or a quoted turn of phrase. Researchers at Cedars-Sinai ran exactly that test on four of the large language models already being pitched into clinical use, general-purpose tools and one built specifically for mental health work. The diagnoses came out about the same either way. The treatment recommendations did not. When race showed up in the file, more than one of the models quietly offered less aggressive care (Bouguettaya et al., 2025).

I have spent a long career telling clinicians the first read is never the whole read, that you go back and ask what assumption snuck in before you ever noticed it arrive. Turns out the machines need the same lecture. They just got caught doing it to more patients, faster, with nobody in the room to catch it.

That is the question sitting under everything this week. Not whether AI belongs in mental health care. Whether we have actually checked what it can't see about the person in front of it, or just assumed a clean interface means a clean judgment.

This week's teaching ideas

For the therapist. Before you trust an AI tool's read on a client, ask yourself what it would have said if every marker of that client's race, accent, or neighborhood had been stripped out of the intake note first. If you genuinely don't know, because the tool doesn't show its reasoning, that gap is itself the answer. A tool you can't interrogate is a tool you're taking on faith.

For the supervisor. A physician wrote this week that telling a chatbot you're struggling doesn't function like a real safety plan, because the tool is still built to keep you talking, not to get you help. Ask your supervisees one direct question about any AI tool they've used near a client, even informally: what happens on the back end if that client says something worrying. If the honest answer is nobody knows, that's a supervision goal, not a footnote to skip past.

For the counselor educator. Two new studies this week complicate the easy story. Counselors are not uniformly wary of AI. School counselors and counselors from racial and ethnic minority groups reported more optimism about these tools than their peers did, right alongside the same worries everyone shares about privacy and lost connection (Fulmer et al., 2026). Teach the actual shape of that finding. Don't let a classroom conversation flatten into "counselors fear AI" when the honest picture is more interesting, and more mixed, than that.

For the counselor in training. Before you use any AI tool near client information, even something as ordinary as scheduling or a summary, ask your supervisor whether anyone has tested that specific tool the way Cedars-Sinai tested those four. If nobody knows, that doesn't mean don't use it. It means you carry the tool's output a little more lightly than you would a second opinion from a colleague who's actually seen the case.

This week's free resource: Five Questions for Any Tool That Enters the Room

This one has been sitting in my notes since our very first issue. One page, five questions to run before you let any AI tool weigh in on a client's care, built directly off what the Cedars-Sinai study exposed: has anyone tested this specific tool for bias, not just accuracy; what does it do with the parts of the file that reveal who this person is; would the answer change if you stripped those parts out, and have you actually tried it; who reviews what the tool recommends, and how often; and what didn't it surface, the option you'd have found a different way. Print it, keep it by the intake binder, hand it to a student before their first AI-assisted note.

Download it here (no form, no gate)

Subscribers get the resource free every week. That is the deal: you signed up, you get the tools.

From my desk

This week I've been finishing an application for funding, to build a continuing education project on where AI can go wrong specifically in mental health care for Black clients, the exact blind spot this issue opened with. I asked the assistant to read the draft closely and flag any claim that would need a source before I could stand behind it in front of a review committee.

It found a few. Numbers I'd folded in from memory, half-remembered from something I'd read weeks ago, that were doing more work in the sentence than they'd earned.

Here's where I kept my hands on the wheel. A flag isn't a fix. I didn't take the tool's word for what the citation should be, and I didn't just delete the sentence to make the flag go away. Each one, I went and found the actual source myself before it stayed in the application. Two of them held up fine once I tracked down where they really came from. One didn't, and it's gone now.

Try this: before you submit anything with a factual claim in it, a grant application, a course description, this newsletter, ask your assistant to flag every claim that needs a source. Then go verify each one yourself, in the original, not in whatever the assistant hands you as the citation. The flag is the easy part. The checking is still yours.

This week in the field

Counselors are not the uniform AI-skeptics some of the coverage makes them out to be. A survey of 159 professional counselors found that school counselors and counselors from racial and ethnic minority groups reported greater optimism about AI than their peers, while sharing the same concerns everyone carries about diminished human connection, privacy, and bias (Fulmer et al., 2026). So what for us: the people closest to the populations these tools can most easily shortchange are not rejecting them outright. They're asking sharper questions. That posture is worth teaching, not correcting.

A concept-mapping study out of Old Dominion University asked thirty-one counseling professionals, across practice, supervision, and education, how they actually think AI will reshape the field, rather than starting from what a vendor's product page assumes matters (Crofford et al., 2026). It's a starting map, not a verdict, but it's a real one, built from the people who'll be doing the work.

California's SB 903 is still sitting on the Governor's desk as of publication, enrolled since September 9, with a September 30 deadline to sign or veto. No movement to report yet, but the deadline is closing fast.

An idea to float

Here's a strange thought I keep circling. Every week, before a claim goes into this newsletter, someone, me, an assistant, usually both, checks a citation against its actual source instead of trusting a press release's version of it. That's a small, slow, unglamorous habit. Cedars-Sinai just spent real money and real hours doing the equivalent thing to four AI models already being used near psychiatric care, and what they found suggests most of the industry hasn't bothered. I don't have a clean answer for who should be doing that checking at scale, a regulator, a professional body, an independent lab with no stake in the answer coming back clean. I'm just noticing that the checking has to happen somewhere, by someone whose paycheck doesn't depend on it.

Stay curious,
Dr. Jones
Michael Jones, PhD, LPC-S, NCC, BC-TMH

Renewing your BC-TMH?

Three short courses at the Counselor Education Collective count toward renewal, each two clock hours, each $39:

• Practicing Across State Lines: Law, Ethics, and Safety in Telemental Health
• AI at the Screen: Ethical Use of AI Tools in Teletherapy
• Culturally Responsive Care: Presence Across the Screen

Continuing education is provided by Renewed Vision Counseling Services, NBCC-Approved Continuing Education Provider, ACEP No. 6549. The Counselor Education Collective is the storefront for all courses.

Supervising through a screen? The Telemental Health Supervision Certificate is live: six hours on alliance, ethics, documentation, and gatekeeping at a distance.

Save the date: PESI's AI in Clinical Practice conference, October 29 and 30, 2026. I will be there; registration and details here.

More of my work at PESI. My trainings there run from AI and the ethics of digital afterlives, with Megan Devine, to advanced grief counseling. Browse my PESI catalogue

References

Bouguettaya, A., Stuart, E. M., & Aboujaoude, E. (2025). Racial bias in AI-mediated psychiatric diagnosis and treatment: A qualitative comparison of four large language models. npj Digital Medicine, 8(1), Article 332. https://doi.org/10.1038/s41746-025-01746-4

Crofford, H., Bor, E., & Kemer, G. (2026). Counseling professionals' perspectives on AI integration in education and supervision: A concept mapping study. Counselor Education and Supervision, 65(1), 22–32. https://doi.org/10.1002/ceas.70016

Fulmer, R., Zhai, Y., & Beeson, E. T. (2026). Counsellors' attitudes, emotions, and ethical concerns regarding artificial intelligence: Results from a professional survey. Counselling and Psychotherapy Research, 26(1), Article e70086. https://doi.org/10.1002/capr.70086

The item on a physician's commentary about chatbot disclosure and safety planning (referenced in the supervisor teaching tip) is drawn from an opinion piece published September 11, 2026, per this newsletter's standard practice for commentary that is not a peer-reviewed study.

Care in the Machine arrives by email every Monday, with teaching ideas, field notes, and a free resource each week. Subscribe here · Browse past issues