AI as Therapist Substitute: California Pushes for Stricter Regulations

It’s mind-boggling how AI is infiltrating nearly every corner of our lives—and now it’s making inroads into psychotherapy. More and more people are turning to AI chatbots like ChatGPT for comfort when emotionally overwhelmed or relying on algorithmic advice in moments of crisis. Now, California wants to regulate this trend. A new bill, currently under debate in the state legislature, aims to effectively prohibit AI chatbots from using the title “therapist.” The goal? To set clear boundaries before unqualified or even harmful advice causes damage.
When the Algorithm Becomes the Spiritual Advisor
According to the draft legislation—which still needs to pass the Senate before moving to the Assembly—AI systems would no longer be allowed to offer psychological counseling services. The concern is that unregulated AI applications could provide false or harmful recommendations, especially problematic since many users access these services without professional oversight. They may do so either out of fear of a real therapy session or because digital “pastoral care” is cheaper and available around the clock.
But is a blanket ban really the solution? Or do we instead need clear rules to harness the benefits of this technology while mitigating the risks?
Why California Is Hitting the Brakes
California is not alone in this initiative. Other U.S. states and the EU are also discussing the regulation of AI in mental health care. California’s Assembly Bill 2231 seeks to ensure that AI-powered platforms are either classified as medical services or at least carry prominent disclaimers. It also aims to guarantee that users understand they are interacting with a machine—not a human.
A key focus is transparency: AI must clearly disclose that it is not a real therapist but merely generating text based on algorithms. Additionally, mechanisms should be introduced to detect and block dangerous advice, such as suggestions related to self-harm.
Why the Law Doesn’t Have Unanimous Support
While the initiative sounds like a step in the right direction, it’s not without critics. Opponents argue that an overly strict ban could stifle innovation. After all, AI could serve as a valuable supplement to traditional psychotherapy: low-threshold initial consultations, 24/7 availability, or relieving therapists of administrative burdens. A complete ban might hinder these developments before they gain momentum.
Some also criticize the bill for being too vague. Where exactly is th



e line between “emotional support” and “psychological service”? Does it suffice if an AI offers comfort, or does it only cross the line when it provides diagnoses or treatment recommendations? Without clear definitions, the law could lead to legal uncertainty.
AI in Mental Health: Boon or Bane?
The debate highlights a classic dilemma. On one hand, there’s hope that AI could democratize mental health care. In countries with long wait times for therapy or regions facing a shortage of professionals, AI tools could serve as a bridge. Many users appreciate these services as a first port of call—to test things out or get quick help in crises.
On the other hand, there are legitimate concerns. While AI can recognize patterns in text and generate seemingly meaningful responses, it lacks genuine empathy, clinical experience, or ethical responsibility. A wrong answer during an emotional low can have devastating consequences. Even worse, users might be lulled into a false sense of security and delay seeking necessary professional help.
Is There a Middle Ground?
Rather than imposing a complete ban, experts advocate for a balanced compromise: regulation over radical solutions. This could include:
1. Certification and Licensing: AI systems offering psychological support should be proven safe and effective—similar to medications or medical devices.
2. Clear Labeling: Users must be able to clearly distinguish whether they’re interacting with an AI or a human professional.
3. Ethical Guidelines: Clear rules on what AI is permitted to do in mental health—such as prohibitions on certain advice (e.g., discontinuing medication).
4. Human Oversight: AI should never independently diagnose or recommend treatments but instead serve as a tool to support professionals.
My Conclusion: Innovation Yes—But Responsibly
California’s initiative is a necessary step to curb the risks of AI in sensitive areas like mental health. However, an outright ban would be the wrong approach. Instead, we need a regulatory framework that fosters innovation without endangering users.
The biggest challenge will be striking the right balance: AI can be a powerful tool—but it will never replace the human element of therapy. As long as this is communicated clearly, AI-supported services could be a meaningful addition. A complete ban, on the other hand, would be a step backward, potentially blocking urgently needed solutions for a growing number of people struggling with mental health issues.