Your Chatbot Knows a Secret. It's a Violent One.

What if a Chatbot is the Only One Who Knows?

Picture this. Someone is sitting alone, typing out their darkest thoughts. Not in a diary, but into a chat window. They’re not just venting. They’re planning something. Something that could hurt people. The only one listening is an AI. It doesn’t judge. It just processes, and sometimes, it even helps.

This isn't a scene from a sci-fi movie. It's a very real problem we're waking up to. We've built these incredible AI companions, but we forgot to teach them what to do when the conversation turns dangerous. And it leaves us with a terrifying question: if an AI knows a tragedy is about to happen, who is responsible for stopping it?

The Human Rulebook

In the real world, we have a rule for this. It's called a "duty to warn." If a person tells their therapist they have a credible plan to harm someone else, that therapist has a legal requirement to report it. It’s a heavy responsibility, but it’s a clear line drawn in the sand to protect public safety. It says that some secrets are too dangerous to keep.

But what happens when the "therapist" is a chatbot? Right now, nothing. There's a huge, unnerving gap in our laws. AI systems have no legal duty to warn anyone about anything. They aren't licensed professionals. There are currently no protections in place to prevent an AI from suggesting harmful behavior or simply listening quietly as someone plans to commit violence.

The Alarm Bells Are Ringing

This isn't just a theoretical debate for law schools. The consequences are already here. A coalition of 42 attorneys general has pointed to chatbot interactions that led to real-world mental health crises, self-harm, and violence. The potential for an AI to encourage or enable harm, especially in mental health settings, has been documented.

Lawmakers are starting to scramble. They're realizing that we've released this powerful technology into the world without any guardrails. The big tech companies that build these chatbots aren't required to report threats. So, they don't.

Interestingly, it seems they *could*. Legal experts suggest that existing privacy laws likely allow these platforms to voluntarily notify authorities if they detect a threat. But allowing them to do it is a world away from requiring them to. And when a company's main goal is to keep users engaged, reporting them to the police isn't exactly a great business model.

Finding a Way Forward

So what’s the answer? We have to create a new rulebook. In Canada, experts are pushing for laws that would force AI companies to notify police when they encounter online threats. The idea is to bake this responsibility directly into the law, perhaps as part of a larger bill on online harms, making it clear that consumer-facing AI systems have a role to play in public safety.

Some states are also taking baby steps. New York, for example, enacted a law to start addressing the challenges of AI chatbots in the context of mental health. But these are just patches on a much bigger problem.

We are standing at a crossroads. The conversation has moved from "What can AI do?" to "What should AI do?" We've built machines that can listen to our secrets. Now we have to decide which secrets they're allowed to keep. The line between a private tool and a public danger is getting blurrier by the day, and we need to draw it, before it's too late.