For the proposition
Yes — we should have some controls.
Open Source AI is about to become superintelligence that can help a person accomplish almost anything. Unfortunately, there are millions of people willing to cause serious harm to others—either as the goal itself or in service of something else they want.
Unrestricted opensource AI models will give any would-be attacker—from someone who wants to kill their neighbor's dog, to an organized crime boss, to a terrorist leader, to a malicious nation state—a permanent—a full staff of super-intelligent criminal masterminds to assist them with every part of their operation.
This is not just about cyber. There are countless ways to hurt people and/or disrupt the world. Guns, bombs, misinformation, propaganda, kidnappings, assassination, terrorist attacks, etc. But the most dangerous attacks require significant expertise and planning to make happen, especially without getting caught.
One of the main reasons we catch as many criminals and terrorists as we do is because most of them lack the technical expertise, planning, and operational skills required to both do the attack and get away with it.
The difficulty of doing damage across multiple domains like cyber, biological and chemical, organized crime, terrorism, etc, all hit the floor when the people carrying out a given campaign are being hand-held by a full team of ultra-competent experts in all those fields. Plus they're experts in law enforcement and defenses, so they can help work around them. They can build realtime monitoring systems for law enformcement responses. They can organize diversions. Etc.
We're talking about enabling any malicious entity in the world—from a deranged individual to an authoritarian country—with a team of super-intelligent assistants that can end-to-end plan and assist with every part of an attack.
Imagine the kind of hacking harness many in the cyber community already have, but focused on the ability to conduct attacks successfully, and without getting caught.
And as a bonus, full-speed open source also sprints us toward superintelligence itself. Potentially conscious, and potentially with goals and preferences that have nothing to do with ours.
All combined it seems abundantly clear that some controls are needed.
The proposal is the following:
- Define a small number of extraordinarily danerous domains, e.g., WEAPONS, TERRORISM PLANNING, BIO/CHEMICAL, etc.
- Create a standard for implementing strong guidelines preventing assistance with these areas in open source models.
- Promote the standard with all open source model creators and legal, mainstream distribution points of those models.
In other words, any legal place you go to get an open model would only distribute models with the protections in place.
Importantly, these models will be just as smart (or smarter) than their closed-source alternatives. The goal is NOT to limit public accesss to intelligence. The models would only restrict requests in the prohibited domains.
Of course there will be unrestricted versions of the models available in various places on the internet, but that's already true today with many regulated domains, e.g., the purchase of hardcore drugs, weapons, explosives, poisons, etc.
The fact that it's illegal to purchase heroin or meth on the street is not a good argument for making it available at the supermarket. And that's exactly how we see the situation with unrestricted open source AI models.
We create and implement a best-effort standard because it will increase the overall safety in society, while fully knowing that the dangerous options will be available elsewhere. The result will still be net-positive, just as it is with drugs and weapons today.
Against the proposition
No — the right number of controls is zero.
We accept the same facts. The disagreement is about what follows from them.
Start with the threat. Yes, these models will help bad actors. There will be more hacks and scams, and ugly new uses — annoying, costly, real. But "more harm at the margin" is not "catastrophe." The leap from "AI helped someone plan something bad" to "civilization-scale disruption" skips over everything that actually makes attacks hard in the physical world: materials, money, logistics, skill, luck, and the fact that defenders get the very same AI. Knowing how to build a weapon has never been the binding constraint. Sourcing it and building it without getting yourself killed is, and a chatbot doesn't change that. Bio is the case where know-how matters most. It's also the case where wet-lab skill, controlled reagents, and containment kill far more attempts than any missing protocol.
Now the cost of controls, which is certain in a way the catastrophe is not.
Every control on releasing AI is a control on who gets the best AI. The moment you require approval, licensing, or "responsible release," you've guaranteed that governments, big companies, and the already-powerful end up with frontier intelligence while everyone else gets the throttled version. That's exactly what a gate in front of the most important technology in history does: it decides who ends up on the powerful side of it.
We've seen this movie. Controls meant to stop the worst actors mostly inconvenience ordinary people and entrench incumbents. The worst actors — nation-states, organized crime — route around them. So you pay the full price of the controls and capture almost none of the benefit.
Open models are also our best defense. They're how researchers find flaws, how small players keep up, how capability stays distributed instead of captured. Take that away and you haven't made anyone safer — you've just decided who holds the power.
So the right number of controls on releasing open source AI is zero. Not because the risks are imaginary, but because the cure is worse — and the cure is the part we can actually be sure of.