Should we have some level of control over open source AI models?
A good-faith debate — and your vote.
This is a real argument over one hard question: should the most powerful open-source AI models be released to everyone with no controls — or should industry, experts, and government agree on some basic guardrails first?
It's a real disagreement between people who agree on almost all of the facts. So instead of shouting past each other, we wrote down what we both accept as true, stated each side as fairly as we could, and put it to you. Read both sides on the Arguments tab, then vote for the one you actually believe — and tell us why.
Where both sides agree
The software running our critical infrastructure is still, after decades of checking and flexing, horribly vulnerable.
The new AI models are becoming genuinely surprising — and concerning — in their ability to find and exploit vulnerabilities.
The chances of serious compromises of critical systems are very high, and increasing.
There's also real danger in handing control of AI to non-democratic governments, or to unelected industry leaders who could exert control over regular people.
The ideal world is everyone having access to the highest quality of intelligence — so the rich and powerful don't get much smarter AI than everyone else.
The proposition
We should have some level of control over open source AI models.
We should have some level of control over open source AI models.
For the proposition
Yes — we should have some controls.
In short
Open models are about to put superintelligent capability in anyone's hands — including the millions of people who want to hurt others.
What stops most attackers today is a lack of expertise and planning. A full team of expert AI assistants erases that barrier at once — across cyber, bio, weapons, and more.
The fix is narrow: agree on a standard that blocks help with a few extreme domains — weapons, bio/chem, terrorism planning — at the legal places people get these models. They stay just as smart everywhere else.
Unrestricted versions will still leak, the way drugs and weapons already do. We don't sell those at the supermarket either — a best-effort standard is still net-positive.
Open Source AI is about to become superintelligence that can help a person accomplish almost anything. Unfortunately, there are millions of people willing to cause serious harm to others—either as the goal itself or in service of something else they want.
Unrestricted opensource AI models will give any would-be attacker—from someone who wants to kill their neighbor's dog, to an organized crime boss, to a terrorist leader, to a malicious nation state—a permanent—a full staff of super-intelligent criminal masterminds to assist them with every part of their operation.
This is not just about cyber. There are countless ways to hurt people and/or disrupt the world. Guns, bombs, misinformation, propaganda, kidnappings, assassination, terrorist attacks, etc. But the most dangerous attacks require significant expertise and planning to make happen, especially without getting caught.
One of the main reasons we catch as many criminals and terrorists as we do is because most of them lack the technical expertise, planning, and operational skills required to both do the attack and get away with it.
The difficulty of doing damage across multiple domains like cyber, biological and chemical, organized crime, terrorism, etc, all hit the floor when the people carrying out a given campaign are being hand-held by a full team of ultra-competent experts in all those fields. Plus they're experts in law enforcement and defenses, so they can help work around them. They can build realtime monitoring systems for law enformcement responses. They can organize diversions. Etc.
We're talking about enabling any malicious entity in the world—from a deranged individual to an authoritarian country—with a team of super-intelligent assistants that can end-to-end plan and assist with every part of an attack.
Imagine the kind of hacking harness many in the cyber community already have, but focused on the ability to conduct attacks successfully, and without getting caught.
And as a bonus, full-speed open source also sprints us toward superintelligence itself. Potentially conscious, and potentially with goals and preferences that have nothing to do with ours.
All combined it seems abundantly clear that some controls are needed.
The proposal is the following:
Define a small number of extraordinarily danerous domains, e.g., WEAPONS, TERRORISM PLANNING, BIO/CHEMICAL, etc.
Create a standard for implementing strong guidelines preventing assistance with these areas in open source models.
Promote the standard with all open source model creators and legal, mainstream distribution points of those models.
In other words, any legal place you go to get an open model would only distribute models with the protections in place.
Importantly, these models will be just as smart (or smarter) than their closed-source alternatives. The goal is NOT to limit public accesss to intelligence. The models would only restrict requests in the prohibited domains.
Of course there will be unrestricted versions of the models available in various places on the internet, but that's already true today with many regulated domains, e.g., the purchase of hardcore drugs, weapons, explosives, poisons, etc.
The fact that it's illegal to purchase heroin or meth on the street is not a good argument for making it available at the supermarket. And that's exactly how we see the situation with unrestricted open source AI models.
We create and implement a best-effort standard because it will increase the overall safety in society, while fully knowing that the dangerous options will be available elsewhere. The result will still be net-positive, just as it is with drugs and weapons today.
Convinced? Cast your vote.
Against the proposition
No — the right number of controls is zero.
In short
We accept the same facts. The harm is real but stays at the margin — knowing how was never the barrier; sourcing it, building it, and not getting caught is, and a chatbot doesn't change that. Defenders get the same AI.
The standard can't work on open models: anyone holding the weights can fine-tune the guardrails off in an afternoon, so it only restrains the people who were never going to cause harm.
The drugs-and-weapons analogy backfires — bans bite because those things are physical and seizable; a model is a file you can't seize, and one leak is permanent and global.
The cost is certain where the catastrophe is speculative: every control on release decides who gets the best AI, entrenching governments and incumbents. Better to harden defenses and keep capability distributed. The right number of controls is zero.
We accept the same facts. The disagreement is about what follows from them.
Start with the threat. Yes, these models will help bad actors. There will be more hacks and scams, and ugly new uses — annoying, costly, real. But "more harm at the margin" is not "catastrophe." The leap from "AI helped someone plan something bad" to "civilization-scale disruption" skips over everything that actually makes attacks hard in the physical world: materials, money, logistics, skill, luck, and the fact that defenders get the very same AI. Knowing how to build a weapon has never been the binding constraint. Sourcing it and building it without getting yourself killed is, and a chatbot doesn't change that. Bio is the case where know-how matters most. It's also the case where wet-lab skill, controlled reagents, and containment kill far more attempts than any missing protocol.
And whatever the model hands the attacker, it hands the defender too — faster patching, better detection, more eyes on every flaw. The pro case quietly assumes the attacker gets superintelligence and the defender stays frozen in 2026. That's not how any previous tool has worked, and it's not how this one will either.
Now look hard at the proposal itself, because this is where the case breaks. The thing that makes a model "open" is that you hold the weights. And once you hold the weights, the guardrails come off in an afternoon — a modest fine-tune strips the refusals and hands back the same intelligence with the safety training sanded away. This isn't a loophole to be patched; it's the math. A "best-effort standard" on open weights restrains exactly the people who were never going to cause harm, and does nothing to the one who showed up to cause it. He downloads the model, removes the refusals, and stands precisely where he'd stand with no standard at all. You haven't locked a door. You've taped a "please don't" note to a doorway that has no door.
The drugs-and-weapons analogy is the tell. Banning heroin does something because heroin is physical — scarce, perishable, seizable, gone once it's gone. A model is a file. You can't seize a copy, you can't make it rarer by outlawing it, and one leak is permanent and worldwide. The very property that makes interdiction work for drugs is the property open models don't have. So the analogy that makes controls sound sensible is the one that proves they won't bite.
Now the cost of controls, which is certain in a way the catastrophe is not.
Every control on releasing AI is a control on who gets the best AI. The moment you require approval, licensing, or "responsible release," you've guaranteed that governments, big companies, and the already-powerful end up with frontier intelligence while everyone else gets the throttled version. That's exactly what a gate in front of the most important technology in history does: it decides who ends up on the powerful side of it.
We've seen this movie. Controls meant to stop the worst actors mostly inconvenience ordinary people and entrench incumbents. The worst actors — nation-states, organized crime — route around them. Worse, the controls are unilateral: a rule that binds American labs and open researchers does not bind a foreign adversary training its own unrestricted model. So you slow your own side in a race you haven't stopped, pay the full price of the controls, and capture almost none of the benefit.
So what do we do about the real risk? Not nothing — just not this. Put the effort where attacks actually break instead of where they don't:
Harden the defense with the same models, and fund it like it matters — detection, patching, monitoring, response.
Guard the physical chokepoints that actually bind an attack: reagents, materials, money, and logistics, not knowledge.
Keep capability distributed, so the largest possible number of people are standing on the defending side of this technology.
Open models are also our best defense. They're how researchers find flaws, how small players keep up, how capability stays distributed instead of captured. Take that away and you haven't made anyone safer — you've just decided who holds the power.
So the right number of controls on releasing open source AI is zero. Not because the risks are imaginary, but because the cure is worse — and the cure is the part we can actually be sure of.
Convinced? Cast your vote.
Watch — Dario Amodei testifies to the Senate
Anthropic CEO Dario Amodei on the potential regulation of frontier AI.