The Quiet Death of the Open-Source AI Dream

Open source has a precise meaning: the right to view, modify, and redistribute source code. It says nothing about training data, compute requirements, or the economic realities of running frontier AI. Yet every week, another lab announces an "open" model and expects applause.

The Problem with "Open Weights"

What most labs release are model weights — the compressed numerical output of a training run that cost tens of millions of dollars. You can download the numbers. You cannot replicate the process that produced them. Meta's Llama 2 license explicitly prohibits commercial use above 700 million monthly active users. That isn't open source. That's a conditional commercial licence with marketing copy wrapped around it.

The distinction matters enormously. The original open-source movement was about power: the power to audit software, to modify it for your needs, to build on it without permission. A model weights release without training data is like releasing a compiled binary and calling it open. You can run it. You cannot understand it, reproduce it, or meaningfully improve it.

Who Benefits from the Confusion

Labs benefit in three ways. First, they receive goodwill and talent from a developer community that associates "open" with trustworthiness. Second, they preempt regulation — legislators who might otherwise impose transparency requirements get told the models are already open. Third, they maintain a moat: researchers at competitor labs can study the weights and learn something, but they cannot retrain the model without the data and compute that only frontier labs possess.

The confusion also benefits a specific rhetorical move: framing open weights as a counterweight to proprietary models. When Zuckerberg argues that "open source AI" is safer than closed models, he is making a claim that depends entirely on conflating weights-release with genuine openness. There is no peer-reviewed evidence that weights-only release substantially improves safety auditability.

What Genuine Openness Would Require

The AI Now Institute and researchers at the Center for AI Safety have argued that meaningful openness requires: full training data disclosure, training code and methodology, evaluation benchmarks and results, and a licence permitting genuine modification and redistribution. By these standards, no frontier model currently qualifies. EleutherAI's PILE dataset and the associated GPT-Neo models come closest, but at a capability tier substantially below the models generating the most attention.

The tension is real. Truly open training data would expose copyright-violating scrapes. Truly open compute estimates would reveal energy costs that are politically inconvenient. The open washing is, in part, a solution to genuine exposure risk.

Why It Matters

Policy is being made around the assumption that "open source AI" is a coherent category. The EU AI Act, US executive orders on AI, and proposed NIST frameworks all treat openness as a risk variable. If the underlying concept is undefined — or deliberately muddied — the regulations will not achieve their stated goals.

The open-source software community spent decades building legal and technical infrastructure around clear definitions. The AI industry is importing the brand while discarding the substance. The result will be a regulatory framework designed around a fiction.