Open Does Not Mean True
The biases and dangers of open source, from software supply chains to civic data
Codex Americana · June 2026
Abstract. "Open source" has quietly become a trust signal. Audiences read it as a proxy for neutrality, safety, and truth: if the code, the data, or the methodology is published, surely it has been checked, surely it conceals nothing, surely it is fair. This paper argues that openness is a property of artifacts, not a guarantee about their content. It is often a prerequisite for meaningful public accountability, but it is not sufficient for trust. Drawing on documented cases in software security, machine-learning datasets, and civic data, it maps four failure modes, open-washing, the "many eyes" security fallacy, propagation harms, and the epistemic trap of openness as a credibility heuristic, with economic capture running across them as a common cause. It then turns the ledger over, examining cases where openness clearly succeeded, and shows that even these confirm the paper's central distinction: openness can verify facts but cannot, by itself, verify framing. It closes with a reusable test for reading an open artifact, and a single claim: openness should lower the cost of verification, never the standard of it.
1. The promise, and the slippage
The open-source ideal carries a real moral and epistemic promise. Eric Raymond's The Cathedral and the Bazaar (1999) gave it a slogan that hardened into doctrine: "given enough eyeballs, all bugs are shallow," now remembered as Linus's Law. The intuition is democratic and appealing. If anyone can inspect a thing, then errors get caught, hidden agendas get exposed, and the artifact earns trust the honest way, by surviving scrutiny rather than demanding faith.
The same logic underwrites the broader modern faith in transparency as accountability. Ananny and Crawford (2018) name it the "logic of accumulation": the assumption that the more of a system we can see, the more we can know about it, and therefore the more defensibly we can govern it. Their paper is the necessary corrective. Their central finding is that "seeing without knowing" is the common case: being able to look inside a system is routinely inadequate for understanding it, and understanding it is in turn inadequate for holding it accountable. The chain from visible to understood to governable breaks at every link.
That broken chain is where every danger in this paper lives. The failure is rarely the openness itself; it is the heuristic that rides on top of it, the silent inference from "I can see it" to "I can trust it."
It helps to separate three things that the word "trust" runs together, because openness treats them very differently. The first is factual verification: is a given claim accurate, and does it trace to a real source? The second is interpretation: what does that fact mean, and what follows from it? The third is framing: which facts were chosen and foregrounded, are competing claims given proportionate weight, is the overall posture faithful to reality? This is the single most important distinction in the paper, so it is worth stating plainly at the outset. Openness is powerful, often decisive, at the first level; it assists but does not settle the second; and it cannot, on its own, verify the third at all. An open artifact lets you check whether the numbers are right. It cannot certify that the story told with those numbers is the true one. Nearly every danger below is an instance of mistaking verification at the first level for assurance at the third.
The four failure modes that follow are the ways that mistake plays out. A fifth force, economic capture, runs across all of them rather than standing beside them, and is treated where it bites.
2. Failure I: Open-washing, or openness claimed but not delivered
The first danger is that the label is simply false. By analogy to greenwashing, open-washing describes claiming the benefits of openness, the goodwill, the regulatory relief, the presumption of transparency, without delivering the substance.
The clearest contemporary evidence comes from generative AI. Liesenfeld and Dingemanse (2024), in work presented at the ACM Conference on Fairness, Accountability, and Transparency, surveyed more than forty large language models and several text-to-image models that market themselves as "open." Using an evidence-based framework of fourteen dimensions of openness, from training data to documentation to licensing, they found that many systems are "open weight at best" and some are "open in name only," with several providers actively avoiding meaningful scientific documentation. A companion summary in Nature (Gibney, 2024) put it plainly: almost all the major firms claim to offer open models, and very few actually do.
Two structural lessons follow. First, openness is not binary; it is composite and gradient. A model can publish weights while withholding the training data that would let anyone audit its biases, and "open" then becomes a marketing term rather than a property. Meta's Llama models are the most familiar instance: promoted broadly as "open source," they are released under a license that carries use restrictions and without the training data, and so fall short on several of the dimensions the label is supposed to guarantee. Second, regulation can perversely reward the mislabeling: where a legal regime grants lighter obligations to "open source" systems, the label acquires cash value, and the incentive to claim it without earning it grows. The Open Source Initiative's response, a formal Open Source AI Definition meant to discipline the term, is itself evidence of how loosely the word had come to be used. There is a political economy underneath this, the cross-cutting force of capture: the label is most worth claiming, and least worth honoring, for the largest incumbents, who can take the reputational goodwill and regulatory relief of "open" while keeping the control that genuine openness would surrender, and who gain when the surrounding ecosystem is commoditized around their release. The practical takeaway for any reader: never accept the label; ask which specific dimensions are open and which are quietly closed.
3. Failure II: The "many eyes" fallacy and the trust supply chain
The second danger is the one Raymond's slogan invites directly. "Many eyes make bugs shallow" is true only if the eyes exist, actually look, and are competent and funded to look. Openness creates the possibility of review; it does not create the review. Treating the possibility as the fact manufactures a false sense of security.
The historical record is unambiguous. Heartbleed (CVE-2014-0160), a catastrophic flaw in OpenSSL, one of the most widely deployed and most "many-eyed" open-source libraries on earth, sat exploitable for roughly two years before discovery. Log4Shell (CVE-2021-44228), a maximum-severity vulnerability in the ubiquitous Log4j logging library, revealed in December 2021 that critical global infrastructure depended on a project maintained by a handful of unpaid volunteers. The cartoonist Randall Munroe captured the structural picture in xkcd #2347, "Dependency": the entire edifice of modern digital infrastructure balanced on a single component "some random person in Nebraska has been thanklessly maintaining since 2003."
The xz Utils backdoor of 2024 (CVE-2024-3094) turned this fragility from accident into attack. As documented across the security community and disclosed by Microsoft engineer Andres Freund, an actor operating under the pseudonym "Jia Tan" spent roughly two to three years making legitimate contributions to xz Utils, a compression library bundled into nearly every Linux distribution, in order to earn the trust of an overburdened solo maintainer and gain co-maintainer status. The attacker then hid a backdoor, targeting OpenSSH, inside the release tarballs rather than the version-controlled source, so that ordinary code review of the public repository would not catch it. The payload carried a CVSS score of 10.0 and could have compromised hundreds of millions of servers. It was discovered almost by luck: Freund noticed that SSH logins were running about half a second slow and pulled the thread.
The xz case is the definitive refutation of naive "many eyes" optimism, because it weaponized the trust model itself. Open-source development runs on the norm that contribution earns trust; a patient adversary simply contributed until trusted. The structural enabler was maintainer burnout, an unpaid volunteer in a difficult personal situation, pressured into ceding control. This is also where capture reappears: enormous commercial value is extracted from components like xz, Log4j, and OpenSSL, while the cost and risk of maintaining them are socialized onto a thinly resourced volunteer commons, which is precisely the condition a patient attacker exploits. Openness here was not a defense; it was the attack surface.
None of this makes openness worthless against bugs and backdoors. The opposite: open scrutiny is precisely why Heartbleed, Log4Shell, and the xz backdoor were eventually found at all, where a closed system might have concealed them indefinitely. Openness can raise the probability of eventual discovery. The fallacy is narrower, and it is twofold: treating that raised probability as a certainty, and assuming the eyes are looking right now rather than merely being permitted to look someday. The honest reading of an open codebase is therefore not "many people have surely checked this," but "has anyone with the relevant expertise actually audited this, and is anyone funded to keep doing so?"
4. Failure III: Propagation and irreversibility in open data and open weights
The third danger is specific to open data and open model weights: when an artifact is both open and copied at scale, its defects propagate downstream and become nearly impossible to recall.
The defining case is LAION-5B, an open dataset of roughly 5.85 billion image-text pairs scraped from the web and used to train widely deployed image generators including Stable Diffusion. In December 2023, the Stanford Internet Observatory, in analysis led by David Thiel, found that the dataset contained thousands of suspected instances of child sexual abuse material, with more than a thousand externally validated. The dataset had been assembled from the open web with no human in the loop and no consultation with child-safety experts, filtered only by an automated model whose own designers had warned against using it for that purpose. The report's blunt conclusion was that possessing a populated copy of the dataset implied possessing thousands of illegal images. LAION withdrew the dataset and later republished a cleaned version, but by then the data had been downloaded, redistributed, and baked into models already in the wild. Earlier work by Birhane and colleagues had already documented hateful, explicit, and stereotyped content pervading related open datasets.
The lesson is twofold. First, openness multiplies harm: a defect in a closed dataset harms one organization, while a defect in an open one harms everyone downstream and resists clawback. Second, an open artifact is not a documented one. Provenance, consent, and known limitations do not travel with a file unless someone deliberately attaches them; this is precisely why documentation practices such as datasheets for datasets (Gebru et al., 2021) and model cards (Mitchell et al., 2019) had to be invented. The mere availability of the data tells you nothing about where it came from or what it carries. The same property that makes open data auditable, that anyone can hold it, is what makes its harms scalable and permanent.
5. Failure IV: The epistemic trap, credibility laundering and false balance
The fourth danger is the most subtle and the most relevant to anyone publishing analysis, civic data, or "neutral" information products. Here openness functions as a credibility heuristic, and a heuristic can be gamed.
Start from Ananny and Crawford again: transparency does not deliver understanding or accountability on its own. Goodman and Tréhu (2022), writing on "audit-washing," extend the point to a now-common gesture: publishing an audit, or an open repository, can launder accountability, conferring the appearance of having been checked without the substance of meaningful oversight. A published self-audit is evidence of good faith. It is not proof of neutrality, and it can be mistaken for one.
The deeper trap is false balance, and it has clean, historically settled demonstrations. The clearest is tobacco. As Oreskes and Conway document in Merchants of Doubt (2010), the cigarette industry, having privately concluded that "doubt is our product," deliberately recruited scientists to manufacture the appearance of an open question about smoking and cancer and worked to convince the press that responsible journalism required presenting "both sides" of a matter that was not, scientifically, two-sided. The balance norm was not merely tripped; it was weaponized. Boykoff and Boykoff (2004), in their study of US prestige-press coverage of global warming from 1988 to 2002, found the same pattern operating through ordinary journalistic habit rather than design: coverage that gave roughly equal weight to an overwhelming scientific consensus and to a handful of skeptics systematically misrepresented the state of knowledge. In both cases the two sides were not of equal evidentiary weight, and presenting them as though they were is what produced the distortion. When reality is lopsided, manufactured symmetry is itself the bias.
Openness does not protect against this. It can make it worse, and the mechanism is specific. Because an open artifact's data is traceable and its workings are visible, a reader naturally extends the credibility earned by that verifiable openness to the artifact's framing as well, accepting the even-handed presentation as if it too had been vetted by the same transparency that checked the numbers. But openness audits the facts; it does not audit the decision to treat two unequal positions as equal. The credibility is borrowed by the framing without being earned by it. A false balance wrapped in an open, well-sourced repository is more persuasive, and therefore more dangerous, than the same false balance in an unsourced op-ed.
There is a particular mechanism worth naming, because it is invisible from inside the artifact. A self-audit that checks a document against a balance standard, "did I present both sides," can never detect false balance, because false balance satisfies a balance standard. To catch it, you must check the framing against external reality, "is even-handedness the truthful posture here," and a closed-loop self-audit that never consults the outside world cannot perform that check. The audit will keep returning "balanced," and the artifact will keep publishing that result as proof of fairness, while the question of whether balance is faithful to reality goes permanently unasked.
Case study: a genuinely open civic-data project
Consider marbleheaddata.org (source repository github.com/agbaber/marblehead), a civic-data project covering a Massachusetts town's fiscal-year 2027 budget shortfall and a Proposition 2½ override on the June 9, 2026 ballot. It is a real, independently created project, inspected directly for this paper rather than constructed for it. Its value as an example is the reverse of the usual cautionary tale: it matters not because it is unusually bad but because it is unusually good. It is openness done about as conscientiously as the format allows, which is exactly what makes it the strongest available test of the thesis. If even a meticulous, transparent, self-auditing project cannot escape the framing problem, then the problem is not a defect of careless work; it is a limit of openness itself.
On the openness axis, it passes the tests that the AI models in Section 2 fail. The published repository is the complete running site, not a hollow shell; it carries a genuine dual license (MIT for code, Creative Commons for content); it has a long, un-backdated, multi-month commit history; and its data is traceable, with a source lookup that ties individual figures to specific documents and pages. By the standard of Section 2, this is the real thing, not open-washing.
And yet it still illustrates the Section 5 traps, which is the point. The site publishes its own bias audit and a remediation plan, an admirable transparency move, and inspection confirms the documented fixes are actually present in the deployed files rather than merely promised. But that self-audit was itself machine-generated and ran in one direction, hunting only for a pro-override lean and never asking the inverse question of whether it had over-corrected toward a false middle; and by its own statement it checked only what shipped in the repository, never reaching outside the artifact to test its framing against reality. A residual lean of emphasis survived.
Most instructive is a case the openness cannot resolve. The site presents two official figures for the change in school staffing that point in opposite directions, one showing a rise and one a substantial fall, and reconciles them as "both partly true." It would be tempting to call that the textbook false-balance move; honesty forbids it, because the two figures may not be measuring the same quantity at all. A financial report's full-time-equivalent count of all education staff is simply not the same thing as the state's count of licensed educators, and if the metrics differ, then "both partly true" is not a wash but the correct answer. That is exactly the difficulty. Full transparency shows a reader both numbers and the reconciliation; it does not tell the reader whether that reconciliation is honest synthesis of two different measures or a false balance laundering a genuine contradiction. Adjudicating it requires an external judgment about what the metrics actually mean, which the open data alone does not supply. This is the thesis in miniature: openness laid every number on the table and still could not certify whether the framing was faithful. The facts are checkable; the framing is contestable; and no quantity of published code closes the gap.
None of this makes the project dishonest. It makes it human, and it makes the general lesson sharp. Openness let an outside reviewer check the project's facts against its sources, which is real and valuable and forbidden by closed systems. Openness could not certify that the project's framing was faithful to reality, and it could not settle the project author's intent, which remains unfalsifiable from the outside no matter how much code is published. The most open artifact in the world can show you that its facts are right while leaving the harder question, whether its balance is true, exactly where it found it.
6. The other ledger: what openness is genuinely good for
A paper built entirely on failures invites a fair objection: that it has gone looking for cases where openness disappointed and ignored the cases where it delivered. The objection has to be met head-on, because the cases where openness delivered are real, and on inspection they point to the same conclusion rather than against it.
Open development has produced some of the most robust and trusted infrastructure in existence. The Linux kernel, openly developed and auditable by anyone, runs the majority of the world's servers in part because its openness invites the scrutiny, forking, and independent patching that a closed kernel forbids. The value here is not the slogan about "many eyes" but the structural fact that no single vendor can quietly bury a defect or refuse a fix. That is openness working as advertised.
Two further cases matter most, because they show openness delivering precisely at the level where it is strong, factual verification, while leaving the higher levels untouched. The first is the encyclopedia. A blind expert comparison published in Nature (Giles, 2005) found the science coverage of the openly editable Wikipedia roughly comparable in accuracy to Encyclopaedia Britannica's, with somewhat more minor errors; Britannica disputed the methodology, and the gap in serious errors was small. The decisive point is not the near-tie but the mechanism: an error found in Wikipedia can be corrected within minutes and the correction inspected by anyone, a self-healing property no closed reference work has. And yet the same openness that continuously repairs Wikipedia's facts does nothing to guarantee its framing; the platform's edit wars and well-documented systemic biases live exactly at the level openness cannot reach. The upside and the limit are the same property seen from two sides.
The second is the cleanest single proof that openness corrects facts. In 2010 the economists Carmen Reinhart and Kenneth Rogoff published a finding, widely cited to justify austerity, that growth collapses once public debt passes ninety percent of GDP. Because they eventually shared their actual spreadsheet, Thomas Herndon, Michael Ash, and Robert Pollin (2013) replicated it and found that a coding error had silently dropped several countries from the key average, alongside contestable data and weighting choices; corrected, the dramatic cliff largely vanished. A closed analysis would have hidden that error indefinitely; openness exposed it. And yet, tellingly, openness did not end the argument. Even with the computation fixed, whether high public debt slows growth remained a live interpretive dispute, contested to this day. Openness settled the fact and handed the framing back, unresolved, to the people arguing about it.
This is why the thesis is narrow rather than hostile. Openness is genuinely, sometimes irreplaceably good at the thing it is good at. Every honest counterexample turns out to be a victory at the level of fact and a silence at the level of framing. Having actively looked for the upside, one finds the same boundary the failures trace, approached from the other side.
7. The Open Artifact Test
The analysis converts into a short, reusable discipline. Call it the Open Artifact Test: before extending trust to anything because it is "open," work through six questions.
- Treat "open" as a gradient, not a badge. Ask which dimensions are actually open, code, data, weights, provenance, license, documentation, and which are quietly closed. Name the gaps (Liesenfeld and Dingemanse, 2024).
- Count the eyes, do not assume them. "Many eyes" protect nothing unless someone with relevant expertise actually audited the artifact and is funded to keep doing so. Ask whether the project is maintained, resourced, and reviewed, or balanced on one exhausted volunteer (Heartbleed, Log4Shell, xz).
- For data and models, trace provenance and propagation. Where did the material come from, was it documented in a datasheet or model card, and what flows downstream from it? Remember that an open defect is a permanent, replicable one (LAION-5B; Gebru et al., 2021).
- Separate what openness verifies directly from what needs an external check. Openness lets you verify factual accuracy against sources directly. Whether the framing is faithful, whether the balance is true and the emphasis fair, requires a reality check the artifact cannot perform on itself. That is harder, but it is not beyond scrutiny; an outside reviewer can catch a false balance, just not by trusting the artifact's own audit to do it. Watch for it specifically: symmetry imposed on an asymmetric reality (Boykoff and Boykoff, 2004).
- Read a published audit as good faith, not as a verdict. A self-audit that checks balance rather than reality, in a closed loop, cannot detect its own false balance, and a published audit can launder credibility it has not earned (Goodman and Tréhu, 2022; Ananny and Crawford, 2018). Intent is unfalsifiable from outside; say so, and hold it open.
- Ask whom the openness serves. Distinguish openness that distributes power from openness that concentrates it or shifts cost and risk onto a commons.
8. Conclusion: a powerful means, not a guarantee
Openness is a genuine good, and nothing here is an argument against it. An open repository let an outsider contest a civic dataset's numbers line by line; an open mailing list let one engineer expose a backdoor that nearly compromised the internet; open scrutiny is the reason we know about Heartbleed, about LAION-5B, about open-washing at all. Closed systems forbid every one of those corrections. The capacity to be checked is exactly what closed artifacts deny and open ones permit, and it is worth defending.
But capacity is not performance, and visibility is not virtue. The dangerous move, the one this paper is written against, is the silent heuristic that converts "open" into "therefore trustworthy," "therefore neutral," "therefore safe." Openness lowers the cost of verification. It must never be allowed to lower the standard of it. The eyes still have to look; the data still has to be sourced; the framing still has to be tested against a reality that no amount of published code can settle on your behalf. Open does not mean true. It means you have been given the means to find out, and the obligation to actually do so.
References
Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989. https://doi.org/10.1177/1461444816676645
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021). Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. https://arxiv.org/abs/2110.01963
Boykoff, M. T., & Boykoff, J. M. (2004). Balance as bias: global warming and the US prestige press. Global Environmental Change, 14(2), 125–136. https://doi.org/10.1016/j.gloenvcha.2003.10.001
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723
Gibney, E. (2024). Not all "open source" AI models are actually open: here's a ranking. Nature, June 2024. https://doi.org/10.1038/d41586-024-02012-5
Giles, J. (2005). Internet encyclopaedias go head to head. Nature, 438(7070), 900–901. https://doi.org/10.1038/438900a
Goodman, E. P., & Tréhu, J. (2022). AI Audit-Washing and Accountability. German Marshall Fund of the United States.
Herndon, T., Ash, M., & Pollin, R. (2013). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2), 257–279. (Originally a Political Economy Research Institute working paper, April 2013.)
Liesenfeld, A., & Dingemanse, M. (2024). Rethinking open source generative AI: open-washing and the EU AI Act. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), 1774–1787. https://doi.org/10.1145/3630106.3659005
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19), 220–229. https://doi.org/10.1145/3287560.3287596
Munroe, R. (2020). Dependency. xkcd #2347. https://xkcd.com/2347/
Oreskes, N., & Conway, E. M. (2010). Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming. Bloomsbury Press.
Raymond, E. S. (1999). The Cathedral and the Bazaar. O'Reilly Media.
Thiel, D. (2023). Identifying and Eliminating CSAM in Generative ML Training Data and Models. Stanford Internet Observatory.
Incidents referenced by identifier: Heartbleed (CVE-2014-0160, OpenSSL, disclosed 2014); Log4Shell (CVE-2021-44228, Apache Log4j, disclosed December 2021); xz Utils backdoor (CVE-2024-3094, disclosed March 2024 by Andres Freund).
