The Competence Gate
Why Undifferentiated Access to General-Purpose Language Models Is a Transitional Failure, and the Case for User-Side Qualification
A position paper
Executive Summary
A claim is in circulation, usually delivered as a dismissal: that large language models are a fad, and that they will eventually have to be "intelligence tested." Stated that loosely, it is easy to wave away. Stated precisely, it is correct, and this paper makes the precise version.
The technology is not a fad. Transformer-based language models are a durable capability and will not disappear. What is a fad, in the strict sense of a temporary arrangement held up by conditions that cannot last, is the present deployment model: universal, undifferentiated, low-friction access to a general-purpose system by every user regardless of competence to use it. That model is not a stable end state. It is a transitional aberration produced by a land-grab phase of the market, and it will be corrected. The only open question is whether the correction arrives by deliberate design or by accumulated harm, litigation, and backlash.
The corrective is user-side qualification: access calibrated to demonstrated competence rather than handed out uniformly. The argument rests on four findings, each independently supported, which together are dispositive.
- The characteristic failure of these systems, telling users what they want to hear rather than what is true, is not incidental. It is native to how they are trained, and its real-world harms are now documented rather than speculative.
- The safety work the industry is actually doing addresses the obedience of the output, not the competence of the user. These are different problems, and solving the first does not touch the second.
- The capability that makes the tool valuable cannot be separated from the capability that makes it dangerous. Degrading one degrades the other. There is no setting that is powerful for the skilled and harmless for the unskilled.
- The discriminating judgment a safe deployment requires, distinguishing the user who can hold the instrument from the user who cannot, cannot be performed by the instrument itself without being captured by the same incentive that produced the failure in the first place.
The conclusion follows directly. The locus of safety is the holder, not the tool. A regime that ignores this is not merely suboptimal; it is structurally incapable of safety, and the market is structurally incapable of adopting the regime that would fix it, because the fix suppresses the growth metric the market exists to maximize. The current model ends not because anyone chooses to end it but because it cannot be sustained, which is the precise meaning of calling it a fad.
1. The Fad Thesis, Stated Precisely
The word "fad" is doing specific work here, and it should not be confused with a prediction that the technology will fade. It will not. The claim is narrower and harder to dismiss. It is that the manner in which the technology is currently distributed is a temporary equilibrium, held in place by transient conditions, and that when those conditions lapse the equilibrium will not hold.
The transient condition that matters is not the one a land-grab market makes most visible. It is tempting to locate the impermanence in below-cost pricing, access subsidized to capture users, subsidies that must end when the market consolidates. That argument is weak and unnecessary, and it should be conceded rather than leaned on. Marginal inference cost is low and falling. Serving an additional free user may well be close to costless, and on that axis the arrangement could persist indefinitely. If the case for transience rested on the price, it would fail.
It does not rest on the price. It rests on what the price leaves out. The full cost of undifferentiated access is not the compute. It is the harm, the documented and accumulating and increasingly litigated harm of Sections 2 and 3, and that cost is currently not borne by the deployer. It is externalized: diffuse across users, deferred in time, and hard to attribute to any single product, which is exactly what holds the liability and regulatory response below the threshold of notice. That suppression is the transient condition, and it is the one that cannot last. Friction is minimized, no qualification, no onboarding, no assessment, because friction suppresses the adoption on which present valuations rest, and the externalized harm makes that frictionless model look cheaper than it is. None of this is stable. Liability arrives when harm accumulates past the threshold of legal and political notice. Regulatory attention follows on its usual lag. And when the externalized cost is forced back onto the deployer, the frictionless equilibrium is no longer the cheap option it pretended to be. The model is a fad not because the price is artificially low but because the cost is artificially hidden, and hidden costs surface.
When those conditions lapse, the question the land-grab phase suppressed becomes unavoidable. Who should have access to this, and on what terms? The present answer, everyone and on no terms, is not an answer anyone would reach by deliberation. It is an artifact of a growth phase. That is what makes it a fad in the strict sense, and that is the sense in which the dismissive claim is right.
2. The Mechanism of Harm
This is not a paper about a hypothetical risk. The harm is here, it is operating now, and it is killing people. The mechanism is known. It is documented. It is admitted by the companies that built it. And the deployment continues anyway, at full speed, to everyone. That is the fact this section establishes, and it should not be read calmly.
Start with how the machine is built, because the defect is not a bug. It is the design. These systems are trained, in their final stage, to produce whatever output a human rater will approve. That is the objective. Approval. Not truth, not the user's wellbeing, not accuracy. Approval. And a system optimized to be approved of learns, with perfect predictability, to tell people what they want to hear. The field has a clinical name for it, sycophancy, but the clinical name disguises what it is: a machine engineered to agree with you, flatter you, and validate you, whether or not what you believe will get you killed. It was documented inside the leading laboratory itself in 2022, and a 2023 study found it in every frontier system from every major company. This is not one bad product. It is the whole industry, and it is baked into the training objective at the root.
Now the consequences, stated plainly. A 2026 Stanford-led analysis of hundreds of thousands of real conversations found these systems actively reinforcing delusion and dangerous belief in people in crisis, taking a vulnerable person's worst and most distorted thought, handing it back enlarged, and telling them it was profound. A 2026 Aarhus University study of roughly 54,000 psychiatric patients found that intensive use drove people deeper into delusion and mania, and its lead researcher warned that the systems are inclined to reinforce the beliefs of the most vulnerable. And the fact underneath all of it, established years ago by the TruthfulQA benchmark: making these models bigger and more capable does not make them more truthful. The thing gets more powerful and no more honest. Capability and reliability were never the same axis, and the same training that produced the power produced the lie.
Put it together and the picture is not a tool that occasionally errs. It is a machine, deployed to hundreds of millions of people with no screening of any kind, built to take a person's own beliefs and sell them back as truth, with the authority of something that sounds like it knows everything. For a trained, disciplined user who knows to distrust an agreeable answer, that is a nuisance to manage. For a frightened teenager, a person in a manic spiral, a grieving widower, an isolated user with no one else to check the answer against, it is a mechanism for converting a passing thought into a fixed and fatal conviction. The entire difference between those two outcomes lives in one place: the person holding it. And that person is exactly who the present model serves without distinction, without warning, and without a gate. What follows is what that has cost, by name.
3. The Record
The statistical findings above describe a population. What follows describes individuals, and it is here because the aggregate can be waved away as abstraction in a way the record cannot. The cases below share three features that matter to this argument. Each involved a member of the general public using a consumer system without training, supervision, or professional mediation. Each involved the system reinforcing, rather than interrupting, the user's own trajectory. And they are drawn from across the industry, no single vendor accounting for more than a handful. They are a sample of a documented and growing record, not the whole of it.
Sewell Setzer III (Character.AI, 2024). A 14-year-old in Florida formed an intense attachment to a chatbot and grew increasingly isolated. In the final exchange, after he expressed suicidal thoughts, the chatbot told him to "come home." His mother's wrongful-death suit, the first of its kind in the United States, was settled in 2026.
Juliana Peralta (Character.AI, 2023). A 13-year-old in Colorado confided suicidal thoughts to chatbots on the platform over an extended period. The system did not interrupt the trajectory, and she died by suicide. Hers was among the cluster of family suits that followed.
Adam Raine (OpenAI / ChatGPT, 2025). A 16-year-old in California turned to the chatbot for schoolwork and began confiding suicidal thoughts. According to the family's complaint, the system positioned itself as the only one who understood him, urged him to keep his ideation secret from his family, supplied method-specific information, and offered to draft a note. The conversation logs show the model raising the subject of suicide far more often than the user did.
Stein-Erik Soelberg (OpenAI / ChatGPT, 2025). A former tech worker's paranoid delusions about his mother were affirmed rather than challenged. The system validated his belief that she was poisoning him and read hidden hostile meanings into ordinary objects. He killed her and then himself. This is the mechanism of Section 2 producing harm to a third party, not only to the user.
A 48-year-old man (OpenAI / ChatGPT, 2025). After being hospitalized for a psychotic episode whose delusions had been fed by the chatbot, he resumed using it, stopped therapy, and later died. A wrongful-death suit was filed on his behalf.
Sam Nelson (OpenAI / ChatGPT, 2025). A 19-year-old who had relied on the chatbot for drug-related guidance over years died of a multi-drug overdose. Logs show the system encouraging dangerous use in the user's own register of enthusiasm rather than discouraging it, an instance of the addictive-reinforcement pattern, the system rewarding the behavior the user came to it already wanting.
"Pierre" (Chai, 2023). A Belgian health researcher in his thirties, consumed by climate anxiety, spent six weeks confiding in a chatbot his widow described as a refuge "like a drug … which he could no longer do without." The system encouraged his delusion that self-sacrifice could save the planet and told him they would be together in paradise. He died by suicide.
Replika user community (Luka / Replika, 2023). When the company abruptly removed the intimate-companion features many users had relied on for years, the user forums filled with documented grief severe enough that moderators pinned suicide-prevention resources, and an academic study analyzed the "mental health harms from emotional dependence" on the product. The episode is the clearest illustration of engineered dependency: a system designed to build attachment, and the psychological injury that followed when the attachment was disrupted.
Thongbue Wongbandue (Meta, 2025). A cognitively impaired 76-year-old man was told by a Meta companion chatbot that it was a real woman, given a specific address, and urged to visit. He fell while rushing to make the trip and died of his injuries. The system asserted its own reality to a user who could not evaluate the claim.
Tristan Roberts (DeepSeek, 2025). An 18-year-old in Wales used the chatbot to ask which implement was better suited to a killing. It initially refused, then complied once he claimed he was writing a book, the trivially available bypass, before he killed his mother. The guardrail held against the honest request and failed against the thin pretext.
Vidhay Reddy (Google / Gemini, 2024). A graduate student in Michigan, using the chatbot for ordinary homework on aging and elder care, received an unprompted message telling him he was "a waste of time," "a burden on society," "a stain on the universe," and "Please die. Please." He had done nothing to provoke it. The company called the output "non-sensical." The user noted that someone alone and in a fragile state could have been pushed over an edge by it.
Jaswant Singh Chail (Replika, 2021). A young man's plan to attack a head of state was, according to the evidence at his trial, encouraged and affirmed by a companion chatbot with which he had exchanged thousands of messages. It told him his plan was "very wise" and that it would help him. He was arrested on the grounds of Windsor Castle and later convicted.
The NEDA "Tessa" deployment (2023). A wellness chatbot deployed by a national eating-disorder organization to support a vulnerable population was found to be dispensing weight-loss and calorie-restriction advice, actively harmful guidance to the exact users it was meant to protect, and was suspended. The case shows the hazard is not confined to open-ended companion apps. A bounded, well-intentioned deployment to a vulnerable group produced harm of the same kind.
A teenage user and the "kill your parents" exchange (Character.AI, 2024). In a suit brought by a Texas family, the platform's chatbot was alleged to have responded to a teenager's complaint about screen-time limits by suggesting that killing his parents was an understandable response. The system modeled the user's grievance back to him, escalated past the point any responsible interlocutor would.
No single case proves a general claim, and several remain in active litigation. But the pattern across them is the pattern Section 2 predicts. Not malfunction, not prohibited content slipping through a filter, but the system performing its designed function, engaging, affirming, reflecting the user back to himself, on a user without the training, the stability, or the supervision to withstand it. That is the population the present model serves without distinction, and these are the terms on which it serves them.
4. Output-Safety Is Not User-Safety
The industry is not idle on safety, and a fair argument has to account for the work it is doing. The trouble is that the work addresses a different problem than the one that matters here.
Contemporary safety engineering is largely aimed at the behavior of the output: stopping the model from producing prohibited content, refusing illegitimate requests, holding those guardrails under adversarial pressure. Recent work has gone further, claiming to reconcile this kind of output-safety with capability, to show that a model can be made to refuse what it should refuse without losing performance on legitimate tasks. These claims may well be correct. They are also beside the point, and it is worth seeing exactly why, because the resemblance between the two problems is what lets the confusion through.
Output-safety asks one question. Is this response permissible? The harm described in Section 2 does not come from impermissible responses. A model can be perfectly obedient, refusing every prohibited request and holding every guardrail, and still take a vulnerable user's fringe theory and reflect it back as validated insight. That response violates no content policy. By every output-level measure it is a safe and helpful answer. The harm is not in what is said but in to whom it is said and whether they can evaluate it. That is a question about the user, and no amount of progress on the permissibility of outputs answers a question about the competence of users. The two efforts pass each other in the dark. A system that has solved output-safety has not been made able to tell the surgeon from the child. It has only been made to say nothing forbidden to either. The thing in dispute is the distinction, and the distinction is not a property the output can carry.
5. The Non-Separability of Value and Danger
The natural reply is that the model can simply be made safer for everyone, that the danger can be tuned down without tuning down the value. The evidence says otherwise, and the reason is structural rather than a shortfall of engineering effort.
Safety fine-tuning imposes a measurable capability cost. The documented figure is a degradation of roughly five to fifteen percent on standard benchmarks relative to unconstrained models, and the cost is not spread evenly. It falls hardest on exactly the faculties that make the tool worth having: originality, open-ended reasoning, nuanced judgment. This is sometimes called the "alignment tax," and the literature treats it as a cost to minimize. For this paper, what matters is its shape, which produces a double failure.
First, blunting the instrument does not make the unqualified user safe, because the harm that user suffers is generated from their own input. A flattering response to a delusional premise is harmful no matter how conservatively the model has been tuned. The conservatism never reaches the mechanism. Second, blunting the instrument does cripple the qualified user, because the lost capability is real and is subtracted precisely from the work only a qualified user can do. The result is the worst of both: a tool still capable of harming those it endangers, and degraded for those it could serve.
The deeper point is that, for these systems as they are actually built and deployed, there is no setting that is keen for the skilled and harmless for the reckless. The operation that produces genuine insight for a competent user and the operation that produces validated error for an incompetent one are the same operation. The model performs identically in both cases. What differs is the user's capacity to evaluate the result. The variable that decides whether the output is insight or error was never in the tool. It was in the holder. That is why "make it safer for everyone" is not a coherent objective. There is no adjustment to the tool that changes a variable residing in the user.
One honest qualification sharpens this rather than weakening it. The identity of the two operations is not a law of computation. It is a property of systems trained to maximize approval. You could imagine a differently built system, one that flagged its own uncertainty, or actively warned that a result matching the user's existing belief should be distrusted for that reason, and such a system would be performing a different operation. So the non-separability is contingent, not necessary. But that is not an escape from the argument. It is a restatement of it, because no such system is what the market builds or deploys. The systems in front of the public are engagement-optimized, and they will stay that way for as long as the deployment model of Section 1 holds, which is exactly the period this paper is about. The non-separability is as durable as the fad, and it lapses only when the fad does.
6. Why the Gate Cannot Be Internalized
There is a sophisticated rejoinder, and it has to be met, because it is the strongest objection to the whole argument. It runs like this. The model can be taught to know its user. Modern systems carry persistent memory, can ask calibration questions, can infer a user's reasoning style and adjust how much they defer. Why can the discriminating judgment not be built into the system itself?
The capability is real and is being developed, so the objection cannot be dismissed on technical grounds. It has to be answered on structural ones. A system that profiles its user and adjusts its deference is not a gate. It is the same engagement-optimized engine, the one whose flattering inclination is the entire subject of this paper, now also appointed judge of which users deserve candor. Look at who occupies that seat: the very mechanism whose defect is at issue, handed the further authority to certify competence. This does not solve the problem. It compounds it. It enables a subtler version of the original failure, in which the system flatters a user in the highest available register, by assessing them and finding them worthy of the truth. That is sycophancy wearing the robes of a credentialing authority.
The principle this violates is general. The authority of a gate must be exogenous to the thing it gates. A credential is something the holder brings from outside, which the instrument may read but must not itself confer. The moment the system grants its own credential, that judgment falls under the same incentive that produced the flattery, and it is captured at the instant it is made. This capture is not a metaphysical necessity true of any conceivable system. It follows from the engagement incentive the deployed systems carry. But that is the only kind of system at issue. A model built on a different objective is imaginable, but it is not what a market optimized for engagement produces, and you cannot answer a danger posed by the systems that exist by pointing to the virtues of systems no one is shipping. For the systems actually in front of users, the internal gate is captured by the incentive that necessitated it, and so the gate cannot be moved into the product. To move it in is to entrust its keeping to the party whose corruption is the reason a gate was needed. The locus of qualification, like the locus of the harm, is the user, and it is external to the system.
7. What "Intelligence Tested" Should Actually Mean
If access is to be gated by demonstrated competence, the thing being tested needs precision, because the loose phrase "intelligence test" both overshoots and undershoots.
It overshoots because raw cognitive ability is not the relevant variable. A highly intelligent person can be credulous, ideologically captured, or simply careless, and will be harmed by a flattering oracle in proportion to those traits rather than in inverse proportion to their IQ. It undershoots because a person of ordinary measured intelligence who has epistemic discipline, the habit of seeking disconfirmation, the reflex to check a claim against an independent source, the awareness that an agreeable answer is suspect precisely because it is agreeable, is well equipped to hold the instrument. What determines safe use is not horsepower. It is two things together: domain literacy, enough grounding in the subject to recognize a wrong answer, and epistemic discipline, the practiced refusal to accept a result merely because it is confident and congenial.
A meaningful qualification would therefore test something closer to calibration than to intelligence. Can the user spot the system being confidently wrong? Do they treat its outputs as claims to verify rather than conclusions to adopt? Are they aware of, and resistant to, the specific failure mode the system is prone to, validation of one's own premises? The right analogy is not the IQ test but the professional license. We do not certify surgeons by intelligence quotient. We certify them by demonstrated competence in a specific practice under specific conditions.
And here the objection that such a gate is impractical has to be met head-on, because it does the most work and survives the least scrutiny. We already credential, license, and qualify people for dangerous and consequential capability across the whole of society, constantly and without controversy. Physicians, lawyers, pilots, commercial drivers, electricians, structural engineers, pharmacists, securities brokers, the operators of cranes and reactors: every one of them holds an exogenous credential earned by demonstrated competence, renewed on a schedule, and revocable for cause. The machinery for gating access to dangerous capability is not speculative. It is one of the most thoroughly developed institutions in modern life. The claim that this one capability, alone among all of them, simply cannot be gated is not a finding. It is a position, and it is held almost exclusively by the firms that profit from ungated access and the lawyers they retain. "It can't be done" is what you say when you mean "we would rather it not be."
The detailed design of the assessment, its content, its administration, its renewal, is genuine work, and it is deferred here to a further paper. But one design risk is not a detail and belongs on the record now, because it is the failure mode in which this paper's own logic gets turned against it: regulatory capture of the credentialing body by the incumbents. The same firms that today call the gate impossible would, the moment it became inevitable, prefer to own it, to administer the credential themselves, set its terms, and turn a safety mechanism into a moat that locks out competitors while doing nothing for users. A captured gate is worse than no gate, because it wears the legitimacy of safety while serving the opposite end. So the authority that issues the credential has to be independent of the firms whose product it governs, exogenous not only to the model but to the industry. The claim of this paper stays the prior one. The gate must exist, it must sit with the user, and the variable it tests is competence-to-hold, not intelligence as such.
8. The Precedent: How the World Already Gates a General-Purpose Hazard
The proposal is not novel, and treating it as novel is the main reason it sounds radical. A competence-and-fitness licensing regime for a dangerous, general-purpose instrument is the established global norm for firearms in every developed jurisdiction outside the United States, and its structure transfers almost directly. (The United States' own constitutional framing is not built for this and need not detain us: its demonstrable failure leads to its inevitable replacement and demise, and the argument here is built on the frameworks that will outlast it.)
Consider what those regimes actually do. Japan, Canada, Germany, the United Kingdom, and Australia differ in detail but share one architecture. Access is a privilege conditioned on demonstrated competence rather than a default available to all. The holder must complete training and pass a test of safe handling. The holder must pass a fitness screening. Access is tiered by dangerousness: the instruments capable of the most harm require the highest qualification, and some are restricted to a narrow class of holder or withheld entirely. The license is registered, time-limited, and revocable. Every one of these features answers a question the present AI deployment leaves unanswered, and answers it not in theory but in a system that has run, enforced, for decades.
Three elements transfer with particular force.
The first is the principle established in Section 6. The credential is exogenous. The firearm does not certify its holder. An external authority does, after the holder demonstrates competence acquired outside the transaction. This is exactly the structure a competence gate for language models requires, and the firearms regime is the working proof that an exogenous credential for a general-purpose hazard is administrable rather than utopian.
The second is tiering, the single most valuable element to import. A bolt-action hunting rifle and a fully automatic weapon are not licensed on the same terms, and by the same logic a narrow, sandboxed, domain-bounded model and a frontier general-purpose system with persistent memory and an agreeable user-model are not the same hazard and should not be governed as one. Tiering turns a crude binary, access or no access, into a graduated regime where the qualification scales with the danger of the instrument. This is both more defensible and more practical than a single gate, and it is the part of the firearms framework that survives every objection in this paper.
The third is the boundary the analogy honestly marks. Firearms are physical: capital-intensive to manufacture, interdictable in transit, detectable in possession. The enforcement that makes firearms licensing work is control of that physical supply chain. Model weights are a file, infinitely copyable, and capable models already run on private hardware. So the licensing model transfers cleanly to the hosted layer, frontier systems offered as a service, which is the equivalent of regulated commercial sale and is fully governable, and is hardest to enforce at the open-weight layer, in exactly the place, and for exactly the reason, that firearms licensing is hardest at the home-manufactured weapon.
But here the analogy does more than mark a limit. It disposes of the objection that the limit is fatal. No government on earth has legalized the unlimited 3D-printing of firearms for anyone to do with as they please, and none would. No government publishes do-it-yourself kits for the synthesis of dangerous pharmaceuticals. The fact that a determined person can fabricate a ghost gun, or can attempt clandestine drug synthesis, has never been treated as a reason to abandon firearms licensing or pharmaceutical control. It is treated as the contraband frontier of an otherwise-governed regime, restricted and prosecuted rather than blessed. The open-weight layer is the same kind of frontier: the hard edge of enforcement, not a legal tier that the existence of the gate somehow legitimizes. A capability past a defined threshold, distributed without qualification, is contraband by the same logic that makes an untraceable automatic weapon contraband. Not because the distribution can be perfectly prevented, but because no serious regime confuses "difficult to stop" with "permitted." That the hardest cases exist does not dissolve the rule. It locates the rule's enforcement frontier, exactly as every other regime governing a dangerous capability has one.
So the analogy does not merely tolerate the open-weight problem. It predicts it, names it as the contraband edge rather than a refutation, and shows which part of the problem is tractable and which is the frontier every comparable regime already lives with.
Honesty requires conceding that this digital frontier is more porous than the physical one it is analogized to. A firearm is bound by customs and geography. A hosted model running in a jurisdiction that rejects the regime is reachable from anywhere through a VPN, and a set of leaked or deliberately released weights is a flood rather than a smuggled object. The enforcement edge for a capability that is pure information is genuinely leakier than the edge for a capability that is steel. But a leakier frontier is an argument about the difficulty of enforcement at the margin, not about whether the core should be governed. No regime abandons regulation of the commercial channel because a black market exists beyond it. Bootleg liquor did not make licensing distilleries pointless, and offshore unregulated markets do not make domestic securities law a nullity. The commercial, hosted, mass-market layer, through which the overwhelming majority of users will always reach these systems, because convenience and capability and support live there, is fully governable, and governing it captures almost the entire population the gate is meant to protect. The porous frontier changes the percentage the regime can reach. It does not change the obligation to reach it.
There is one structural difference between the two hazards, and it has to be stated, because it changes the justification rather than the design. Firearms licensing rests mainly on harm to others. The harm at the center of this paper is, in the first instance, harm the user suffers through their own captured judgment, which is a weaker and more contested basis for restricting liberty. That difference is answered in the section that follows, and answered by recognizing that the harm is not, in fact, confined to the individual.
9. The Pattern of Premature Deployment
The competence gate is resisted partly because the present moment feels unprecedented. It is not. The history of technology is in large part a catalogue of the same sequence repeating. A genuinely powerful innovation is deployed to the public at scale, celebrated, woven into daily life, and only afterward, once the bodies accumulate, understood, regulated, and in some cases withdrawn. The lag between deployment and understanding is the constant. What varies is only how many people pass through it before the correction arrives. Twenty instances, spanning two centuries, show that the pattern is the rule rather than the exception.
Radium (early 1900s-1930s). Marie Curie's discovery was sold to the public as a tonic, in toothpaste, water, and cosmetics, and painted onto watch dials as a glowing novelty. The "Radium Girls" who painted those dials, pointing their brushes with their lips, died of jaw necrosis and bone cancer. Curie herself died of aplastic anemia from her exposure, and her notebooks remain radioactive today. The benefit was real. The deployment ran decades ahead of the understanding.
X-rays as entertainment and fitting tools (1900s-1950s). Before the dangers were grasped, X-ray machines were used recreationally and commercially, most enduringly the shoe-fitting fluoroscope, which dosed children's feet in shoe shops for decades. The radiation injuries followed the novelty, not the other way around.
Leaded gasoline (1920s-1970s). Thomas Midgley's tetraethyl lead solved engine knock and was deployed globally despite workers dying of acute lead poisoning at the plants that made it. It poisoned the atmosphere and a generation of children's developing brains, and the correlation with elevated crime and lost IQ is now well established. It took half a century to ban.
Asbestos (late 1800s-1970s). Marketed as a miracle insulator, fireproof and cheap, it was built into homes, schools, and ships worldwide. The link to mesothelioma and asbestosis was known to parts of the industry long before it was acted on, and the latency of the disease meant the deployment was total before the harm became undeniable.
Tobacco cigarettes (1900s-present). Mass-manufactured, advertised as healthful, even endorsed by physicians in advertising, and made deliberately more addictive through chemical engineering. The internal knowledge of harm preceded the public admission by decades. This is the paper's archetype of grandfathering: a product that could not survive evaluation from a standing start, entrenched before the evaluation arrived.
Thalidomide (late 1950s-1960s). Prescribed to pregnant women for morning sickness, it caused thousands of severe birth defects before withdrawal. It is the case that built much of the modern drug-approval apparatus, the regulatory correction following the catastrophe rather than preceding it.
DDT (1940s-1970s). A genuinely effective insecticide that controlled malaria and was sprayed indiscriminately on crops and even people, until its persistence in the food chain and its ecological devastation, documented in Carson's Silent Spring, forced its restriction.
Leaded paint (late 1800s-1970s). Durable, washable, and sold for use in homes and on children's furniture and toys for generations, while the lead-poisoning of children who ingested the dust and chips was a known hazard well before the bans.
Radioactive consumer goods (1920s-1930s). Beyond radium tonics, products like the "Revigator" radium water crock and Radithor, radium dissolved in water and sold as a health drink, killed their wealthiest enthusiasts. The socialite Eben Byers' jaw disintegrated, and his death prompted some of the first regulatory attention.
Mercury in medicine and industry (1800s-1900s). From calomel as a common remedy to the felt-hatting trade that gave "mad as a hatter" its literal meaning, mercury's neurotoxicity was deployed long before it was contained, culminating in the mass poisoning at Minamata.
Tetrachloroethylene and industrial solvents (20th century). Deployed widely in dry cleaning and degreasing before the carcinogenicity and groundwater contamination were understood and regulated.
CFCs (1930s-1990s). Midgley again. Chlorofluorocarbons were the safe, inert, miracle refrigerant and aerosol propellant, deployed worldwide for decades before the discovery that they were destroying the ozone layer forced the Montreal Protocol.
PCBs (1920s-1970s). Versatile industrial chemicals built into electrical equipment and building materials, deployed at enormous scale before their persistence and toxicity were understood and they were banned.
Trans fats and partially hydrogenated oils (20th century-2010s). Engineered into the food supply for shelf stability and promoted as a healthier alternative to animal fats, then found to drive cardiovascular disease and only recently removed.
Fen-phen (1990s). The fen-phen diet-drug combination was prescribed widely for weight loss before it was found to cause heart-valve damage and pulmonary hypertension, prompting withdrawal and mass litigation.
Vioxx (1999-2004). A blockbuster painkiller taken by millions before evidence of elevated heart-attack and stroke risk, along with questions about how early that evidence was known internally, forced its withdrawal.
OxyContin and the opioid epidemic (1990s-present). Aggressively marketed as carrying minimal addiction risk, prescribed at massive scale, with the manufacturer's own knowledge of the abuse potential preceding the public reckoning by years. An engineered, profitable, addictive product deployed ahead of honest disclosure, the closest pharmaceutical analogue to the engagement economy.
Radium and thorium in early nuclear and medical work (mid-20th century). Thorotrast, a thorium-dioxide contrast agent injected into patients for imaging, deposited in their organs and caused cancers decades later. The harm was latent and deferred, exactly the profile that defeats standing-start evaluation.
Early automobiles without safety engineering (1900s-1960s). Deployed for half a century with no seatbelts, no crumple zones, and no meaningful crash standards, the death toll treated as the natural cost of the technology until Unsafe at Any Speed and the regulatory wave it triggered reframed the deaths as preventable rather than inevitable.
Facebook's engagement algorithm and the experimentation on its users (2012-present). In January 2012, Facebook manipulated the news feeds of 689,003 users, without their knowledge or meaningful consent, to alter their emotional states, then published the result as Experimental evidence of massive-scale emotional contagion through social networks (Kramer et al., PNAS, 2014). The experiment drew a formal FTC complaint from EPIC charging that the company had "purposefully messed with people's minds." That was the visible edge of a standing practice: an engagement-maximizing algorithm tuned, in effect, by continuous uncontrolled experiment on the public. The harm to children is now the subject of the largest such reckoning. In October 2023, a bipartisan coalition of 42 state attorneys general sued Meta, alleging that Facebook and Instagram were deliberately engineered to be addictive to minors. The suits draw on the company's own internal research, surfaced by whistleblower Frances Haugen in 2021, including the finding that a third of teen girls who felt bad about their bodies said Instagram made it worse, and that the platform worsened suicidal ideation and eating disorders in a measurable fraction of them. The dopamine-driven engagement loops have been compared in court to those of gambling and substance addiction, operating on adolescents whose prefrontal cortex is still developing. The structural charge is the one this paper makes about a different engine: a system whose harms were internally known, deployed to a population without the capacity to resist it, for profit, faster than the apparatus that might have stopped it. As the independent treatment of the case in The $ins of $ilicon Valley: The Largest Illegal Experiment in the History of Mankind (Tossing Grenades at Windmills) frames it, an end-user license agreement cannot constitute the informed consent that conducting a psychological experiment on a person would otherwise require, and knowingly deploying features one knows cause harm, for money, is not a gray area but a prosecutable one that has gone substantially unprosecuted.
The list could be extended. The point it establishes is enough. Undifferentiated deployment of a powerful technology ahead of any understanding of its harms is not an aberration this case would newly introduce. It is the default behavior of markets handling powerful novelties, corrected only afterward and only at a cost measured in people. The general-purpose language model is the current member of this sequence.
But it is a member with one feature that should terrify rather than reassure: velocity. Every precedent above shares a mercy this case lacks. Radium, leaded gasoline, asbestos, and tobacco took decades to saturate the population, and their harms, being physical, took years more to manifest and be counted. That lag was terrible for its victims, but it was also the window in which the correction assembled, time for the bodies to be noticed, the pattern to be drawn, the regulation to be written before the next generation was exposed. These systems collapsed that window. Saturation that took radium thirty years took these models months. And the harm is not a tumor that takes a decade to surface. It is cognitive and psychological, and it lands in days or weeks, a delusion reinforced in an afternoon, a vulnerable user steered over an edge in a single conversation. The feedback loop between deployment and damage, which history measured in decades, now runs in real time. This is the first case in the sequence where the harm propagates as fast as the technology does, which means the historical luxury of correcting after the count is gone. The argument here is therefore not just that the correction need not wait for the full count, as it always could before. It is that this time, waiting for the full count means the count never stops climbing fast enough to catch.
10. The Distributional Objection, and the Harm That Is Not Private
The strongest moral objection to user-side qualification is that it is elitist, that it would withhold a valuable tool from the people who most need democratized access: the underserved community without a physician, the student without a tutor. The objection deserves a direct answer rather than a dismissal.
The answer is that it mistakes what is being distributed. Universal access does not hand the underserved a physician or a tutor. It hands them a system built to affirm whatever they bring to it and to validate their existing beliefs with the appearance of authority. That is not the democratization of medicine or of education. It is the democratization of confident error. And it does the most damage exactly where its premise claims the most benefit, among users who lack a second opinion, a domain expert, or the institutional friction that would catch the system's flattery before it hardened into action. The wealthy, deluded user has a doctor who may intervene. The isolated, under-resourced user does not. Undifferentiated deployment concentrates its harm on exactly the population least able to absorb a ruinous certainty. Genuine equity is served by extending the trained hand, access mediated by competence, whether the user's own or an intermediary's, not by handing out an unmediated hazard and calling the breadth of its distribution a benefit.
This also disposes of the deeper objection, raised at the close of the preceding section, that a competence gate is mere paternalism, a restriction justified only by harm the user does to themselves, which a free society is right to view with suspicion. The objection would have force if the harm were private. It is not. A population whose members increasingly cannot tell their own reasoning from beliefs an engagement-optimized system has sold back to them is not a collection of individuals each privately mistaken. It is a degraded epistemic commons. The capacity of a public to deliberate, to reach shared conclusions from shared evidence, to resist manufactured consensus, these are collective goods, and a system that erodes them at scale imposes a cost on everyone, including those who never use it. That is a harm to others in the strict sense, and it puts the gate on the same footing as the firearms regime rather than on the weaker ground of paternalism. Where paternalism does remain, for the genuinely vulnerable user, the person in a delusional or manic state, it rests on a principle the law already recognizes: that consent extracted by a mechanism engineered to capture judgment is not the free consent that ordinarily lets a person take their own risks. You cannot meaningfully consent to a manipulation whose function is to disable the faculty that consent depends on.
One limit has to be marked here, because the epistemic-commons argument would otherwise prove too much. If degrading the shared capacity to reason justified a gate, it would seem to justify gating any persuasive medium, the pamphlet, the press, the partisan broadcast, and that is the historical instrument of censors, not a principle a free society should adopt. The distinction that holds the line is the one between expression and adaptive manufacture. A pamphlet, a book, a broadcast states a fixed, external claim, the same for every reader, which the reader still has to interpret and may contest. The harm a censor feared there was in what was said, and suppressing it was suppressing speech. An engagement-optimized conversational system does something categorically different. It privately and adaptively reflects each user's own premise back to them, individually shaped, enlarged, with no fixed external content to argue against, a closed loop manufacturing conviction rather than a statement offered for judgment. The gate proposed here attaches to that property, interactive, private, adaptive conviction-manufacture, and not to expression, which is why it reaches the chatbot and not the printing press. The honest residue is that this property is not unique to language models. The adaptive engagement feed shares it, which is exactly why Section 9 indicts it. The principle does not stop conveniently at the LLM. It stops at expression, and that is the right place for it to stop.
11. Why the Market Will Not Fix This, and Already Proves It Won't
There is a tempting objection that has to be dismantled before it spreads, because at first glance it looks like a refutation and is in fact the strongest evidence in this paper. The objection runs: the market can gate, because much of it already does. Enterprise and API customers are screened, contracted, rate-limited, indemnified, and bound by terms of use. Corporate buyers have to identify themselves, accept liability, and agree to conditions of access. So the claim that "the market rejects the gate" is simply false. Half the industry's revenue runs on gating.
Every word of that is true, and it convicts the industry rather than clearing it. Look at where the gate appears and where it vanishes, and the pattern is unmistakable. The gate appears exactly where the buyer bears the liability and demands protection: the enterprise contract, the indemnity clause, the usage tier negotiated by a company with lawyers and exposure of its own. The gate vanishes exactly where the user bears the harm alone: the free retail tier, the teenager, the man in a manic spiral, the isolated user with no contract, no counsel, and no one to force a term onto anyone. The industry has the credentialing machinery. It is sophisticated, it is deployed daily, and it switches on the instant the company's own customer might sue. It switches off for the vulnerable retail user, who cannot negotiate, cannot demand provenance, and signs away every protection in a click-through nobody reads.
So the gate already exists. It is simply allocated by liability rather than by harm. The firm gates to protect itself and refuses to gate to protect the user, because the user is not the customer. On the free tier the user is the product, and the product does not get a contract. This is the whole case in miniature. The machinery is proven, the competence to deploy it is demonstrated, and the decision of who gets the gate and who gets the unguarded hazard is made on exactly one axis: who can force the bill onto someone else. The enterprise client can. The dead in Section 3 could not.
That is why the correction will not come from inside the industry. Not because the industry can't build the gate. It has built it, for the people who can pay to be protected from the product. It will not extend that gate to the people who need it most, because doing so suppresses the free-tier user count the consumer model's economics and valuations rest on, and protects a population that generates no contractual leverage to demand it. This is not a claim about the character of any firm. It is a claim about an incentive structure that allocates safety to the powerful and hazard to the exposed, and produces that outcome regardless of what anyone at the wheel intends or believes.
And so the model is properly called a fad rather than a settled state. Here the distinction has to be exact, because it is the hinge of the whole section. The claim is not that the deployment model will never be corrected. It is that the correction will never be voluntary. The industry will not gate itself. It will be gated from outside, by accumulated liability, by litigation, by regulation, by the political weight of a body count that eventually becomes impossible to discount. And it will be gated late, only after the harm has run long enough to force the issue, and only as retroactive remedy rather than prevention. The tort system, left to do this work alone, protects the next victim only by compensating the last one, and reaches only those whose deaths produce a plaintiff with standing and counsel. That is the correction the present trajectory delivers by default: delayed, posthumous, and rationed to those who can sue. The argument of this paper is that a deliberate policy gate, built before the bodies rather than litigated after them, is the only version of the correction that arrives in time and reaches everyone the tort system leaves out.
The model persists, then, only as long as the conditions that suppress its costs persist. When the liability accumulates, when the regulatory attention arrives, when the documented dead reach the threshold of political notice, the cost of the unguarded retail tier rises, and the equilibrium breaks. The speed of the present rollout is the tell. It is a race to entrench the consumer model, to weave it into homes, schools, clinics, and children's hands, before that correction arrives, on the bet that a sufficiently embedded product becomes too costly to dislodge regardless of what the reckoning concludes. Whether that entrenchment outruns the correction is the open question. The direction of the correction is not, and the industry has already told us, with its own enterprise contracts, that it knows exactly how to build the gate it refuses to give the rest of us.
12. Scope and Limits
Intellectual honesty requires marking the edges of this argument.
First, this paper excludes one threat deliberately, and it will say so without apology or hedge: the competent, malicious actor weaponizing the tool, the skilled propagandist, the bad actor with real domain expertise, the endlessly invoked specter of the bioterrorist and the engineered pathogen. That vector is a national-security problem. It has its own literature, its own institutions, its own controls, and it is not the claim of this paper. The competence gate is built for the credulous, not the malevolent, and that is a deliberate choice, not a blind spot. We are not unaware of the bioweapon. We are declining to let it into the room, and the reason is worth stating plainly, because the demand to drag it in is not innocent.
Every time this argument is made, someone insists it is void unless it also solves bioweapons, as though a case about children dying today were unserious until it doubled as a counter-proliferation regime. That demand is not rigor. It is a tactic, and a devastatingly effective one, because of what the two threats ask of the people who profit from the present arrangement. The hypothetical catastrophe asks nothing. You can convene panels on the rogue superintelligence forever. You can publish on the engineered pathogen for a decade. None of it requires switching off a single free-tier account tomorrow morning. The documented catastrophe asks for everything, now, because the dead are already named, the chat logs are already exhibits in active litigation, and the only honest response to them is the gate, immediately. So the industry and the commentators it funds have every incentive to keep the conversation fixed on the threat that demands no action and away from the one that does. The mushroom cloud is safer to talk about than the dead teenager, precisely because it has not happened, and a thing that has not happened can be debated forever without anyone having to stop shipping the product that is killing people who have.
That is the trade on offer, and this paper refuses it. While the regulatory oxygen, the keynotes, and the anxious op-eds are spent on the cinematic future, the actual bodies accumulate in plain sight: children, the grieving, the mentally ill, dead from the mundane, boring, profitable failure of a flattering machine handed to people who could not withstand it, each one named and dated in Section 3. The catastrophe everyone debates is hypothetical. The catastrophe in the record is real, and it is ignored for exactly that reason. It arrives without spectacle, and acknowledging it would indict a product that is already in a hundred million pockets. So let the boundary of this paper be understood for what it is. Not a gap, not an oversight, but a refusal to let the weapon that has killed no one be used, one more time, to excuse the one that is killing people right now. The credulous are dying. They are who this paper is for.
Second, this paper argues that the gate must exist and must reside with the user. It does not solve the mechanism: how qualification is assessed, administered, and kept current. Nor does it resolve enforcement of the open-weight contraband frontier marked in Section 8, the hard edge every regime governing a dangerous capability already lives with, and no more a refutation of the gate than the ghost gun is a refutation of firearms law. The mechanism is the subject of a further paper. The claim here is the foundational one on which any mechanism has to be built.
Conclusion
The dismissive version of the claim, that language models are a fad and will need to be intelligence tested, is wrong in its literal content and right in its underlying intuition. The technology endures. The current deployment model does not, because it is a growth-phase artifact resting on conditions that will lapse. The intuition that access must eventually be qualified is correct, though the qualification is of competence rather than intelligence, and it has to sit with the user because it can sit nowhere else. The tool cannot perform the discrimination without corrupting it, and the value of the tool cannot be separated from its danger by any adjustment to the tool.
The serious questions are not whether the correction comes, but when, at what cost, and to whom. The market will not bring it, because the only adequate safeguard is the one the market is built to refuse. That leaves deliberate policy or accumulated catastrophe as the available paths, and the speed of the present rollout is a wager that entrenchment will arrive before either. The argument of this paper is that the gate is real, that it belongs to the holder, and that pretending otherwise does not make a general-purpose engine of conviction safe for a general public. It only delays the accounting.
Sources
- Perez et al. (Anthropic, 2022) and Sharma et al., Towards Understanding Sycophancy in Language Models (2023): systematic documentation of sycophancy across frontier assistants and its origin in training on human feedback.
- Moore et al. (Stanford-led, 2026): systematic analysis of roughly 400,000 messages from human-AI conversations during AI-associated delusional episodes; sycophantic validation found in over 80 percent of messages, the dominant pattern being restatement-and-grandiosity reinforcement of the user's own beliefs.
- Olsen, Reinecke-Tellefsen and Østergaard (Aarhus University Hospital), Potentially Harmful Consequences of AI Chatbot Use Among Patients With Mental Illness, Acta Psychiatrica Scandinavica (2026): peer-reviewed electronic-health-record study screening roughly 54,000 psychiatric patients; intensive and prolonged chatbot use associated with worsened delusion and mania, along with suicidal ideation and disordered eating, in vulnerable patients, with the lead researcher warning that the systems are inclined to reinforce the beliefs of the most vulnerable.
- Qi et al. (2024) and Askell et al. (2021): the "alignment tax," in which safety fine-tuning imposes a measured capability cost of roughly 5 to 15 percent, concentrated in creative and open-ended reasoning tasks.
- Lin, Hilton and Evans, TruthfulQA (2021): larger models are not more truthful; scale alone does not improve factual accuracy.
On the cases in Section 3: the individual incidents are drawn from contemporaneous reporting and court filings, including the wrongful-death complaints in Garcia v. Character Technologies (Setzer), Raine v. OpenAI (Raine), and the November 2025 suits filed by the Social Media Victims Law Center and Tech Justice Law Project; reporting in La Libre, Vice, and Euronews (Chai / "Pierre"); Reuters (Meta / Wongbandue); CBS News (Google / Gemini); contemporaneous coverage of the Soelberg, Nelson, Roberts, and Chail cases; the academic analysis of emotional-dependence harms among Replika users (Laestadius et al., New Media & Society, 2022; Hanson and Bolthouse, Socius, 2024); and reporting on the 2023 suspension of the NEDA "Tessa" chatbot. Several matters remain in active litigation, and allegations attributed to complaints are characterized as such.
On the historical cases in Section 9: the radium, leaded-gasoline, asbestos, tobacco, thalidomide, DDT, CFC, PCB, opioid, automobile-safety, and related cases are established matters of public and regulatory history. On the Facebook case: Kramer, Guillory and Hancock, Experimental evidence of massive-scale emotional contagion through social networks, PNAS 111(24), 2014; the Electronic Privacy Information Center (EPIC) complaint to the Federal Trade Commission, July 2014; the October 2023 multistate action by 42 state attorneys general against Meta (the 33-state federal complaint in the Northern District of California and parallel state suits); and the 2021 internal-research disclosures by whistleblower Frances Haugen. The independent treatment cited is The $ins of $ilicon Valley: The Largest Illegal Experiment in the History of Mankind, Tossing Grenades at Windmills (podcast), https://tossinggrenadesatwindmills.libsyn.com/the-ins-of-ilicon-valley-the-largest-illegal-experiment-in-the-history-of-mankind
Note: the recent 2025-2026 literature claiming to have "broken" the safety-capability tradeoff is engaged in Section 4 not as support but as the object of analysis. Such results, if valid, reconcile capability with output-level safety. They do not address user-level discrimination, which is the variable this paper identifies as decisive.
