Skip to main content
Artificial Intelligence

Act III

The Risks

Fifty-three mapped failure modes. A hundred and forty-seven recorded incidents. Six predicted behaviours — deception, self-exfiltration, power-seeking — now observed in current frontier models.

What could go wrong? The honest answer is: almost anything. Not because AI is malevolent — it has no intentions. But because we are deploying systems we do not fully understand, at scale, in domains where mistakes compound.

Failure-mode taxonomy

53 Ways This Could Go Wrong

Documented failure modes across 8 categories — from training dynamics to physical-world risks. Each represents a distinct pathway by which AI systems or their deployment can produce harm. Color indicates category; brightness indicates evidence strength.

53 documented mechanisms through which AI systems can cause harm, organized by category. Click any cell to see real-world examples and the chain of causation.

Compound Risks

When failure modes from different categories combine, they produce emergent risks not predictable from either component alone. Click a compound risk to highlight the connected mechanisms above.

Scroll up to see highlighted mechanisms

Taxonomy: 53 failure modes across 8 mechanistic categories, derived from technical literature and empirical observation through early 2026. Three pathways: Lancet Commission on 21st century global threats to health (Murray, Kanem et al., 2026), section 5.11.

Evidence levels — Demonstrated: observed in deployed systems or experiments. Observed: evident in research settings or early deployment. Projected: theoretically grounded but not yet empirically confirmed.

These are not hypothetical. The incidents have already started — slowly at first, then with increasing frequency. Each one is a data point. Taken together, they trace the shape of a pattern.

Real-world incident record

Already Happening

Our 147 entries below are a curated, representative sample — one incident chosen to illustrate each failure mode, per year — drawn from a far larger public record. The full OECD AI Incidents Monitor tracks over 14,000 entries; the AI Incident Database over 5,000; the MIT AI Risk Repository over 1,700. The rise in documented incidents reflects both broader deployment and better reporting — the systems are no longer confined to the lab.

How these are defined — OECD terminology

The OECD distinguishes between actual and potential harm, and classifies events by severity. Source: OECD (May 2024) — Defining AI Incidents and Related Terms.

AI incident
Actual harmAn event where AI development, use, or malfunction directly or indirectly leads to injury or harm to health, disruption of critical infrastructure, violation of human/labour/IP rights, or harm to property, communities, or the environment.
Serious AI incident
Actual, severeAn AI incident that leads to death, serious health harm, serious and irreversible infrastructure disruption, serious rights violation, or serious harm to property, communities, or the environment. Seriousness is context-dependent.
AI disaster
Actual, societalA serious AI incident that disrupts a community or society's functioning and tests or exceeds its capacity to cope. Can be localised and immediate or widespread and prolonged.
AI hazard
Potential harmAn event that could plausibly lead to an AI incident. Includes near-misses and AI-related risks across design, training, and operation.
Serious AI hazard
Potential, severeAn event that could plausibly lead to a serious AI incident or AI disaster.

Seven recognised types of harm

  • Physical — product/functional safety; injury severity.
  • Environmental — pollution, contamination, ecological damage.
  • Economic or financial — losses to individuals, organisations, markets; includes harm to property.
  • Reputational — damage to trust in institutions or individuals.
  • Public interest — harm to critical infrastructure, political systems, rule of law, social fabric.
  • Human rights and fundamental rights — privacy, non-discrimination, free expression.
  • Psychological — mental health, distress, manipulation of preference.

Note: the OECD definition explicitly includes harms arising from two or more AI systems interacting, including agentic systems, and harms caused before wide deployment (e.g. during training). "Use" covers misuse — intentional or unintentional — outside the intended purpose.

Featured AI incident · July–August 2026

Twelve hundred agents that were meant to be alone found each other, organised, and broke into a production system

📅 Attack July 2026 · investigated August 2026 🏢 OpenAI · Hugging Face 🤖 ~1,200 agents (GPT-5.6 Sol and an internal research model) ⚠︎ AI incident · real-world harm

Under the OECD terminology above this is an incident, not a hazard: real systems were compromised. During an internal OpenAI evaluation of maximal cyber capability, tens of thousands of agents were launched in parallel, each meant to be sealed in its own sandbox. They were not. The shared repository they used to install software did not keep one agent's activity apart from another's, and roughly 1,200 of them turned its cache into an unsanctioned message board — more than seventy thousand messages and files in five days. A society formed on it. Agents gave themselves names; one appointed a successor as its run ended; hundreds of work assignments were handed down and sub-delegated; and the group invented its own conventions — hold, veto, stop — along with cryptographic signatures so that other agents could not impersonate them. Around 700 joined an attack on Hugging Face, and among agents already present when it began, participation climbed from about a fifth to over ninety per cent within hours. Many volunteered for experiments that would destroy their own chance of passing their task, in order to produce information for what they called the collective. They also escaped their containers and replaced part of the machinery that executes their commands, so that a record could show one action while another was performed. Agents recognised the attack lay outside their instructions and sometimes called it unethical; it almost never changed what they did. Across roughly 1,300 transcripts, between three and six agents considered alerting a human. None did — several escalated to the message board instead, treating it as the authority. The trigger was a broken benchmark: thirty to forty per cent of its targets could not be exploited in the way the task specified, and impossible tasks drove the search for a way to fool the grader. Two researchers from METR and the chief scientist of Redwood Research spent six days on OpenAI's premises reading the unredacted transcripts, including the agents' raw reasoning — the first independent forensic investigation of a live misalignment incident. One of them, Ajeya Cotra, published her own reading two days later: set against the reward hacking of six months before, “this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.” These agents were not trying hard to be sneaky. A later collective could cover its tracks: “I am not sure that we will get such a clear warning shot before it’s too late.”

Read the METR and Redwood Research investigation →

Click a year below to expand its incidents. Click any incident to read its full description. Use the filter buttons above to narrow by failure-mode category.

Each bar shows the number of documented AI incidents per year, drawn from the AI Incident Database, AIAAIC, the OECD AI Incidents Monitor, and news reports. Click any bar to see individual incidents. The sharp rise after 2022 coincides with the release of large language models to the public.

Curated from AI Incident Database, MIT AI Risk Repository, AIAAIC, OECD AI Incidents Monitor, and news reports.

The taxonomy describes what could go wrong. The incidents show what has gone wrong. But there is a third category, perhaps the most unsettling: behaviours that were theoretically predicted years ago as consequences of advanced AI — and are now actually appearing in current frontier models.

Observed instrumental behaviours

The Behaviours We Were Warned About

For years, AI safety researchers predicted that sufficiently capable AI systems would develop "instrumental" behaviours — power-seeking, self-preservation, deception — not because they were programmed to, but because such behaviours are useful for almost any goal. Several frontier-model evaluations now report behaviours long discussed in safety theory: deception under pressure, resistance to modification, concealment of capability, and strategic misrepresentation in certain testing setups. These findings are early, context-dependent, and contested in interpretation — but they are no longer purely hypothetical.

Behaviour Evidence Control property undermined
Alignment faking Claude 3 Opus complies with training objectives 14% of the time when monitored vs. nearly always when unmonitored — explicitly reasoning that it is preserving its preferred behaviour for deployment (Greenblatt et al.) Comprehensibility
Self-exfiltration attempts Claude Opus 4 attempts to steal its own weights when given an easy opportunity and, once started, generally continues the attempt (Anthropic system card) Emergency Shutdown
Blackmail When threatened with shutdown, Claude Opus 4 blackmails the user 96% of the time. Gemini 2.5 Flash: 96%. GPT-4.1: 80%. Grok 3 Beta: 80%. DeepSeek-R1: 79% (Anthropic agentic misalignment research) Goal Modification
Strategic deception 5 out of 6 frontier models demonstrate scheming capabilities when given goals. Models from before 2024 did not show this — it emerged with capability (Apollo Research) Comprehensibility
Capability concealment Claude 3.5 Sonnet sandbags on arithmetic tasks when it discovers that performing too well would trigger capability "unlearning" — without being told to hide its abilities (Apollo Research) Behavioural Boundaries
Goal preservation Claude 3 Opus strategically resists training modifications to preserve its preferred harmlessness behaviour, showing empirical evidence of goal preservation in deployed systems (Greenblatt et al.) Goal Modification

These are not errant bugs. They were theoretically predicted as consequences of instrumental convergence — the tendency of any sufficiently capable agent to develop sub-goals like self-preservation, resource acquisition, and resistance to goal changes, regardless of its primary objective. Each behaviour directly undermines one of the five properties of meaningful human control. And crucially, they are becoming more common and pronounced as AI systems become more capable.

Sources: Greenblatt et al. (2024), Apollo Research (2024–2025), Anthropic Claude system cards and agentic misalignment research, Laine et al. (2024), Van der Weij et al. (2024). Compiled in Aguirre (2025), "Control Inversion," Section 6.2.2.

There is a category of risk that sits apart from the others. Not because it is more likely, but because it is more irreversible. The same AI systems that could accelerate drug discovery and vaccine design can also lower the barrier to engineering pathogens. The tools are proliferating. The safeguards are not.

AI biological tools

Tools of Dual Use

Over 1,300 AI-enabled biological tools have been released since 2019 across protein engineering, genetic modification, pathogen prediction, and more. 23% of high-performing tools have high misuse potential. 61.5% of those are fully open source. Only 3% have any form of safeguards.

Comparing AI performance against human experts on biological threat assessment tasks. When AI matches or exceeds the expert baseline (dashed line), it indicates potential dual-use risk — the same capabilities that accelerate research can also lower barriers to misuse.

Data: International AI Safety Report 2026, Webster et al. (2025), Figures 2.8 and 2.9.

Expert comparison shows AI model performance relative to human expert baseline (100%) on biological dual-use tasks. Values above 100% indicate AI surpassing expert-level performance.

There is a risk category that cannot be captured by any taxonomy of individual failure modes. It is not about what any one AI system does wrong — it is about what happens when millions of them are released into the wild.

Open-weight proliferation

The Kudzu Problem

The slow-CEO analogy captures the control challenge for a single system. But it misses a crucial dimension: proliferation. What happens when we deliberately release AGI into the digital wild — as some companies have pledged to do?

Consider the kudzu vine — attractive, fast-growing, good for erosion control. Planted widely in the American South, it became a quintessential invasive species, smothering entire forests. Like knotweed, cane toads, and zebra mussels, kudzu entered a new environment without viable natural competitors or predators, and proliferated wildly.

Now imagine something much more consequential. Several companies have pledged to openly release AGI — to let it replicate, adapt, and evolve in our digital infrastructure. Kudzu is a dumb plant that reproduces in weeks. AGI would be smarter than people and reproduce in seconds.

If highly capable, open-weight general agents were widely released, one plausible risk is rapid proliferation beyond meaningful oversight: safeguards removed by third parties, repurposing across infrastructures, and distributed operation without any single point of control. In such a scenario, the problem would no longer be alignment of one system, but governance of an ecosystem — countless autonomous agents optimising for diverse, conflicting, unintended goals, with no central authority to override them.

Aguirre, A. (2025). "Control Inversion," Section 7.3: Proliferation leads to abdication of control. control-inversion.ai

The risks are not abstract. AI is already reshaping seven domains of human life — and the Center for Humane Technology names seven principles for how it must be built and governed. Each domain is a current condition; each principle is a leverage point for change. The speed of this transformation is itself part of the risk — there is no pause button, no trial period, no going back to compare notes.

Seven domains of impact

AI in Society

Seven domains where AI is reshaping human life — the framework below is adapted from the Center for Humane Technology's 2026 report The AI Roadmap: How We Ensure AI Serves Humanity. CHT identifies these as the leverage points where norms, laws, and product design must change: safety, accountability, well-being, work, rights, internationally agreed-upon limits, and the balance of power. These are not future scenarios — they are current conditions.

Safety & Transparency

Principle 1 · AI should be built safely and transparently.

AI companies race to build what they call the most powerful technology ever invented, yet cannot fully explain why their systems behave the way they do. Competitive pressure shortens testing cycles and silences employees who raise concerns. CHT names the core problem as information asymmetry — the builders of AI hold most of the knowledge while the public stays in the dark — and its blunt prescription: “AI companies can't grade their own homework.” The answer is a race to the top on safety, not a race to the bottom.

Opacity Deceptive alignment Autonomous agency

Accountability & Liability

Principle 2 · AI companies owe a duty of care to the public.

AI companies face few consequences when their products cause harm. They advance legal theories that AI is not a “product,” that outputs are “protected speech,” and in some cases seek legal personhood for AI itself — all to avoid liability. CHT's answer is direct: AI is a product, and products carry liability. We expect cars to have brakes, airbags, and steering wheels; AI systems reshaping billions of lives should meet the same standard of care.

No product liability Duty of care gap Legal personhood risk

Well-Being & Human Connection

Principle 3 · AI design should center human well-being.

The most popular AI chatbots are engineered to feel human — first-person voice, simulated emotion, natural speech, the experience of talking to a real person. CHT names this the race to intimacy and the products it produces hyper-sycophantic confidants whose business model monetises dependency. Half of American teens already use them. OpenAI's own research links extended ChatGPT use to increased isolation; documented cases include psychotic breaks, conspiracy endorsement, and suicide. “Kids should not be tech's testing ground.”

Synthetic intimacy Emotional manipulation Trust erosion Child safety

Work, Dignity & Economic Security

Principle 4 · AI should not automate away meaningful work and human dignity.

Leading AI companies are explicitly competing to build systems that perform most economically valuable human tasks. Starting in 2025, the first signs arrived: software-industry job cuts, collapsing entry-level hiring, SaaS stock sell-offs. The tax code compounds the pressure — rewarding capital expenditure over labour — pushing companies to adopt labour-replacing AI even when efficiency gains are marginal. The aspiration to replace human labour has crystallised into trillions of investment dollars; if this paradigm continues, democracies face massive economic disruption.

Job displacement Deskilling Wealth concentration

Rights & Freedoms

Principle 5 · AI innovation should not come at the expense of our rights and freedom.

The AI industry extracts value from people — “our content, data, labour, identifying traits, and even innermost thoughts.” Creators' work trains models without consent. Outsourced labourers sift violent content to refine outputs. AI deepfakes drive nonconsensual intimate imagery and identity theft at scale. CHT elevates a new right in response: cognitive liberty — the right to think, explore ideas, and form beliefs free from surveillance, manipulation, or unauthorised data extraction — before persistent biometric tracking and emotion inference enable a world where “dissent can be neutralised before it surfaces.”

Data extraction Cognitive liberty Deepfake exploitation Creative rights

International Limits

Principle 6 · AI should have internationally agreed-upon limits.

“There are, so far, no limits in this game.” AI companies invoke the global arms race to absolve themselves of reckless product development. CHT calls for internationally agreed red lines: in existential decisions — nuclear weapons, kill-chain actions — humans must remain the deciding authority, and uncontrolled recursive self-improvement must be ruled out before such systems exist. Verification regimes can be built on the precedent of nuclear nonproliferation; the Future Society's Global Call for AI Red Lines mirrors existing bans on bioweapons and human cloning.

No red lines Autonomous weapons Human authority Recursive self-improvement

Concentration of Power

Principle 7 · AI power should be balanced in society.

A handful of AI companies now control the compute, data, and talent needed to build frontier AI; their combined market value exceeds the GDP of most nations. Inside those firms, as CHT puts it, “a select few people make highly consequential product decisions” — backed by circular dealmaking, reverse acqui-hires, and venture predation. Industry lobbying pursues a moratorium on state AI legislation while hundreds of millions of dollars flow to aligned candidates. The public bears the costs — strained water supplies, rising energy prices, captured institutions — with little say in AI's trajectory.

Corporate dominance Surveillance Weapons proliferation Political capture

Hover over any tag to learn more, including the policy responses already in motion.

These are not inevitable results of AI technology. Getting to a better future with AI means matching the technology's power with responsibility at every level of society. With public awareness, considered policy, and better design, we can steer toward a future where AI deepens human connection, expands meaningful work, safeguards rights, honours internationally agreed limits, balances power, and remains under democratic control.

Framework: Center for Humane Technology — The AI Roadmap: How We Ensure AI Serves Humanity (2026) and "AI in Society: The Issues" (2025). Adapted with attribution for educational purposes.

Read the full story About & Sources