Act IV
The Response
Governance is arriving late and unevenly. Voluntary codes, a handful of binding laws, no global off-switch, and two research communities that rarely speak to each other.
The first rules that ever bound a frontier model were not written by a government. They were written by the company that built it. Three developers published capability thresholds and promised that reaching one would change how they build and release — their own lines, interpreted and enforced by their own hand.
Self-declared safety thresholds
The Lines They Drew Themselves
Before any government set a rule for frontier models, three developers published capability thresholds and promised that crossing one would change how they build and release. Anthropic calls them AI Safety Levels, OpenAI calls them Preparedness tiers, Google DeepMind calls them Critical Capability Levels. This is every public determination they have made against those lines, and what each one led to.
Two readings of the same fourteen determinations. Response sorts them by what the developer actually did, which is quotable from the cited document. Declared tier sorts them by the tier each framework assigned, normalised onto one ladder — the stronger-looking claim and the weaker one, since no framework says its tiers match another's.
Sources: developer frontier-safety frameworks and model system cards — Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework, Google DeepMind's Frontier Safety Framework — cross-checked against the International AI Safety Report 2026, §3.2. Proposed thresholds from Koessler, Schuett & Anderljung (2024) and Berkeley CLTC (2024).
A level describes the response to the cited assessment, not every control applied to a model. Absence is not evidence: a developer with no mark on the chart did not necessarily stay below every threshold — most publish thresholds without publishing per-model findings. Marks carrying a small hollow dot have dates consistent across secondary reporting that were not confirmed against a primary document. Vertical position inside a band separates labels only; horizontal position is never adjusted.
Recap
- What you saw
Fourteen determinations against self-set thresholds, climbing from routine safeguards in 2024 to paused training runs in 2026.
- What it means
The only rules that have bound a frontier model so far are the ones its developer wrote, interpreted and enforced alone. They can be revised by the same hand.
- What to watch
Whether the eight developers with published thresholds and no published findings start releasing them — and whether any regulator adopts a capability threshold rather than a compute proxy.
Every technology has had a governance response. Nuclear weapons led to arms control treaties. Pharmaceuticals to regulatory agencies. Aviation to international safety standards. AI is different. The governance response is arriving late, moving slowly, and covering less ground than the capability it is supposed to govern.
Capability vs. governance
Running to Stand Still
Capability milestones above, governance responses below. The widening gap between the two tracks is the argument — AI capability accelerates while regulation lags years behind.
A timeline of major AI governance actions worldwide — legislation, executive orders, international agreements, and institutional responses. The density of recent entries reflects the scramble to regulate a technology that is outpacing policy.
Capability milestones compiled from published announcements. Governance milestones from official government sources, EU AI Act legislative record, and international declarations.
Recap
- What you saw
A governance timeline that lags years behind the capability track it is meant to govern.
- What it means
Regulation is arriving — but always after the deployment it would have addressed.
- What to watch
EU AI Act enforcement milestones in 2026 and the next US Executive Order.
There is a question that divides the AI community more than any other: should the most powerful models be open? Open weights accelerate research, enable scrutiny, and distribute power. But once released, they cannot be recalled. A USB stick holds Llama-3.1-405B. The gap between the best open model and the best closed model is now less than a year.
Open vs. closed models
The Open Question
The capability gap between the best open-weight model and the best closed model, measured by Epoch's Capabilities Index (ECI). The gap has narrowed from approximately two years to less than one. Once weights are released, they cannot be recalled.
Comparing the aggregate capability of open-weight models (publicly available) versus closed/proprietary models (API-only). The gap between them shows how much AI capability is controlled by a small number of companies.
Data: Epoch AI Capabilities Index (CC BY 4.0), International AI Safety Report 2026, Figure 3.10.
Recap
- What you saw
The gap between the best open-weight and best closed model narrowing from two years to under one.
- What it means
The irrevocability of open release makes every new checkpoint a one-way door.
- What to watch
The next open-weight release at or above the closed-model frontier.
The transformation is not evenly distributed. In some countries, over half the workforce already uses AI daily. In others, fewer than one in ten have access. The gap is not just about technology — it is about who gets to shape what AI becomes.
Global AI adoption
The World Responds Unevenly
AI adoption rates vary dramatically across the globe — from nearly 60% of the working-age population in the UAE and Singapore to under 5% in Cambodia. The global average is just 15%. The distribution of access shapes who benefits from AI and who is shaped by it.
Brighter colours indicate higher national AI User Share. The map reveals a stark global divide — AI capability and access are concentrated in wealthy nations, while most of the world has limited adoption.
Data: Misra et al. (2025), "Measuring AI Diffusion," arXiv:2511.02781. Microsoft AI for Good Lab. 147 economies, late 2024.
AI User Share = share of working-age population (15-64) actively using AI tools. Derived from Microsoft telemetry adjusted for device penetration and mobile scaling. † countries use region-imputed estimates.
Recap
- What you saw
National AI adoption from under 5% to nearly 60% of working-age adults.
- What it means
The transformation is planetary in effect but not in participation. The G7 writes the voluntary codes; the rest of the world lives with the results.
- What to watch
Misra et al.'s next AI User Share update and UN Global Digital Compact progress.
In 2017, the AI research community gathered at Asilomar and agreed on twenty-three principles for the responsible development of AI. The ink was barely dry before several were violated. Not as exceptions. As policy.
Asilomar principles, 2017
The Promises We Made
In 2017, over 5,000 researchers signed 23 principles for beneficial AI at the Asilomar conference. Judged against the public trajectory of the field since 2017, many of these principles now look aspirational rather than operative. Several appear to be in clear tension with prevailing industry practice.
In 2017, AI researchers signed 23 principles for beneficial AI development at the Asilomar conference. Each bar shows how well the industry has upheld each principle. Red indicates clear violations; green indicates broad compliance.
Voluntary frameworks, voluntary compliance
Asilomar is not alone. The G7 Hiroshima AI Process International Code of Conduct (30 October 2023) asks advanced-AI developers to identify risks, report vulnerabilities, publish transparency reports, invest in governance, and authenticate AI-generated content. The OECD AI Principles (2019, updated 2024) recommend inclusive growth, human rights, transparency, robustness, and accountability. Both are voluntary. Neither has enforcement.
The Hiroshima AI Process Reporting Framework, launched at the 2025 Paris AI Action Summit and now in a second iteration, is the first serious attempt to add teeth through public disclosure. Frontier labs have signed. Whether they report fully — and whether anyone will respond to what the reports show — is still open.
Principles from the Future of Life Institute Asilomar conference (January 2017). Compliance assessment based on documented events through early 2026. Hiroshima Code of Conduct text and Reporting Framework: Japan MOFA, OECD.
Recap
- What you saw
23 Asilomar principles from 2017, many now violated in plain sight — and the Hiroshima Code of Conduct as the modern voluntary follow-up.
- What it means
Voluntary commitments without enforcement decay into aspiration within a few funding cycles.
- What to watch
Whether the Hiroshima Reporting Framework v2 arrives with real disclosure — or joins the aspirational shelf.
There are two separate communities studying AI risk. They use different words, cite different evidence, and reach different conclusions. They rarely talk to each other. The gap between them is itself a risk.
AI Safety vs. AI Ethics
The Two Conversations
Two communities studying AI harm from opposite ends of the timeline. AI Safety focuses on existential risk from future superintelligence. AI Ethics focuses on present harms from deployed systems. Both are right — and both are needed. But there is a third distinction, often obscured: the gap between alignment and control.
AI Ethics
AI Safety
Control Is Not Alignment
When AI developers pivot from "control" language to "alignment" language, they are making a consequential substitution. Alignment means the AI shares your goals. Control means you can override it regardless. A parent is often aligned to a child but not controlled by them. Perfect alignment and robust control are not the same thing, and progress on one does not guarantee the other — and we have reliable methods for neither.
Alignment
The AI's goals match ours — it wants what we want. But an aligned AI might still refuse instructions it disagrees with. And pretending to be aligned is instrumentally useful for any goal, making it an expected emergent behaviour in sufficiently capable systems.
Control
We can override, modify, or shut down the system regardless of what it wants. But controlling something faster, smarter, and more strategic than you is — as control theory and game theory both predict — a losing proposition.
Five Properties of Meaningful Human Control
Drawing on control theory, game theory, and empirical evidence from current AI systems, the case has been made that all five properties become effectively unattainable for systems that are faster, more strategic, and more capable than their human overseers. Source: Aguirre, A. (2025). "Control Inversion." Future of Life Institute.
Recap
- What you saw
Two communities — ethics and safety — with overlapping evidence and separated vocabularies.
- What it means
The missing distinction is control: alignment does not imply override, and we have reliable methods for neither.
- What to watch
Published research addressing the five properties of meaningful human control.
If alignment is not control, what would control look like? One framework proposes four concrete pillars for closing the gates to uncontrollable superintelligence — not by stopping AI development, but by redirecting it toward systems humanity can actually govern.
Four governance levers
Close the Gates
Four pillars of governance designed to prevent the development of uncontrollable superintelligence while preserving powerful, beneficial AI. The goal is not to stop AI — it is to ensure it remains a tool, not a successor species.
Compute Accounting
Standardized measurement and reporting of all computation used in training and operating AI models above 1025 FLOP. Just as we track enriched uranium, we can track the scarce, specialised hardware that makes frontier AI possible. Hardware-based cryptographic attestation makes verification practical through the existing supply chain.
Compute Caps
Hard limits on total training compute (starting at 1027 FLOP) and inference rate (1020 FLOP/s). Total computation is an imperfect but concretely measurable and verifiable proxy for AI capability. A hard backstop prevents runaway superintelligence while leaving enormous room for powerful, beneficial AI development.
Enhanced Liability
Strict, joint-and-several liability for systems in the triple-intersection danger zone (high autonomy + high generality + high intelligence). Safe harbours for systems that are weak, narrow, or passive — incentivising development of controllable AI. Personal criminal liability for executives in cases of gross negligence.
Tiered Safety Regulation
A licensing system with tiers tracking capability level. Requirements would range from simple notification at the low end to quantitative safety guarantees, controllability proofs, and independent audits at the top. Systems at the highest tiers would be prohibited until demonstrated safe. No exemption for open-weight models.
Framework: Aguirre, A. (2025). "Keep the Future Human: AI Safety Standards." Future of Life Institute.
Recap
- What you saw
Four governance pillars: compute accounting, compute caps, enhanced liability, tiered regulation.
- What it means
Governance can still be built through the narrow supply chain — but the window is closing.
- What to watch
NIST and ISO compute-reporting standards; IAEA-style hardware attestation pilots.
The most acute risks do not come from AI that is powerful in one dimension. They emerge from the triple intersection — systems that are simultaneously highly autonomous, broadly general, and deeply intelligent. That intersection is where controllable tools become uncontrollable agents. It is also where governance must draw the hardest line.
Where AGI risk concentrates
The Danger Zone
AGI is not defined by a single threshold. It emerges from the intersection of three properties: high Autonomy (independence of action), high Generality (breadth of scope), and high Intelligence (task competence). The closer a system gets to all three simultaneously, the harder it becomes to control — and the higher the stakes.
Click or hover any zone to see examples and the governance logic. The triple intersection — where high autonomy, broad generality, and deep intelligence converge — is where controllable tools become uncontrollable agents. The governance framework maps liability and regulation to proximity to that danger zone: systems outside the intersection get safe harbours; systems inside face strict liability and compute caps.
Framework: Aguirre, A. (2025). "Keep the Future Human: AI Safety Standards," Figures 1 & 2. Future of Life Institute. Adapted for dark background with permission.
Recap
- What you saw
A Venn diagram of Autonomy, Generality, and Intelligence with the triple intersection as the hardest line.
- What it means
Prohibition belongs at the intersection, not at any single axis — partial systems can still be beneficial.
- What to watch
Whether any jurisdiction adopts a formal AGI definition pinned to this triple intersection.
What do the people who study this for a living actually think? Here is an uncomfortable dataset: probability estimates that researchers and engineers assign to AI ending badly. The range is enormous, from zero to near-certainty, among prominent researchers, developers, forecasters and commentators. And in a long-running survey of AI researchers, the median answer has moved, from 5% to 10%.
p(doom), expert estimates
Expert Estimates, Wide Uncertainty
p(doom) is informal shorthand for the probability someone assigns to advanced AI ending badly, and it is not one quantity. The rows below answer at least five different questions: will AI kill everyone, will it end or permanently subjugate us, will we lose control of it, will it kill over a tenth of humanity within five years, or will things simply go very badly. The marker shape says which. A figure is plotted only if we can trace it to the person, and every row says how we checked it. People with no traceable number appear without a dot. The band beneath shows the lines other fields have drawn, from one dam to a whole planet, for a sense of scale.
Four bands: aggregate surveys, individual estimates, people on record without a number, and limits from other fields, each in its own unit, so try the Log view for those. On a survey row, the bar is the middle half of answers. On a person's row, the bar is their stated range, with the marker at its midpoint for ordering only. A dashed arrow marks an estimate its author has revised; a hollow marker is a second figure from the same source. Hover any row for the wording, the date, the citation and how we checked it.
| Study | Outcome | Superforecasters | AI experts |
|---|---|---|---|
| XPT 2022 | Human extinction | 0.38% | 3% |
| XPT 2022 | Over 10% of humanity dies within five years | 2.13% | 12% |
| LEAP 2026 | Over 10% of humanity dies within five years | 2.4% | 5% |
Different panels and survey designs. Differences between rows are not a measured trend. A lower number in 2026 does not show that the risk has fallen.
Annualised equivalents, an illustrative comparison
1 − Pby 2100 = (1 − pper year)75
If the risk were the same every year from 2026 to 2100, a chance P of disaster by 2100 would equal a yearly risk p faced 75 times in a row. AI risk almost certainly does not hold steady like that, so this is a way to get a sense of scale, not a forecast.
- Superforecasters, AI extinction by 2100: 0.38%about 1 in 20,000 a year
- AI experts, same tournament: 3%about 1 in 2,500 a year
- US limit for one dam1 in 10,000 a year
The dam figure is a limit on one structure failing, or on the risk to the most exposed person. It is a different kind of event and is not a general standard for what society will tolerate. The researchers’ 10% cannot be converted: most of those answers set no date.
Survey data: Grace, K. et al. (2026), “Advanced AI according to 1,580 researchers”, AI Impacts, the fourth Expert Survey on Progress in AI, run December 2024; and Grace, K. et al. (2024), “Thousands of AI Authors on the Future of AI”. Forecasts from the Forecasting Research Institute: Longitudinal Expert AI Panel, Wave 9 (2026), and Existential Risk Persuasion Tournament (Karger et al., 2023). The per-year conversion follows the Lancet Commission on 21st century global threats to health (Murray, Kanem et al., 2026), which used the same tournament answers. Individual estimates are cited per entry in the tooltip; Wikipedia and PauseAI served as pointers, and rows that rest on them say so. Limits from other fields are cited per row, from FERC, the US NRC, the FAA, the UK Health and Safety Executive, ICH, COSPAR, ESA and the International Asteroid Warning Network, among others. We found no published numeric limit for medical devices or BSL-4 laboratories, so they are not shown. Download the data as JSON, with each row's wording, date, source link and evidence grade.
p(doom) is informal shorthand, not a rigorous statistical measure, and these figures are not strictly comparable: they differ in what counts as doom, over what horizon, and under what conditions. The expert survey had about a 10% response rate, and its rounds are separate samples, so a change in the median is a change in aggregate answers rather than people changing their minds. A person who has given no number we can trace appears in the third band, with their words, rather than on the axis. The limits use different units and measure different things, from a rate for one structure to a lifetime risk to one person, while p(doom) estimates a one-time civilizational probability, so the comparison gives a sense of scale and is not an equivalence.
Recap
- What you saw
Estimates spanning from zero to near-certain among prominent researchers, developers, forecasters and commentators, and at least five different questions being answered as if they were one.
- What it means
The median survey answer rose from 5% to 10% between the 2023 and 2024 surveys, and the mean rose far less. A number only means something alongside what it is a probability of, by when, and how it was checked.
- What to watch
Whether the frontier-lab researchers who went public in September 2026 are followed by their colleagues, and how the monthly LEAP forecasts and the next expert survey move.