Powers of Two
The scale of artificial intelligence, from one byte
A square’s area is its quantity scroll or pinch to grow click any name
In the summer of 1949 Claude Shannon, who had just founded information theory, pencilled a list: how many bits it takes to hold things, from a punched card to the Library of Congress, which he put at about 1014 bits. An early ladder like this one.
Switch the ladder on to light his rungs, the pencil-grey squares, among today’s data. His genome was off by about sixty thousand times; his Library of Congress now fits on one hard drive.
How to read it
- Data, in bytes. One cell is one byte, one letter.
- Computing, in operations. One cell is one arithmetic step: one multiplication or one addition.
- Shannon’s own 1949 estimates.
- Biological anchors: Ajeya Cotra’s 2020 estimates, each modelled on biology, of the computing needed to train an AI able to transform the economy.
- A scenario, not a forecast.
- Limits: totals, projections and the ceilings set by physics.
- Units: 1 KB, 1 MB, 1 GB… and a thousand, a million, a billion… operations
- Bold outline: measured or defined
- Paler outline: estimated
- Thin outline: our arithmetic from sourced figures
About
Every quantity is a square whose area is the number itself, all standing on the same corner, so each square holds every smaller one. Make the view ten times wider and it holds a hundred times more. That is how 46 powers of ten fit in 23 steps of zoom: from one bit to all of Earth’s sunlight spent at the Landauer limit, the least energy physics allows to erase one bit. Four limits of physics lie beyond any square this page can draw; they sit on the rail above 1045.
In 1949 Claude Shannon pencilled a ladder like this one, from a punched card to the Library of Congress, and stopped at 1014 bits. Two years later he measured English at about one bit per letter by asking people to guess the next letter of a text: the same game every language model now plays.
The two wings are tied by a rule of thumb: training a model takes about six operations per parameter for every token it reads. GPT-3, with 175 billion parameters and 300 billion tokens, comes to 6 × 175 × 109 × 300 × 109 ≈ 3 × 1023 operations.
“Emergence” is debated. Wei and colleagues (2022) reported abilities that seemed to switch on suddenly, most of them between 1022 and 1024 operations. Schaeffer and colleagues (2023) showed that many of those jumps soften into steady curves when answers get partial credit. How much is real and how much comes from the scoring is still argued.
Since GPT-4 in 2023, most frontier laboratories disclose almost nothing about their largest models. Much of the training compute above 1025 is Epoch AI’s estimate, and it does not count the extra training that teaches models to reason, or the computing they spend thinking at answer time.
The flow lines behind the squares are a picture of emergence, not a measurement of any model. Low on the ladder they tangle around many small whirlpools. As the scale grows the whirlpools pair up, and inside the band where Wei and colleagues saw abilities appear they bind and vanish, leaving long, ordered flow. Tiny whirlpools in a thin film of liquid helium do the same as it cools: physicists call it the Kosterlitz–Thouless transition.
The faint text on the data side is quoted: Claude Shannon’s 1948 approximations to English, built by hand, then GPT-2 and GPT-3. The ground’s colour follows the scale, cool at one byte and warm at the top.
Bytes are decimal (1 KB = 1,000 bytes). A token, the unit language models read, is a word or a piece of one: English averages about four bytes per token and six per word. Every landmark names its source on its card.
Data and licences
- Epoch AI, Notable AI Models CC BY 4.0 The history in the lower left (540 models with a compute estimate) and the training compute of most model squares.
- Epoch AI, AI Chip Sales CC BY 4.0 All the AI chips sold so far, running for one year.
- Epoch AI, Frontier Data Centers CC BY 4.0 Colossus 2, and every data centre Epoch tracks, today and at end-2028 capacity.
- Epoch AI, Machine Learning Hardware CC BY 4.0 The speed of one H100 chip, the unit behind every chip square.
- Villalobos et al. 2024, Will we run out of data? (Epoch AI) All the public text humans have written.
- Wei et al. 2022, Emergent Abilities of Large Language Models The band from 1022 to 1024 operations where most of the abilities they reported appeared.
- Schaeffer et al. 2023, Are Emergent Abilities of Large Language Models a Mirage? The case that many of those jumps come from how answers are scored.
- C. E. Shannon 1948, A Mathematical Theory of Communication, OpenAI 2019 (GPT-2) and Brown et al. 2020 (GPT-3) The quoted text that assembles out of noise on the data side, as short quotations.
- Lucide icons ISC The small glyphs beside every name.
- Turbo colour map (A. Mikhailov, Google, 2019) Apache 2.0 The colour of the ground and of the scale on the right, cool at one byte, warm at the top.
Each card links the source of its own number. Epoch AI’s data is credited as “Epoch AI, (dataset), epoch.ai”, retrieved 20 October 2026.