What is the memory wall, and why can't data keep up with fast chips?

The memory wall means this: processors compute faster and faster, but memory can't deliver data fast enough to keep up, so the processor often has to stop halfway and wait. Moving data also costs a lot of energy: reading data once from memory outside the chip can take several hundred to over a thousand times the energy of one simple addition. Today AI chips hit this wall hardest, and the HBM, advanced packaging and silicon photonics you read about in the news are all tied to it.

What does the memory wall mean? Think of a kitchen

A cook works very fast, but the ingredients are kept in a warehouse far away and arrive one cart at a time. How long a dish takes depends on when the ingredients get there, so a faster cook doesn't help. What helps is keeping the ingredients you use most within reach, moving the warehouse closer, and using a bigger cart. That is exactly what the industry does about the memory wall.

The processor is the cook, doing the computing; memory is the warehouse, holding the data to compute on (for how the two split the work, see "How is memory different from a processor?").

Why do processors and memory keep drifting apart?

Because they improve at different speeds. From the mid-1980s to the early 2000s, processors got roughly 50% or more faster every year, while the wait to read one piece of data from memory shrank by less than 10% a year. The gap doubled about every two years. Memory chips care most about holding a lot and costing little per bit; speed comes second.

At the end of 1994, two researchers at an American university wrote a short paper, published the next year, called "Hitting the Memory Wall". They estimated that within ten years, processors would be held back by memory. Later, caches were split into more and more levels and memory learned to send more at once, which pushed the wall back a little, but it never went away.

With AI, the wall got higher again. The article The news says "HBM". What is it? mentions a university study: over 20 years, the computing speed of servers grew about 60,000 times, while the speed at which memory delivers data grew only about 100 times. The wall people talked about early on was mostly "how long one wait takes"; today it is more often "how much can be sent every second".

Why does moving data cost more energy than computing?

Inside a chip, a 0 or a 1 is a low or high voltage on a wire. Every switch between 0 and 1 means charging or draining the whole wire, and the longer the wire, the more charge it takes.

The circuits that compute sit close together, so their wires are short. To reach memory outside the chip, data has to leave the chip and pass through the package and the circuit board, a trip of several centimeters. That's why researchers estimate that reading memory outside the chip once can take several hundred to over a thousand times the energy of one simple addition. As switches shrink generation after generation, each computation takes less energy, but the length of that trip doesn't shrink. To save power, the first question is often how to move less.

Why is memory arranged in levels?

Because what's close is fast but small, and what's far is big but slow.

Level	Where	How much it holds	Time to fetch once
Registers	Next to the computing circuits	The few numbers being worked on	Under a nanosecond
Cache	On the processor chip itself	Roughly KB to MB	About 1 to 20 nanoseconds
Memory (DRAM)	Outside the chip	GB and up	Tens of nanoseconds

A nanosecond is a billionth of a second; KB, MB and GB each go up by about a thousand times. A processor works to a steady beat, billions of beats a second, so one wait for memory outside the chip means sitting idle for a hundred beats or more.

Levels work because programs have habits: what was just used is often needed again soon, and after one piece of data, the one next to it is often needed too. Copy what's used most into the closer level, and most of the time it's within reach, which also saves a lot of energy. Cache is made of SRAM, which is fast but needs six transistors to store each bit and takes up a lot of room, so there isn't much of it.

Why does AI hit the wall hardest today?

An AI model is a huge pile of numbers (see The news says "AI chips". What are they?), far too many for the cache, so they sit in memory outside the chip. Every time a chatbot produces a little more text, a large share of those numbers has to be read. When it's working for just one person, each number is brought in, used for about one multiplication and one addition, and then it's on to the next. Keeping it in the cache rarely pays off, so the computing circuits spend much of their time waiting.

So questions from several people are often grouped and computed together: bring a number in once, use it for several people. But in recent years, models have also grown much faster than the memory one processor can have next to it.

What can the industry do, and what does each road cost?

No single trick tears the wall down. The industry takes several roads at once, and each one has a price.

More cache on the chip

• What: make the cache on the processor chip bigger.
• Why: the numbers used most stay on the chip, which is fast and saves energy.
• Cost: SRAM takes up a lot of room, and in recent generations it has become harder and harder to shrink; a bigger chip costs more, and a large AI model still won't fit.

Stack the memory right next to the processor (HBM)

• What: stack several DRAM chips into one pile and place it next to the processor. One stack has 1,024 data lines (the newest generation doubles that); one memory module in an ordinary computer sends only 64 zeros and ones at a time.
• Why: more lines send more at once, and a shorter trip takes less energy for each 0 or 1.
• Cost: every layer has to be good and ground very thin, heat struggles to escape, and it's expensive and hard to make (see The news says "HBM". What is it?).

Use advanced packaging to bring chips close

• What: put a computing chip and its memory, or several computing chips, into one package, side by side on a thin slice with very fine wiring, or stacked directly.
• Why: compared with wires running several centimeters across a circuit board, the trips are short and there are many more lines, so data moves faster and uses less energy.
• Cost: packaging gets harder and more expensive; if one chip inside is bad, the whole set is wasted; stacked chips shed heat less easily (see The news says "advanced packaging". What is it?).

Use light for the longer trips

• What: for longer trips, such as between racks of servers, turn electricity into light and send it through optical fiber.
• Why: the faster data is sent, the shorter the distance it can travel on copper, and the more energy it takes to restore the signal; light loses very little over long distances.
• Cost: it needs extra parts to turn electricity into light and back; silicon itself is poor at giving off light, the fiber has to be lined up very precisely, and heat is a problem. Over very short distances, electricity is still used (see The news says "silicon photonics". What is it?).

Compute inside the memory (mostly still in research)

• What: build simple computing into the memory chip or right next to it; some research even uses the electrical behavior of the memory cells themselves to multiply and add.
• Why: the cheapest move is no move at all.
• Cost: memory is made to hold a lot and cost little, so it's hard to fit computing circuits in; the way programs are written and the design of the whole computer have to change too. The first four roads are already in use (for light, that means optical fiber between racks); this one is mostly still in research.

In fab24hr

There is no memory wall in the game, and no HBM, AI chips or memory fab. Glowmere, the chip in the main story, The life of a chip, is a fictional hearing-aid chip. At a test-and-assembly house it goes into a small SiP module and is connected to the board one metal wire at a time, each finer than a hair; this is called wire bonding. Those wires are Glowmere's roads to the outside. In chapter 11, "The Ball Lifted", two wires come in low on the wire bonder's pull test, and you are the assembly process engineer.

Glowmere cares about power too, but a different kind: the little bit that still leaks away when the chip is doing nothing (leakage; see "Why do chips use power?").

The companies, chips and people in the game are fictional; what the memory wall is, and how each road works and what it costs, come from public, general sources. For other jargon, see Chip fab jargon in plain English.