The news says "HBM". What is it, and why does AI need special memory?

The "HBM" in the news stands for high bandwidth memory: several layers of DRAM memory chips stacked into one pile, connected top to bottom by fine holes through the silicon, and placed right up against the processor, so a lot of data can move at once. When the news says AI chips are "short of HBM", this is what they're short of. To see why it matters, start with where AI chips get stuck.

What does memory do?

In a computer or an AI server, the processor does the computing and the memory holds the data in use right now. Think of a kitchen: the processor is the cook, the memory is the fridge next to them, and food has to come out of the fridge before it can go in the pan.

The memory that holds this data is mostly a kind of chip called DRAM. Each 0 or 1 is kept in a tiny capacitor, a part that can hold a little electric charge; whether it is charged or not is the 0 or the 1. The charge slowly leaks away, so every few dozen milliseconds all of it has to be topped up again; cut the power and the data is gone. For other ways memory and processors differ, read How is memory different from a processor?.

Why do AI chips get stuck moving data?

At its core, AI computing is a huge pile of numbers being multiplied and added. There are far too many of these numbers to fit on the processor itself, so they sit in the memory next to it and are brought over when needed. When a chatbot answers a question, for example, every small piece of text it puts out means reading a large amount of the model's numbers out of memory and using them once.

The processor computes very fast, but if the data doesn't arrive fast enough, it can only stop and wait. Picture a cook who chops at lightning speed, with a helper who can only bring a little food from the fridge at a time: the cook spends most of the time waiting.

And the gap keeps growing. A university study looked back over 20 years of server hardware: computing speed grew about 60,000 times, while the speed at which memory delivers data grew only about 100 times. Some people call this gap the "memory wall".

Moving data also costs a lot of energy. Researchers estimate that reading data once from memory outside the chip can take several hundred times the energy of one simple addition. So the memory an AI chip needs doesn't just have to hold a lot; it has to deliver a lot of data every second. That's called "bandwidth", and it's the "high bandwidth" in HBM's name.

How is HBM different from ordinary DRAM?

HBM is still DRAM inside, and it stores data the same way. The difference is in how it's arranged and connected.

The memory in an ordinary computer is a few DRAM chips, each in its own package, lined up on a long, narrow circuit board (a memory module) that plugs into the motherboard, several centimeters from the processor. One memory module sends 64 zeros and ones side by side at a time.

HBM does three things differently:

1. It stacks up: several layers of DRAM chips form one stack; the public standard for the latest generation, HBM4, covers 4, 8, 12 and 16 layers. At the bottom there's often a base chip that talks to the processor and also helps with testing.
2. It connects through the silicon: very fine holes are made in each layer and filled with metal, so signals pass straight through the chips from layer to layer. These holes are called through-silicon vias (TSVs). The layers are soldered together with tiny bumps.
3. It sits right next to the processor: the whole stack goes next to the processor, and both usually sit on a thin slice of silicon called an interposer. The wiring on an interposer is far finer and denser than on a circuit board. This side-by-side layout is covered in the article on "advanced packaging" in the news.

The result: one HBM stack has 1,024 data lines and sends 1,024 zeros and ones side by side, 16 times as many as one memory module; the newest generation doubles that again. Each line isn't necessarily any faster; HBM is fast because it has so many lines. Same speed limit, but the road goes from 64 lanes to 1,024. An AI chip often has several stacks around it, which makes the total wider still.

Why stack it, and why so close? Connecting over a thousand lines to the processor takes wiring finer and denser than an ordinary circuit board can carry, so an interposer usually does the job; and there is only so much room next to the processor, so to fit in more memory, it goes up. The path gets shorter, too, and each 0 or 1 takes less energy to send.

Why is HBM so expensive and hard to make?

• Every layer has to be good: if a single layer in a stack is bad, or one connection doesn't hold and can't be repaired, the whole stack is unusable. Try some made-up numbers: if each layer is good 95% of the time, a stack of 8 has only about a two-in-three chance of being all good; a stack of 12, just over half. It's the same multiplying shown in the article on yield in chip news. So every layer is tested before it's stacked (wafer sort), and only good ones go on.
• Every layer has to be ground very thin: the more layers, the thinner each one must be, so the stack doesn't get too tall. The wafer is ground from the back until only a few tens of micrometers are left (a micrometer is a thousandth of a millimeter), about as thick as a human hair. Silicon this thin bends and cracks easily, which makes the holes, the alignment and the soldering all harder.
• Heat can't get out: the layers in the middle and at the bottom are covered by other chips, so heat has a hard time escaping from them; and right next door is a very hot processor. When DRAM gets hot, its charge leaks faster and has to be topped up more often; too hot, and it goes beyond the range where it works properly.
• Many steps, costly materials: making the holes, thinning, stacking layer by layer, soldering and testing again add many steps compared with ordinary memory, and some use tools and methods much like a wafer fab's. The interposer underneath is also made with chip-making methods, which cost far more than making a circuit board.

Next time you see it in the news

• "AI chips are short of HBM", "HBM demand outstrips supply": not enough of this stacked memory is being made. An AI chip needs several stacks next to it; without enough memory, a finished processor still can't be put together into a complete chip.
• "12-layer" or "16-layer HBM": how many DRAM chips are stacked in one pile. More layers hold more data per stack, and are harder to make.
• "HBM3", "HBM4": which generation of HBM. Its specifications are a public standard the industry writes together, and the first version came out in 2013. The HBM4 standard doubles the data lines per stack from the previous generation's 1,024 to 2,048, so it can move more data every second.
• "The memory wall": the gap between processors that compute faster and faster and memory that can't deliver data fast enough to keep up.

In fab24hr

There is no HBM in the game, and no stacked chips. The game's chip, Glowmere, is a fictional hearing-aid chip that goes into a small SiP module at a test-and-assembly house; the module holds just one chip, connected by wire bonding.

The closest thing is the assembly line at the test-and-assembly house: in "A day in the fab", tap "Test-and-assembly house" on the "Your role today" card and swipe to the assembly process engineer, who looks after dicing, die attach, wire bonding and molding. The note at the dicing saw says the wafer is first ground thin from the back, then cut apart. The wafer test area shows "test first, keep only the good ones" too: before the wafer is cut, every chip is tested, and only the good ones go on to the next station.

The companies, chips and people in the game are fictional; what HBM is and why it's hard to make come from public, general sources, and the "each layer is good 95% of the time" figure is made up. For other jargon, see Chip fab jargon in plain English.