The news says "AI chips". What are they, and how are they different from ordinary processors?

The "AI chip" in the news is a catch-all name for chips built to run AI. What they do best is a huge number of very simple multiplications and additions at the same time; an ordinary processor (CPU) is much slower at this. The graphics processors (GPUs) in data centers, accelerators designed just for AI, and the small AI accelerator inside a phone (often called an NPU) can all be called AI chips in the news. Some people also say "AI semiconductors". To see how they differ, start with what AI actually computes.

What does AI actually compute?

An AI model holds a huge pile of numbers. When you use it, the work is a lot like ringing up a bill at a checkout: multiply the quantity of each item by its price, then add everything up.

Take recognizing a photo. To a computer, a photo is a long string of numbers that record the brightness and color of each dot. The model multiplies each dot's number by one of its own numbers and adds it all up to get a score; with another set of numbers, it gets another score. The next layer treats those scores as new "quantities" and multiplies and adds again. Layer after layer, the final score says how likely it is that the photo shows a cat.

This kind of computing has three features:

• Each step is simple: one multiplication, one addition.
• There are a lot of steps: a big model holds billions of numbers or more. Each time a chatbot puts out a small piece of text, a great many of those numbers are each used for one multiplication and one addition.
• Most steps don't depend on each other: one bill doesn't have to wait for another to be finished, so they can be added up at the same time.

So what AI needs isn't "one calculation done very fast" but "a great many calculations at once".

What is an ordinary processor (CPU) good at, and why is it slow at AI?

The CPU, or central processing unit, is the part of a computer or phone that carries out a program. It usually has a few to a few dozen "cores", each of which can work on its own, and each one is powerful: when a program reaches a fork in the road ("if this, then that"), it guesses the next step ahead of time, and it keeps a cache, a small memory close at hand, so the data it needs is within reach. It's like a few experienced accountants who can handle any kind of bookkeeping. It's best at work like opening an app or loading a web page, where the next step keeps changing.

But for every step, a CPU first reads an instruction, works out what to do, then goes and fetches the numbers. That preparation often uses more energy than the multiplication itself. Faced with billions of multiply-and-adds, a few dozen cores end up with a very long queue. So in AI systems the CPU mostly organizes the work and hands the multiplying and adding to something else.

How does a graphics processor (GPU) do "many at once"?

GPUs were first made for pictures on a screen. A common screen has over two million dots, the color of each dot is worked out separately with the same recipe, and the whole picture changes dozens of times a second. So a GPU is built the opposite way from a CPU: hundreds or thousands of simpler calculating units, where one instruction sets a big group of units going at the same time, each on its own numbers.

Picture a whole auditorium of people, each with a calculator, all pressing the keys on the same signal. AI's multiply-and-add is also "the same thing done to lots of numbers", so GPUs came to be used for AI on a large scale, and newer GPUs have added units made just for multiplying and adding.

So what is a dedicated AI accelerator?

If most of AI is multiply-and-add, you can give much more of the chip's area to multiply-and-add units. A common layout is a grid: numbers are fed in from the left and the top, one square at a time, each square does one multiplication and one addition, then passes the numbers on to the next square. It's like a line of people passing buckets: a number comes in once and gets used many times inside the grid. University research proposed this layout back in the late 1970s.

Another trick is to use fewer digits. Numbers inside a computer are strings of 0s and 1s, each one called a bit, and more bits means a more precise number. AI often gets by with 16 bits, 8 bits or even fewer. The size of a multiplier circuit grows roughly with the square of the number of bits: halve the bits and the circuit shrinks to about a quarter, so more of them fit in the same space, and they use less power.

The price is flexibility, since it's no good at other kinds of work: a GPU is a generalist, an accelerator a specialist.

Why do AI chips need so much memory?

However many multiply-and-add units there are, they're useless if the numbers can't get to them. A model has far too many numbers to fit on the chip, so most of them sit in memory next to it and are brought over when needed, and moving data often takes more time and energy than the computing does. That's why AI chips often have several stacks of special memory right next to them, called HBM (The news says "HBM". What is it, and why does AI need special memory?). For how processors and memory share the work, see How is memory different from a processor?.

Why use the most advanced process and packaging?

Multiply-and-add units are built from transistors, the tiny switches controlled by electricity (What is a transistor?). A newer process generation fits more transistors in the same space, and each switch uses less power (The news says "2 nm", What is Moore's law?). AI chips need room for a great many multiply-and-add units and use a lot of power, so they need the newest generation on both counts.

But a chip can't grow without limit. Some big AI chips are already close to the largest area a scanner can print in one go, and the bigger the chip, the lower the yield. To add more computing and more memory, several chips are placed side by side or stacked and put in one package. That's advanced packaging (The news says "advanced packaging").

How are chips for "training" different from chips for "inference"?

Training is teaching the model. At first, the numbers in the model are random. You give it an example, let it work out an answer, compare that with the right answer, then work backward to find which way each number should be nudged, and repeat this a vast number of times. Each example takes about three times the computing of a single pass, and next to every number the chip also has to keep track of how to adjust it. So training often needs many big chips linked together, running for days or weeks, and the numbers have to be more precise.

Inference is using the trained model to answer questions. The numbers are fixed, and the model only has to run through once. Each run takes much less computing, but it happens every time anyone uses the model, and it has to be quick. Often rougher numbers, such as 8 bits, are good enough. So inference chips care more about saving power and answering fast. Many chips can do both; the difference is which one the design favors.

Is there an AI chip in my phone?

Yes, though it's usually a small part of the phone's main chip rather than a separate chip. A phone's main chip puts the CPU, the GPU and many other parts on one chip (What is an SoC?), and the part made for AI multiply-and-add is often called an NPU (neural processing unit). It does inference, for example sorting your photos, recognizing your face or turning speech into text. A phone can only use a few watts for long stretches, so this part is built to save power above all; models that are too big are still sent to a data center.

Next time you see it in the news, here is how to read it

• "AI chips are in short supply": not enough of the big chips used in data centers are being made. The holdup can also be the memory next to them or the packaging.
• "Training chips", "inference chips": the first are built for teaching models, the second for using a trained model to answer questions, with more attention to saving power.
• "A phone with a built-in NPU", "on-device AI": the phone's main chip has a part made for AI multiply-and-add, and some AI features run right on the phone.
• "Custom AI chips", "ASICs": chips designed for one particular job (ASIC stands for application-specific integrated circuit). They only do that job, and they use less power doing it than a general-purpose chip.

In fab24hr

There are no AI chips in the game, and no stacked packages. Glowmere, the chip in the main story The life of a chip, is a fictional hearing-aid chip: what matters most for it is saving power on standby, and it's made with a mature process generation.

But every chip goes through the same stages (What is a chip?). In the fab, the wafer is built up layer by layer: litho prints the pattern, etch cuts it in, and the line width is measured. At the test-and-assembly house, every chip goes through wafer sort first, after dicing only the good ones go on to packaging, and after packaging comes final test. Tap "Clock in", choose "A day in the fab", then on the "Your role today" card tap "Production fab" or "Test-and-assembly house" and swipe to the process engineer (then tap "Litho" or "Etch" under the name), the test engineer or the assembly process engineer to work those stations. Chapter 10, "Standby Current", and chapter 11, "The Ball Lifted", of the main story are about testing and packaging too.

The companies, chips and people in the game are fictional; the AI chip parts of this article come from public, general sources, and where there is no agreed number, it gives only the rough size. For other jargon, see Chip fab jargon in plain English.