Inside the GPU

The whole assembly

Before we go any further, here is the whole card opened up. Take a moment to look it over and get familiar: drag to rotate it, and tap any name below to light that part up in the diagram.

Every part on this board has a job. Here is the whole cast, tap any name to light it up in the diagram:

• The GPU, the GA102 chip at the centre, where all the real computation happens.
• The memory, the ring of GDDR6X chips that holds the model and its working data.
• The PCIe slot along the bottom edge, which plugs into your computer; the model's weights get loaded onto the card through here.
• The display sockets, which drive your monitors over HDMI and DisplayPort, nothing to do with AI.
• The power connector, feeding the card up to 350 watts.
• The VRM, which conditions that power into the clean, exact voltage the chip needs.
• The cooler, the fans and heatsink that carry away the heat.

Two of these do the real work of running a model: the memory that holds it, and the GPU that computes it. Let's start with the memory.

Now let's focus on the GPU, where the main computation actually happens.

GPU

Now that we are inside the GPU chip, the first thing you notice is that it is split into seven big blocks. These are the Graphics Processing Clusters, or GPCs.

For a language model, these seven clusters are the first way the chip splits up the work. One of Qwen's giant matrix multiplications is far too big for any single unit to do alone. So the chip cuts it into seven pieces and hands one to each cluster, and all seven work at the same time. And the splitting does not stop there.

Inside every one of those clusters sit the real workers, called Streaming Multiprocessors, or SMs. The 3090 has 82 of them switched on. The full chip is built with 84, but a slab of silicon this big almost always comes out with a few faulty ones, so Nvidia switches those off and sells the rest. That sorting is called binning, and it is why the 3090 ships with 82 and not 84.

Each worker has its own small memory, its own schedulers, and three different kinds of cores. A language model uses those three kinds very differently, and that is what the next three stops are about.

CUDA cores

Every worker holds 128 small units called CUDA cores. That comes to 10,496 in the whole chip. Each one is simple. It just does basic math. The power comes from having so many of them running together.

In a language model these handle all the work that is not a big matrix multiply: turning raw scores into probabilities (softmax), keeping numbers in a sane range (normalization), the small non-linear steps between layers (activation functions), and finally picking the next token Qwen writes. None of it is heavy, but there is a lot of it, and these thousands of little cores mop it all up.

Tensor cores

This is the important one. Each worker has 4 Tensor cores, 328 in the whole chip, and they do just one job very fast. They multiply matrices.

A language model is almost entirely matrix multiplication, and this is where it happens. More than 90 percent of the math in a model runs right here.

The tensor cores work hardest at one particular moment: when the model first reads in your prompt, a phase called prefill. Since you have already typed the whole prompt, all of its words are available at once, so the model runs them through the tensor cores together, in parallel, instead of one after another. With so much work happening at the same time, the cores stay completely full, and the only limit on speed is how fast they can multiply. Hold on to the word prefill: a few sections from now we will meet its exact opposite.

Ray Tracing cores

Each worker also has one Ray Tracing core. This is special hardware for games. It works out how light rays bounce off surfaces to make realistic shadows and reflections.

SRAM

We just watched the Tensor cores tear through matrix multiplication. But a core is only as fast as the data reaching it, and if every number had to come all the way from the 24 GB of main memory (the one we saw in the Memory section, made up of 24 GDDR6X chips), those blazing cores would spend almost all their time waiting. Main memory is simply too far and too slow to keep them fed.

So the chip keeps whatever the cores are using right now much closer, in small pools of very fast memory built onto the die itself. This on-chip memory is SRAM, and it comes in levels, each one smaller but closer and faster than the last. The 6 MB shared in the middle of the chip is a shelf every worker can reach; each worker also has its own 128 KB drawer right beside its cores; and closest of all, the registers, a few numbers each core can grab instantly. The nearer the data, the less the core waits, so the whole game is keeping what you need right now as close to the core as you can.

FlashAttention is the classic example. The attention step would normally build one giant table of numbers, far too big to fit on those close, fast shelves, so it would spill out to the slow warehouse. Instead, FlashAttention works on the table in small chunks that do fit, finishes each chunk, and drops it, never building the whole table. With far less data travelling to and from slow memory, it runs much faster, and the cores stay fed.

Why answers come token by token

Now the twist. Reading your prompt was fast because every word went through at once. Writing the answer is the opposite: the model produces it one word at a time, and it cannot begin the next word until the current one is done. This phase is called decode.

Here is the catch. To produce even a single word, the model has to run that word through its entire stack of layers, using every one of its billions of weights once. So for each word it writes, the chip reads the whole model, all 16 GB of Qwen3's weights, out of memory, plus the growing KV cache. Not part of it. All of it. For every single word.

And that data can only leave memory so fast. It all travels through the twelve memory controllers on the edge of the die, the chip's link to the GDDR6X memory (the 384-bit bus, split into twelve 32-bit channels). Together they move about 936 GB/s, and no faster. Pushing 16 GB through that pipe takes roughly 17 milliseconds, which caps the model at around 55 words a second, no matter how many cores sit idle waiting. Here the memory is the bottleneck, not the math.

This explains a lot. A model twice as big has twice as much to read per word, so it types about half as fast. As the conversation grows, the KV cache grows with it and must be read too, slowing things further. And it is why quantization helps: that 16 GB is already the model squeezed to about 4 bits per number, and squeezing further means fewer bytes to move per word, so decoding speeds back up.