CPU vs GPU vs TPU vs LPU vs NPU: The AI Hardware Guide for Everyone

Every AI system — from the model running in your phone to the data center powering ChatGPT — runs on one of five types of computer chips. Each chip type represents a different answer to the same question: do you want to be able to do anything, or do you want to do some things extremely well?

That tradeoff is the key to understanding all of AI hardware. This guide breaks it down in plain English, then goes deep enough for the people who want numbers and architecture details.

The core idea: a spectrum from general to specialized#

Imagine a spectrum:

On one end is the CPU — the processor in every laptop and server. It can do anything: run a website, calculate a spreadsheet, encode a video, or run a small AI model. It is the most flexible chip ever built, and also the slowest for parallel math tasks.

Moving right along the spectrum you find the GPU — originally built to make video games look good, now the workhorse of AI. It can run thousands of calculations at the same time, which makes it excellent at the kind of math AI models need. It is less flexible than a CPU but vastly faster for AI work.

Further right is the TPU — Google's custom AI chip, built for one purpose: the specific matrix math that makes neural networks work. It is even faster and more power-efficient than a GPU for that narrow task, but it only runs on Google Cloud and works best with Google's own software.

Near the far end is the NPU — a tiny chip designed to run AI locally on phones and laptops. It uses very little power and keeps your data on your device, but it can only run small AI models.

At the far end is the LPU — the most specialized chip in this list, built by Groq specifically to run large language models as fast as possible. It cannot train AI models and does not work for images or video, but for real-time text-based AI, it is the fastest thing available.

Each step rightward trades flexibility for speed in a narrower domain. That is the fundamental tradeoff driving all of AI hardware.

CPU: the versatile processor#

What it is in plain terms: The CPU is the "brain" of any computer — it can run any program, any instruction, in any order. Think of it as a highly skilled craftsperson who can switch between tasks but only does one thing at a time.

What it does for AI: CPUs are not replaced by AI accelerators — they work alongside them. Every GPU, TPU, and LPU still needs a CPU to tell it what to do: load the model weights, schedule the next job, handle the API request, return the result. The CPU is the foreman of the factory. It never touches the assembly line directly, but nothing moves without it.

For AI specifically, CPUs handle data preprocessing (converting raw text, images, or audio into the format a model needs), orchestration, and serving very small models (under 100 million parameters).

Workload	CPU verdict
Training a large AI model	Too slow — not the right tool
Preparing data for training	Yes — this is where CPUs shine
Running small AI models	Yes — fine for models under ~100M params
Coordinating AI systems	Essential — every system needs a CPU

GPU: the parallel compute engine#

What it is in plain terms: A GPU has thousands of small processors that work in parallel — like a factory floor with thousands of workers all doing the same simple task at once. Where a CPU has a handful of very fast, very smart workers who can switch between tasks, a GPU has many slower workers who are brilliant at one specific thing: multiplying large matrices of numbers at the same time. That matrix multiplication is the core operation in every neural network.

GPUs were originally built for video games, not AI. The AI community adopted them because the same property that makes a GPU good at rendering millions of pixels (thousands of parallel calculations) makes it excellent at the parallel math AI models require. Every major AI model — GPT, Claude, Stable Diffusion, Gemini — was trained on GPUs, almost exclusively NVIDIA GPUs.

Why NVIDIA dominates: NVIDIA built CUDA, a programming platform that gives developers direct access to the GPU's parallel compute hardware. Twenty years of AI research, libraries, and tooling have been built on CUDA. That ecosystem is the real moat — not the chip design itself. AMD GPUs are technically competitive but lack the software ecosystem.

Workload	GPU verdict
Training large AI models from scratch	Best available option
Image, video, and audio generation	Best available option
General LLM inference	Good — LPU is faster for real-time
Real-time multi-agent AI	Good — LPU is 10-50x faster for latency-critical calls

The market is shifting. NVIDIA held roughly 86% of the AI accelerator market in 2025. By 2026 that had dropped to around 75%, as specialized chips began eating into the inference market where GPUs were previously the only option.

TPU: Google's AI-specific chip#

What it is in plain terms: A TPU is a chip that Google designed from scratch to do one thing extremely well — the specific type of matrix multiplication that underpins neural networks. It is not trying to be a general computer. It is a purpose-built AI accelerator, and it achieves that focus through a design called a systolic array.

What is a systolic array? Imagine a grid of tiny calculators, each one multiplying a number it receives from its neighbor and passing the result to the next neighbor — no detours to memory, no waiting. Because the data flows in a disciplined, predictable pattern, the chip can be extremely power-efficient. Google's TPUs use a 128x128 grid of these units, giving 16,384 multiply-accumulate (MAC) units all working simultaneously. MAC is just the basic operation in neural networks: multiply two numbers, add the result to an accumulator. It is the atomic unit of AI computation.

The other advantage Google has is vertical integration: Google designs the TPU, builds the compiler (XLA), writes the framework (JAX), runs the runtime (Pathways), and owns the interconnect (ICI) that links thousands of TPUs together in a TPU Pod. When all the software and hardware are controlled by one team, you can optimize at every layer simultaneously. That is why training large models on TPU Pods can be significantly cheaper than equivalent GPU clusters on other clouds.

The tradeoff: TPUs only run on Google Cloud, and they work best when you write your models in JAX (Google's framework). If you want to run PyTorch models or work outside Google Cloud, GPUs remain the better choice.

Workload	TPU verdict
Training large models on Google Cloud	Excellent — often 2x cheaper than equivalent GPU at scale
Running inference on Google Cloud	Excellent
Running anywhere other than Google Cloud	Not an option
Models with dynamic or irregular computation graphs	Struggles — prefers static, regular computation

LPU: the real-time inference chip#

What it is in plain terms: The LPU (Language Processing Unit) was built by Groq with one obsession: making large language models respond as fast as possible. Not throughput (tokens per second at batch scale), but latency — the time between hitting enter and seeing the first word appear. For real-time conversation, this is what matters.

The LPU is the most architecturally unusual chip in this list. Most chips store data in memory (RAM) that is separate from the compute units, and every calculation requires fetching that data from memory. Moving data between memory and compute is slow and energy-intensive. The LPU takes a different approach: it puts SRAM (the fastest memory available) directly on the chip, giving it 80 TB/s of memory bandwidth versus a GPU's roughly 8 TB/s. For LLM inference, this is the critical metric — not raw compute speed.

LPUs are inference-only. They cannot train AI models. The math required to train (backpropagation and gradient computation) requires a different hardware design, and LPUs are not built for it.

Groq was acquired by a major chip company in 2025 — verify this from current public sources before citing in external publications.

Why LPUs are faster for LLMs: LLM inference is memory-bandwidth-bound, not compute-bound. Generating each token requires reading the entire model's weights from memory. A GPU takes roughly 17ms to do this per token on a large model; an LPU does it in roughly 2ms. That is why an LPU running a 70B parameter model can produce 300-800 tokens per second — fast enough for real-time conversation — while a GPU doing the same task produces 30-80 tokens per second.

Workload	LPU verdict
Real-time LLM inference (conversational)	Best available — 10x more efficient than GPU
Multi-agent AI (sequential calls)	Best — 5 LLM calls take the time of 1 GPU call
Training AI models	Not supported — inference only
Image, video, or audio AI	Not the right tool

NPU: AI in your pocket#

What it is in plain terms: An NPU is a small, low-power chip built directly into your phone or laptop's main processor (the SoC). Its job is to run AI tasks locally — no internet connection, no cloud server, no data leaving your device. Apple Neural Engine, Qualcomm Hexagon, and Google Tensor all contain NPUs.

What does "TOPS" mean? TOPS stands for Trillion Operations Per Second. It measures how many AI calculations an NPU can perform in one second. The Apple Neural Engine in recent iPhones delivers 35+ TOPS. Qualcomm's Hexagon NPU hits 45+ TOPS. Google Tensor chips manage around 20+ TOPS. For context, a high-end GPU can hit thousands of TOPS, but it also consumes 300W versus an NPU's 4-10W.

The NPU is not trying to compete with GPUs. It is designed for a different world: running small AI models (up to about 3 billion parameters) on your device, at 40-60x better energy efficiency than a GPU, with complete privacy because your data never leaves your phone.

Workload	NPU verdict
On-device AI on a phone or laptop	Best — private, low-power
Small language models (up to 3B)	Excellent efficiency
Mobile camera AI (hdr, portrait mode)	Best
Training large models	No
Large inference workloads	No — not designed for this

The decision guide#

If you need to…	Use this chip
Train a large AI model from scratch	GPU (H100 or B200)
Run real-time AI chat or conversation	LPU (Groq)
Process large batches of LLM queries	GPU or TPU
Run AI on your phone or laptop	NPU (built in)
Train or run AI on Google Cloud at scale	TPU (v5p / Ironwood)
Build a system with multiple AI agents	LPU + GPU hybrid
Preprocess data before training	CPU (it is already there)
Maximum flexibility across all AI tasks	GPU
Maximum efficiency for large language models	LPU
Maximum scale with unified compute	TPU Pod (thousands of chips)

The future: no single chip wins#

The interesting development in 2026 is that the best AI systems are not choosing one chip — they are orchestrating multiple chips together, each doing what it does best.

A modern AI system might use a GPU to handle the initial processing (called prefill, where the model reads your prompt), an LPU to generate the response token by token (called decode, where speed per token matters most), a CPU to coordinate everything, and an NPU on the end user's device for privacy-preserving local processing. Together, this heterogeneous approach delivers 35x higher throughput per megawatt than running everything on a GPU.

The winner in AI hardware is not a chip. It is the orchestration layer that routes each part of the job to the right processor — and the engineers who understand how to build that routing.

---

If you found this useful, share it with someone building or evaluating AI systems. For more on AI infrastructure and inference optimization, explore the other posts at blog.sdad.pro.