Where AI actually runs: cloud, local, edge

Every time you "use AI", a model file gets loaded into memory on some specific machine. That machine is in one of three places. A datacenter you reach over the internet, your own laptop or workstation, or the device the model ships inside of. Cloud, local, edge. Every product you've ever touched picks one of these, sometimes two, and the pick is almost never an accident.

This is post 1 of 7 in the Running series. I'm starting here on purpose. The rest of it (which models, what hardware, which runtime, what it costs) only makes sense once you know which of the three boxes you're standing in.

Cloud: someone else's GPU

Call the OpenAI API, open Claude in a browser, fire a message at a Gemini app. In every case the real model, say Claude Opus 4.7 at around 2 trillion parameters of weights, is sitting on H100 or B200 GPUs in a datacenter. Your prompt goes over the wire, the GPU runs the forward passes, the tokens stream back to you.

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-opus-4-7",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Three numbers tell you most of what you need to know about cloud inference:

• Latency: 200 to 1500ms time-to-first-token, depending on prompt size and model. Streaming hides this, since you watch tokens appear as they generate, but the round trip is still real.
• Cost: priced per million tokens. Frontier models in 2026 land around $3 to $15/Mtok input and $15 to $75/Mtok output. Turn on prompt caching and that drops 5 to 10x for repeated context.
• Capability: this is where the frontier lives. That 2T-parameter model topping every benchmark does not run on your laptop. It can't. Frontier models need 8 to 32 H100s just to hold the weights.

Here's the deal you're signing. Your data leaves the building, you ride a vendor's uptime, and they own the pricing. In exchange you write zero infra, you get the best model on the planet, and the thing stops existing the second your AWS bill stops.

For most product features, that's the right trade. Capability beats data sovereignty most of the time. It's the wrong trade when you're handling PII you legally can't ship offshore, or when you're doing 10M calls a day and the bill starts eating the P&L alive.

Local: a model file on your disk

Local means the model weights sit on your hard drive and the runtime loads them into RAM (CPU) or VRAM (GPU). When I say "I run llama3.1:8b on my M3 Max", what I mean is a 4.7GB GGUF file on the SSD, mmapped into the M3's unified memory, with Metal kernels doing the matrix multiplies.

brew install ollama

ollama run llama3.1:8b "explain TCP slow start in two sentences"

That's the whole on-ramp. The model downloads once (~5GB), and after that every call runs local. No network. No API key. No rate limits, no per-token bill.

The numbers look different here:

• Latency: 50 to 500ms time-to-first-token on consumer hardware. Throughput is hardware-bound. On an M3 Max with 64GB unified memory an 8B model does about 60 to 90 tok/s, while a 70B Q4 quant does 10 to 15 tok/s. An RTX 4090 is faster on the small models and slower on the big ones, because the 24GB VRAM ceiling becomes the bottleneck.
• Cost: $0 marginal per call. Capex is the laptop or workstation. Opex is electricity, and a 4090 pulls 450W under load.
• Capability: the ceiling is your RAM/VRAM. Consumer hardware tops out at 70B-class quantized models. The frontier (1T+) is out of reach without a server.

What you give up is a generation or two of model quality. Llama 3.1 70B is genuinely good at a lot of things, but it is not Claude Opus. Tool use, agentic work, long-context reasoning, all of that still leans toward the frontier models.

What you get back is privacy, predictable latency, no rate limits, and a model that still works on a flight with no wifi. There's a quieter payoff too. You start to understand what an LLM actually is. Once you've watched 4.7GB of weights mmap into memory, it gets hard to keep treating these things as magic.

My own setup splits the work. Local model for drafting commit messages, fast text reformatting, anything I run hundreds of times a day where the round trip is the thing that hurts. Cloud model for real reasoning, code generation, anything where the output has to be at the frontier.

Edge: the model is the device

Edge AI means the model is baked into the device's silicon. Your phone's NPU, the chip in a Ring doorbell, an AC controller, an industrial sensor. No internet in the loop at all. The model wakes up, runs inference on local sensor data, takes an action, and goes back to sleep.

You've used edge AI today and probably didn't notice:

• Face ID: the model deciding "this is you" runs on the iPhone's Neural Engine in around 100ms. Apple never sees your face.
• Live Captions on Pixel: Whisper-class speech-to-text, running right on the device.
• Smart compose in Gboard: token suggestions from a tiny language model that ships inside the keyboard.
• Garage cameras doing person and vehicle detection with no cloud subscription.

Three constraints define edge:

• Model size: usually under 500M parameters, often a quantized vision or audio model under 100MB. Nothing here is a frontier LLM.
• Latency: 5 to 50ms. That's the whole point. There's no network in the loop.
• Power: it has to run on a battery or a small SOC. NPUs (Apple's Neural Engine, Qualcomm Hexagon, Google Tensor) tune for tokens-per-watt, not raw throughput.

You don't deploy edge AI the way you deploy cloud or local. You ship it inside the firmware. The toolchain is Core ML on Apple, NNAPI/TFLite on Android, ONNX Runtime if you want cross-platform. All three take a trained model and compile it down to run on the target NPU.

This is the most invisible of the three. Every "smart" device you own is doing some flavor of edge inference, and the promise the user actually feels is just "works fast, works offline".

Choosing between them

There's no universal answer. There is a useful decision sequence, and you run it top-down:

1. Does the data legally have to stay on-device? Health PII, biometrics, certain financial data under specific rules. If yes: edge or local. Decision's over.
2. Is the latency budget under 100ms? Real-time speech, AR overlays, sensor loops. Edge or local. Cloud round-trips can't hit it consistently.
3. Do you need frontier capability? Multi-step reasoning, long-context retrieval, tool-using agents. In 2026 cloud is the only honest answer. Local 70B is good, not frontier.
4. What's the call volume? At 10M+ calls a day the cloud bill starts to dominate. Local or self-hosted turns into a real option, same model family on your own hardware.
5. Is offline a hard requirement? Field tools, remote sites, planes. Local or edge.

Most production systems end up hybrid anyway. A voice assistant does wakeword on-device (edge), ships the transcribed audio to a cloud LLM (cloud), then renders the answer locally with on-device TTS (edge). A coding agent runs quick edits against a local model and escalates to a cloud model when the reasoning gets hard. Forcing one box on the whole product is usually the wrong move.

What's next in this series

The next six posts walk through the running part:

• which models are at the frontier in 2026, and what their context windows actually buy you
• what it takes to run a model on your machine (VRAM math, quantization, the real hardware tiers)
• why Apple Silicon punches above its weight on local LLMs
• the runtimes (llama.cpp, Ollama, LM Studio), and when to pick each
• the economics of cloud tokens, and how prompt caching changes the math
• what actually leaves your machine when you use AI

The pattern under all of them is the same. The model sits somewhere. The somewhere decides almost everything else.