Streaming, throughput, and the KV cache
Why TTFT and tok/s are different numbers, why streaming feels faster than it is, and the KV cache that makes the 1000th token cost about the same as the first.
Blog
Local Llms
Posts on AI engineering, LLM systems, and software development.