← Blog
Latency
General/dictionary/latency
Definition
The time between sending a request and getting a response back. In LLM systems this includes both time-to-first-token and total generation time.
Posts that use this term
- Observability: what a production AI system actually logs
Closing the AI Engineering series. Tracing prompts and tool calls, tracking spend and latency, and catching failures that don't throw an error.
- Agent memory: scratchpad vs long-term store
An agent forgets everything the moment its context window ends, unless something is built to save it. Short-term scratchpad vs a real long-term store, and what's actually worth keeping.
- Prompting as an engineering discipline
A prompt that runs a thousand times a day needs to be treated like code, not a note you typed once. Few-shot, chain-of-thought, and structured output, with the trade-offs.