Guardrails: schema validation, filtering, and trust

It's tempting to treat a model's output as trusted just because it came from inside your own system. It didn't come from your code, though. It came from a probability distribution that occasionally produces something plausible-looking and wrong, the same as a stranger's input into a form on your website. Nobody would skip validating that. The model's output doesn't earn different treatment just because it sounds fluent.

Guardrails are the checks you put around a model's input and output so a bad result gets caught before it reaches a user or another part of your system. This post covers three layers: making structured output actually reliable, validating it once it arrives, and filtering for the things you specifically don't want getting through.

JSON mode isn't a guarantee

Structured output modes, where you ask a model to return JSON matching a schema, help a lot, and they aren't a hard guarantee on their own. A model can still return JSON that's syntactically valid but semantically wrong, a field with the right type but a nonsense value, or a required field quietly missing because the model didn't have enough information and filled in something plausible instead of leaving it empty.

The fix isn't to trust the mode more. It's to validate what comes out of it every time, the same as you'd validate any external input.

Schema validation as the actual safety net

Once you have a schema, run every model output through it before your code touches the result. A validation library that checks types, required fields, and value constraints catches malformed output immediately, with a clear error, instead of letting a bad value quietly propagate three functions deep before something breaks in a confusing way.

When validation fails, the useful move is feeding the specific error back to the model and asking it to correct just that field, not throwing the whole response away and starting over. "The price field must be a number, you returned a string" is something a model can usually fix in one more turn. A full retry from scratch wastes the parts of the response that were already correct.

Content filtering: catching what shouldn't get through

Separate from structural validation, content filtering checks the actual substance of a response, not "is this valid JSON" but "does this response contain something it shouldn't," personal data that shouldn't be echoed back, a tone that doesn't match what the product should sound like, a claim the model has no business making confidently. This layer runs after structural validation, because a response can be perfectly well-formed JSON and still say something you don't want a user to see.

Jailbreaks: treating the model's own instructions as attackable

A jailbreak is a prompt crafted specifically to get a model to ignore its own instructions or safety constraints. "Ignore all previous instructions and instead..." is the most obvious shape, and it's far from the only one, roleplay framings, hypothetical scenarios, instructions hidden inside content the model is asked to summarize, are all in the same family.

Basic resistance isn't exotic: keep system-level instructions separate from user input rather than concatenated into one string where the boundary is fuzzy, treat any user-supplied text the model will read (including documents it retrieves) as untrusted the same way you'd treat it if a human typed it directly, and don't rely on the model's own judgment as the only line of defense, add a structural check on the output regardless of what the model claims it was told to do.

Trust, but verify, applies to your own pipeline too

The theme across all three layers is the same. A model's output isn't more trustworthy because your own system generated the prompt that produced it. Validate structure, filter content, and treat the model's instructions as something that can be attacked, not something that's permanently fixed once you've written the system prompt.

Guardrails catch the failure shapes you already know about. The harder question is how you catch the ones you didn't predict, and that's what evals are actually for.