LLM structured output JSON schema is the foundation of any production pipeline that relies on machine‑readable responses. But even with a strict schema and a well‑tuned prompt, our daily content‑generation pipeline broke twice—once because a provider silently dropped support, and once because a widely‑recommended parsing trick ate our own data. Neither failure appeared in local testing. Both teach the same lesson: a structured output contract is not a reliability guarantee; you must still code for the real world.

Why LLM JSON Parsing Still Breaks in Production

The promise of llm structured output json schema is that you get a pristine JSON object that matches your definition. In practice, two things go wrong. First, not every model gateway supports json_schema with strict: true; some return HTTP 400. Second, many parsers blindly strip markdown code fences, a heuristic that corrupts any field that legitimately contains triple‑backtick code blocks—like the Markdown body of a generated article. We hit both in a pipeline that runs daily behind our AI‑powered comparison pages.

The Right Way to Request Structured Output

Start by setting response_format to { type: "json_schema", json_schema: { strict: true, schema: yourSchema } }. This eliminates prose drift and guarantees valid JSON shape. But assume nothing: read the model name at call time so a runtime environment override can swap models without a redeploy. If you hardcode the model at module load, your only escape during an outage is a full deploy.

// Always resolve the model at call time, not at module load
export async function getStructuredCompletion(
  messages: ChatMessage[],
  schema: Record<string, unknown>,
  model = process.env.STRUCTURED_MODEL || "gpt-4o",
): Promise<Record<string, unknown>> {
  const body = {
    model, // <-- runtime override friendly
    messages,
    response_format: {
      type: "json_schema",
      json_schema: {
        strict: true,
        schema,
      },
    },
  };
  // …
}

Failure Mode #1: Provider Capability Drift

Even when your API client sends a valid json_schema request, an upstream provider may reject it. Some gateways don’t support strict: true at all and return HTTP 400. Our client catches that exact status and retries with response_format: { type: "json_object" }. The model then outputs unconstrained JSON (possibly wrapped in prose), so we follow up with a tolerant parser. The key insight: strict json_schema is the right default, not a lowest‑common‑denominator compromise. Detect and degrade, never pre‑weaken your request.

try {
  const res = await fetch(this.baseUrl, {
    method: "POST",
    headers: this.headers,
    body: JSON.stringify({
      ...body,
      response_format: {
        type: "json_schema",
        json_schema: { strict: true, schema },
      },
    }),
    signal,
  });
  if (res.status === 400) {
    // Provider rejected strict schema; fall back to json_object
    console.warn("400 on strict json_schema; retrying with json_object");
    const fallbackRes = await fetch(this.baseUrl, {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify({
        ...body,
        response_format: { type: "json_object" },
      }),
      signal,
    });
    return this.handleJsonResponse(fallbackRes, schema);
  }
  return this.handleJsonResponse(res, schema);
} catch (e) {
  // …
}

Failure Mode #2: When Fence‑Stripping Parsers Mangle Content

The usual advice for parsing LLM JSON is to strip triple‑backtick fences before calling JSON.parse. That heuristic is actively dangerous when any field in your schema contains fenced code blocks. Our blog generation pipeline produces articles whose markdown field routinely includes code blocks. A fence‑stripper that greedily removes everything between backticks will corrupt the payload. We learned this the hard way when a generated article lost half its code examples.

Our parser does the opposite: it tries JSON.parse(raw) first, because native structured output should never produce fences. Only if that initial parse throws do we extract the substring from the first { to the last }. We never remove backticks.

function tolerantParseJson(raw: string): Record<string, unknown> {
  // Try direct parse first (works for native structured output)
  try { return JSON.parse(raw); } catch { /* fall through */ }

  // Fallback: grab everything between the first { and the last }
  const firstBrace = raw.indexOf("{");
  const lastBrace = raw.lastIndexOf("}");
  if (firstBrace !== -1 && lastBrace !== -1 && lastBrace > firstBrace) {
    const extracted = raw.slice(firstBrace, lastBrace + 1);
    return JSON.parse(extracted);
  }

  throw new Error("No JSON object found");
}

This approach preserves embedded code fences, handles models that occasionally wrap JSON in a single pair of backticks (the braces still exist), and doesn’t penalize providers that return pristine JSON.

Additional Patterns That Keep Pipelines Running

Keep the system prompt explicit about forbidden invention. Structured output guarantees shape, not truth. Our prompt includes: “If a requested fact is not supported by the provided sources, leave it out rather than inventing it.” Schema compliance and factual accuracy are unrelated problems; mixing them leads to beautifully‑typed hallucinations.

Add a lightweight evaluation step for mission‑critical pipelines. Even after parsing succeeds, run a few assertions against the output—expected keys, value types, content‑based rules. This catches the infamous “valid JSON, wrong content” failure without a heavy ML eval framework.

Log every raw response (sanitized if needed) for at least a few days. When parsing fails in production, the first question is always “what did the model actually return?” Having the raw payload in your logs turns a mystery into a five‑minute fix.

FAQ

Why does my LLM return 400 when I request json_schema with strict mode?

Some providers or gateways don’t support strict: true. The fix is to catch the 400 status and retry the same messages with response_format: { type: "json_object" }, then use a tolerant parser that handles unconstrained output.

Can’t I just strip triple backticks before parsing JSON from an LLM?

Avoid blanket fence‑stripping. If any field in your schema can contain backtick code blocks (like Markdown), the stripping logic will corrupt the payload. Instead, attempt JSON.parse directly, then fall back to extracting text between the first { and last } if that fails.

Does structured output prevent hallucinations?

No. Structured output guarantees that the JSON matches your schema, but not that the content is correct. Always prompt the model to omit unsupported facts rather than invent them, and consider adding a post‑parse validation or eval step for high‑stakes pipelines.