If you've tried to get an AI model to return JSON and then parse that JSON in your application, you've probably run into the same problem everyone runs into. Sometimes it works perfectly. Sometimes the model adds a sentence before the JSON. Sometimes it wraps it in markdown code blocks. Sometimes it returns something that looks like JSON but has a trailing comma or an unquoted key that breaks your parser. And the frustrating part is that it works nine times out of ten, which makes the one time it breaks feel impossible to predict or prevent.
Getting consistent, valid JSON from AI models is one of the most practically important prompt engineering skills for anyone building applications, and it's more solvable than it seems.
Why models struggle with JSON consistency
The core issue is that AI models are trained to generate natural language, and JSON is a strict format that doesn't leave room for the small variations that natural language tolerates. A model that's very good at writing coherent paragraphs might add an explanatory sentence before the JSON because that's what would be helpful in a natural language context. It might wrap the JSON in code blocks because that's how JSON is typically presented in the technical writing it was trained on. It might include a comment inside the JSON because comments are natural in code even though they're not valid in JSON.
None of these are mistakes in a general sense. They're the model doing what it's trained to do in a context that requires a different behavior. The solution is to make the required behavior as explicit as possible so the model has a clear target to aim for.
The most effective prompt instructions for JSON output
The single most effective instruction is to tell the model explicitly that it should return only the JSON and nothing else. Not "return the response as JSON" but "return only valid JSON with no additional text, no explanation, no code blocks, and no markdown formatting." The specificity of that instruction matters because it addresses each of the common failure modes directly.
Providing the exact schema you want is the second most important thing. Don't describe the structure in words if you can show it instead. Give the model an example of the exact JSON structure you expect, with the correct keys, the correct nesting, and the correct value types. The model will follow a concrete example much more reliably than a verbal description.
If there are fields that might not always have a value, tell the model explicitly what to put in those cases. Should it use null, an empty string, an empty array, or omit the key entirely? If you leave this ambiguous the model will make its own choice, and that choice might be different from request to request.
According to The Verge's coverage of AI developer tools, structured output features that enforce JSON schema compliance at the API level have become one of the most requested features among developers building production AI applications, precisely because prompt-level instructions alone aren't always reliable enough.
Using structured output features when available
Several AI providers now offer structured output modes that enforce JSON compliance at the API level rather than relying on the model to follow prompt instructions. OpenAI has a response format parameter that you can set to JSON mode or to a specific JSON schema. Anthropic has similar capabilities for structured outputs with Claude.
These features are more reliable than prompt instructions alone because they constrain the model's output at a lower level than the prompt. The model can't accidentally add extra text or malformed syntax because the output is validated and corrected before it reaches you.
If you're building something where JSON consistency is critical, using these structured output features is worth doing even if it requires a small amount of additional setup. The reliability improvement is significant.
Handling the cases where it still breaks
Even with good prompt instructions and structured output features, it's worth building your parsing code defensively. A few simple things make your application much more robust.
Strip any leading or trailing whitespace and look for markdown code block markers before parsing. A response that starts with three backticks and "json" can be cleaned up with a simple string operation before you try to parse it. This one check handles a large percentage of the formatting inconsistencies that slip through.
Wrap your JSON parsing in a try-catch and handle parse errors gracefully. When parsing fails, log the raw response so you can see what the model actually returned. That logging is invaluable for understanding failure patterns and improving your prompts over time.
For critical applications, consider adding a validation step after parsing to check that the parsed object has the structure you expect before you use it. A response that's valid JSON but has the wrong keys or missing fields is just as problematic as invalid JSON, and schema validation catches those cases before they cause downstream problems.