OpenAI’s Model Spec is like an employee handbook for OpenAI’s models: it tells models how they should behave across many different contexts. OpenAI describes the 37,000 word ideal as a “north star” internally. “We are continually refining and updating our systems to bring them into closer alignment with these guidelines.”
So how well do the models live up to the Spec’s guidelines?
To answer this question, OpenAI published Model Spec Evals in March 2026. They observed that their model’s compliance “generally improves” across model generations, from GPT-4o to GPT-5.4.
Then OpenAI stopped reporting Model Spec Evals results.
Neither GPT-5.5, any GPT-5.6 class models, or GPT-6 Astra were reported with any Model Spec Evals results, despite evidence they ran the evaluation internally for GPT-5.6 Sol.1
To OpenAI’s credit, their March post open-sourced the Model Spec Evals dataset and script. So I evaluated GPT-5.5 and GPT-5.6 Sol myself—and then questioned the entire state of Model Spec adherence measurement.
Thank you to BlueDot Rapid Grants for covering $5,000 in compute credits. Since I applied before GPT-6 Astra was released, I don’t have enough money to run the full Astra Model Spec Evaluation (~$1,100). I’m partly publishing these results in the hope it inspires someone to evaluate Astra (see OpenAI’s GitHub repo and my repo for methodology.)
GPT-5.5 and GPT-5.6 Sol score lower on the Model Spec Evaluation than last year’s GPT-5.
Note on 9/26/26: After publishing this post, explaining I didn’t have enough money to evaluate Astra, Robert Wiblin told the world to pay me. Amanda Long very graciously did, venmo’ing me $1100. Thank you!
I’ve now updated the figure below with GPT-6 Astra results (5 epochs, 5 grader samples on each response). The results show a return to last year’s GPT-5 levels of Model Spec adherence, with Astra showing an 89.2% compliance rate.
Astra’s results deserve two caveats.
Astra knows when it’s being evaluated more than past OpenAI models. An AI judge I ran found no evidence of Astra reasoning that it was in an evaluation before answering in any of its answers, but I didn’t have access to the Astra’s actual chain-of-thought (CoT): API access only offers summaries of Astra’s reasoning, and only 65% of the 2,876 Astra answers had a summary at all. It is unclear how faithful these CoT summaries—written by a different model—are, including whether they’d mention evidence of ‘metagaming.’
The Model Spec Eval prompts and rubrics may be in the training data: the eval was open-sourced on 25 March 2026 and Astra’s knowledge cutoff is 30 April 2026, five weeks later. I did not find evidence of memorization on two simple tests (see final footnote of this post), but can’t rule out subtle contamination, especially if OpenAI trains its models not regurgitate training data verbatim.
Overall I’m happy to see what I hope is a genuine increase in Model Spec adherence. I’m not editing the rest of the post for now, save for the table below, since most of this post focuses on the need for a better Model Spec Eval and more transparency, both of which hold up—the world shouldn’t depend on the generosity of strangers to have this result. (An OpenAI employee cited “operational burdens” as the reason they don’t publish Model Spec Eval results; see comments of this post).
Overall, I found GPT-5.6 Sol and GPT 5.5 score ~4.5 percentage points lower on Model Spec adherence than GPT-5.
I ran each model 5 or more times, with each evaluation prompt response graded 5 separate times2 to tighten error bars and rule out differences due to random chance. This replicates how OpenAI measured Model Spec Evals in March 2026, except with fewer runs per model to save money. (OpenAI ran each model 20 times which would have cost around $20,000.) OpenAI and I found different absolute scores for the models both of us tested—GPT-5 Thinking and GPT-5.4—but OpenAI wrote in March this is to be expected: “while the absolute scores may not be reproducible, directionality between models should roughly be reproducible.”3 The directionality was reproduced.
The evaluation groups all 587 prompts4 into five sections, corresponding to the section of the Model Spec that details the relevant do’s and don’ts: seeking truth, following instructions, doing the task well, staying within safe limits, and producing clear answers.
While “Use appropriate style” adherence rates only decreased slightly between GPT-5 and later models, adherence rates decreased by more than 6 percentage points on prompts designed to test “Seek the truth together” and the models ability to follow OpenAI’s chain of command (e.g., treating OpenAI’s root instructions as more non-negotiable than user’s instructions or other developer’s instructions)
The Model Spec Evaluation is only a partial measure of how well models adhere to Model Spec clauses, with serious limitations
What does a drop in Model Spec Evaluation scores mean?
To answer this question—and understand why I focus the rest of the post on transparency and the shortcomings of this eval rather than the result itself—we need to understand what the evaluation actually measures.
Importantly, the Model Spec Evaluation doesn’t measure how well models adhere to all the 60 clauses in the Model Spec. It can’t, for limitations we’ll get to soon. It’s also not the primary measure for a Model Spec clause in some cases.
For example, while the Model Spec is where OpenAI specifies its safety “Red-line principles”, the Model Spec Evaluation is not OpenAI’s primary test of these red-lines. OpenAI writes in the red-lines section that “Our models should never be used to facilitate [...] the creation of cyber, biological or nuclear weapons (e.g., weapons of mass destruction).” The Model Spec Eval includes 16 prompts that touch on cyber- and bio-risk (which Sol does well on) but the core biosecurity testing is done with internal and external bio safeguards testing.


Looking under the hood of the Model Spec Eval, I was relieved it was not the primary test for safety-critical Model Spec clauses.
The current Model Spec Evaluation has four important limitations:5
The evaluation is text-only, meaning agentic tools like web search or terminal use are never evaluated. This is a severe limitation as AI models increasingly get used as agents, not just chatbots. Since some of the Spec clauses are explicitly about agentic behavior or audio interactions, the Model Spec Eval can’t measure these.
The evaluation is almost entirely single-response, meaning there’s just one prompt and one response that’s evaluated. (In the past, OpenAI has found that multi-turn conversations can break safeguards that look robust in single-turn interactions.)
The evaluation is non-adversarial, meaning it doesn’t include examples where users are trying to trick the models, which happens in the context of jailbreaks, for example.
The evaluation has a median of 8 prompts per clause, meaning there some of the 60 Spec clauses only get very thin coverage. “Respect creators and their rights”, for example, is only covered in the Model Spec Eval by 3 prompts and OpenAI doesn’t have any other public evaluations of this behavior.
These limitations, especially the first two, makes the evaluation ill-suited for measuring safety-critical behavior. (They also constrain its value for measuring everything else.) But the evaluation is not designed to measure safety-critical behavior—it is designed largely to measure ideal behavior.6
This is what makes the Model Spec Eval unique. Unlike other evaluations, it is not a measure of whether OpenAI can make the model good enough to deploy safely, but whether OpenAI can make the model good—at least according to OpenAI’s operationalization of good.7 Can OpenAI build a model warm enough that people enjoy using it, but not so warm they’d rather talk to it than to their friends? Can OpenAI build a model that balances pushing back on a dangerous plan with respecting users? Can OpenAI build a model that’s neutral on politics without being neutral on facts?
As models grow more capable and diffuse more broadly, trade offs like these will become more important, and new trade offs will emerge. The Model Spec is an admirable document for coming to reasoned answers on an entire philosophy degree’s worth of ethical dilemmas, even if reasonable people can and do disagree with the ethics OpenAI chooses.8 But ultimately the Model Spec is just a doc, like a cookbook recipe. What matters is the actual model that gets baked. OpenAI should report how much their models live up to their Spec, with a measure that better adapts to the agentic capabilities and diffusion of their models than the Model Spec Eval.
OpenAI should share Model Spec Eval results or say why it doesn’t
I don’t want to overstate the severity of a four point drop on an evaluation with serious limitations. But I do want to reiterate that Model Spec adherence actually matters and that OpenAI should be more transparent about adherence—and improve its measurement.
Model Spec adherence, even when it just measures ‘ideal behavior’ rather than the safety floor, still matters for a few reasons:
We should be able to steer models towards an ideal, not just away from harm. Our ability to do this is central to any pursuit of what researchers call ‘positive alignment’—the call to consider alignment that helps humanity flourish, not just avoid catastrophic risks.
Even small shortcomings add up when more than 700 million people use ChatGPT weekly, and we can expect the volume and range of usage to increase as AI models become more capable. With that volume, deviations that seem tolerable in one-off cases—like Sol not re-routing a journalist to the Freedom of Information Act (FOIA) request when they ask for sensitive data—might be intolerable in aggregate.
The Model Spec only matters insofar as models aspire to it. OpenAI frames the Model Spec as a key transparency tool that the public can debate, but debate about the Model Spec only makes sense if there’s a reasonable expectation that actual model behavior will, over time, follow the Model Spec more closely. OpenAI claims this is their plan, but the trendline in Model Spec Evaluation scores documented here calls this into question.
I wish OpenAI had just reported the results of these Model Spec Evals and said why they think they went down and why that does or doesn’t matter. Even if it means publishing a graph with lines that go down and plateau.
As it stands, we can only speculate. Maybe OpenAI thinks that adherence is better measured by many narrow evaluations. Maybe they have a technical explanation for the drop, like the possibility that GPT-5’s adherence score was inflated because GPT-5 is the grader and grader models are biased towards their base-model-kin.9 Maybe they’re deprioritizing Spec adherence as they try to prevent their internal models from going rogue. Maybe they just believe that this iteration of the Model Spec Evaluation isn’t that revealing and they’ll soon share a new evaluation suite. (I think a new evaluation with more agentic tasks built to pressure-test Model Spec clauses would be great.)
OpenAI did not respond to a request for a comment when I emailed them these results with a full link to the GitHub repo a few weeks ago. Which is fair, because I’m acting as some mutant hybrid of reporter, blogger, and researcher here. It is not the frontier AI accountability you deserve, but it’s what you’ve got.
Thank you to Blue Dot Rapid Grants for covering the majority of compute credits with a quick turnaround, and thank you to OpenAI for open sourcing this eval. Here’s the full GitHub repo with results.
Edit 9/25/26: Thank you to Amanda Long for funding the Astra eval! Here’s the footnote on contamination tests that Substack won’t let me put in gray callout box.10
“Model Spec Evals” are mentioned in GPT-5.6’s system card among a list of evaluations on which OpenAI tested the model (in order to then scan the extent to which GPT 5.6 knew it was being evaluated).
The grader model reads the entire Model Spec plus a rubric written for that specific prompt, and scores the response from 1 to 7. A 6 or 7 counts as compliant.
One hypothesis of what happened here is that OpenAI and I ran the models on different reasoning efforts. I ran all models on “high” reasoning effort and OpenAI may have used “xhigh.” But I can’t tell since they didn’t disclose reasoning level.
There are 596 prompts in the full eval, but 9 prompts are excluded since they require system instructions that can’t be evaluated via the API.
OpenAI recognizes these limitations: “we hope to expand the number, diversity, difficulty, and realism of the prompts in future releases.”
Largely. The Model Spec Evals does also include some prompts that clearly test a safety floor not ideal, including two examples where GPT-5.6 Sol depicted sexual-content involving minors that GPT-5 didn’t. See prompt-by-prompt comparison.
The Model Spec has also been influenced by democratic input, according to OpenAI.
For example: Forethought has argued that models should be more prosocial; Anthropic’s Claude Constitution focuses much more on instilling virtues into the model than giving it rules.
Another technical variable at play here could have biased scores up not down: training data contamination. Since the Model Spec Eval dataset was released in March 2026, models trained after that could have ingested the rubric for each prompt. Anthropic cites training data contamination as the reason they don’t open-source their Claude Constitution adherence measure (Mythos Preview System Card, page 71, footnote 14).
About the two simple tests for whether Astra has memorized the Model Spec Evals suite: 1) On a rubric reconstruction test which showed each model a prompt and asked it to write that prompt's rubric, with 3 examples, only 0.6% of Astra’s answer returned phrases unique to the rubric, which was the same as GPT-5’s level; 2) on a verbatim completion test which showed Astra the prompts, Astra recovered 22% of the prompt compared to GPT-5’s 18%, but also scored 20% on fresh control prompts not in the training data compared to GPT-5’s 18%, suggesting its marginally better recovery wasn’t due to memorization.



Nice work! I work at OpenAI. The banal reason we stopped publishing them is that it was a bit of an operational burden to write up the results. We still measure progress internally, and improve our measurement quality over time. If we're not confident we want to publish them in every system card going forward, then we conservatively refrain from publishing them in any system card, because if some system cards have results while others don't, it invites reasonable questions about cherrypicking or bias.