In June 2026, Anthropic launched Claude Fable 5. The official announcement says that when classifiers judge a request to involve cyberattacks, biology and chemistry, or “distillation,” the previous-generation Opus 4.8 answers instead. Distillation means extracting a strong model’s outputs at scale to train another model. Anthropic had earlier published a report explaining why it guards against unauthorized distillation.

But according to Gizmodo, one safeguard aimed at frontier model development was originally designed to be invisible to users: instead of refusing or switching models, it quietly altered requests and produced faulty results. After strong backlash from researchers and developers, Anthropic admitted it “made the wrong tradeoff,” apologized publicly, and committed to making these safeguards visible to users.

The lesson isn’t limited to one company. Rules, routing, and safeguards on the vendor’s side can change at any time, and you can’t see them. What you can control is your own side: keep a fixed set of questions and the answers you got for important workflows, and when quality seems to change, rerun and compare, judging by evidence rather than impressions.

“It was better yesterday” is usually just an impression

The same tool that summarized research clearly yesterday is vague and skips details today. The instinct is to rewrite the prompt, start a new chat, or rephrase. Sometimes the prompt really is the problem, but if the cause is on the vendor’s side, all that effort is guesswork.

Worse, “it was better yesterday” is usually just an impression. If you didn’t save yesterday’s input and output, you can’t prove quality changed, nor know when it started or which batch of results it affected.

Build a fixed question set for important workflows

Pick the workflows whose output gets passed on, delivered, or merged into official versions, such as research summaries, draft customer replies, or code changes. For each, choose five to ten representative inputs and save them along with the model name, date, and output at the time.

When quality seems to change, rerun the same questions and compare against the saved answers to see what kind of change it is:

Explicit refusal: the model says it can’t answer. At least you know a limit exists and can check the official documentation.

Shorter, vaguer: details it used to give are missing. The model may have been swapped, the service updated, or a safeguard added.

Only one category of question got worse: for example, questions about security, medicine, or data extraction degrade while others are fine. This usually means rules changed for that area.

The task gets redirected: you asked for an evaluation and got a warning; you asked about A and got a safer B. This is the most dangerous, because it looks like an answer but isn’t what you asked for.

When checking, start with the vendor’s status page, release notes, and model documentation. Fable 5’s official announcement now states that users are informed when Opus 4.8 answers instead, that this happens in under 5% of sessions, and that the classifiers sometimes flag benign requests. Only rules that are written down can be checked against.

Advertisement

Let task risk decide the response

For low-risk tasks like brainstorming drafts or rewriting tone, if quality drops, change the prompt or the tool; no big response needed.

For medium-risk tasks like research summaries, internal reports, and code drafts, you can switch models, but keep the input and output differences so new results aren’t compared directly with old conclusions.

For high-risk tasks like security reviews, contract language, customer commitments, and merges into production code, as soon as the fixed questions show a difference, pause delivery, retest key cases, and let the owner decide when to resume. If a vendor admits a change was originally invisible, results from the affected period should all be rechecked.

A fixed question set follows the same logic as defining scoring criteria before letting AI automatically change code; see Before Letting AI Search for Faster Code, Write Down What “Better” Means.

A fixed set only detects what it covers

A fixed question set only detects what you test. Vendor safeguards often target specific types of requests, and if your questions don’t cover them, you won’t see a difference; the questions also need regular updates as workflows change.

Also, detecting a difference doesn’t mean you can find the cause. Often you can only be sure that “it changed,” not “why.” That’s enough to make decisions: which deliveries to pause and which tool to use instead. For when AI should stop and ask a person, see Before Connecting AI to a Workflow, Define When It Should Stop and Ask a Person.

AI handoff card

I want to build a fixed question set for several important AI-dependent workflows on my team, so that when AI output quality suddenly changes we can judge by evidence rather than impressions.
The method comes from this article: https://boosterminiclass.com/en/posts/claude-fable-invisible-guardrail-checklist/
Its view: rules, routing, and safeguards on the vendor's side can change without users seeing it. What you can control is your own side: keep a fixed set of questions and the answers you got for important workflows, rerun and compare when quality seems to change, then decide by task risk whether to pause delivery.

First ask me: which workflows use AI, who receives each workflow's output, and which model we currently use.
Then:
1. Classify the workflows as low, medium, or high risk, with reasons;
2. Design five to ten fixed questions for each medium- and high-risk workflow, covering both typical cases and cases likely to be restricted, and explain what to save (model name, date, input, output);
3. Write a comparison method for reruns that can tell apart four kinds of change: "explicit refusal," "shorter and vaguer," "only one category got worse," and "task redirected";
4. List the official information to check first when a difference appears, and which deliveries to pause and who decides to resume.
Don't judge which vendor has a problem for me; just organize the method.
Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References