On Monday morning, a sales manager asks AI in a chat window: “How many new customers did we get last month?” The AI quickly answers with a number and a reasonable explanation. At the meeting, finance reports a figure 20% lower. Digging in, they find the AI counted “new accounts created” in the sales system, while finance counted “first payment.” Both tables are called “customers,” neither was wrong, and nobody had ever said which one this question should use.

Letting colleagues ask AI about company numbers in plain language is one of the things teams look forward to most after connecting their data. But a fast, fluent answer doesn’t mean the number is ready for a meeting. Whether the answer is right usually depends less on the model than on whether someone named the data source, wrote down the metric definition in advance, and keeps both maintained.

Same model: 21% versus 95%

In June 2026, Anthropic published its own internal case. It has Claude handle the company’s internal business analytics questions; about 95% of analytics queries are now completed by Claude, with about 95% accuracy overall.

That number comes with a condition. Before the background material was organized, the same Claude scored no higher than 21% on Anthropic’s internal evaluations. What raised it to 95% was a set of instructions they call skills, which tell the AI which source to check first for each kind of question and which analysis steps to follow. They also require the AI to check the semantic layer first (a data interface that manages metric formulas centrally so every tool calculates the same numbers), and to write its own queries only when the semantic layer doesn’t cover the question. InfoQ, reporting on the case, quoted Anthropic’s conclusion: the results came mainly from data governance, semantic definitions, and operational discipline, with model advances playing a smaller part.

Definitions are written by people, and they go stale

Two details in this case matter more than the 95% figure.

First, people own the metric definitions. Anthropic tried having AI generate metric definitions automatically, and evaluation results got worse. How to count “active users,” “churn,” or “closed deals” is a company decision, not something AI can derive.

Second, definitions go stale. They found that accuracy of about 95% at launch fell to about 65% after a month without maintenance. Tables change, fields get renamed, business rules shift, and instructions that were correct become misleading. So they treat maintaining these instructions as real engineering work, not a one-time setup.

Together, these mean that letting AI answer number questions adds a document someone has to own. Without an owner, accuracy slides while nobody is watching.

Advertisement

Before opening up, settle three things for one question

There’s no need to build a data platform first. Start with the number question your team is asked most often and settle three things for it:

  • The official source. Which table, report, or system answers this question. In the opening example, “new customers” should be written down as “customers making their first payment, from the billing system.”
  • The formula and exclusions. How the time range is counted, whether test accounts are excluded, whether refunds are deducted. If departments calculate it differently, the person responsible for the metric decides which version to use.
  • The maintainer. Who updates the definition when tables or rules change, and how often it’s checked.

Once that’s done, require the AI to state with every answer which source it used, which conditions it applied, and what it excluded. An answer that can’t show this is only a draft.

Plausible-looking errors still slip through

Even with sources and definitions in order, one kind of error is still hard to catch: an answer that’s wrong but looks completely reasonable. Anthropic admits it doesn’t yet have a robust solution for these “silent” failures. Its current practice is to attach a source note to every answer, require human sign-off on any number going to leadership, and compare daily against a few reports known to be correct. It also tried having a second AI dedicated to finding flaws in answers. Accuracy on its evaluation rose by about 6%, at the cost of 32% more tokens and 72% longer response times. Whether that cost is worth paying depends on how bad a wrong number would be.

So numbers that affect budgets, performance reviews, pricing, or commitments to customers still need a person to confirm them, even when AI can find them. AI can greatly shorten the time spent finding data and framing questions, but which number is fit for decisions is still the company’s call.

As a next step, pick the number that’s asked about most and argued about most, and write down its official source, formula, and maintainer. Once that’s done, let AI answer it. If you’re concerned about the usage and cost of AI queries, plan alongside Measure an AI Bill by the Cost of Each Accepted Result, Not the Price per Million Tokens. If your data often has blank cells, how AI interprets them also affects the numbers; see Don’t Let AI Count Blank Cells as Zero.

AI handoff card

Ask AI whether a number question is ready for self-service This prompt asks what the question is for and where things stand, checks whether source, formula, and maintainer are all in place, and judges whether AI can answer it directly.

I want colleagues to be able to ask AI one company number question directly, but I want to check first whether it's ready.
The method comes from this article: https://boosterminiclass.com/en/posts/ai-analytics-source-of-truth-checklist/
Its view: whether AI gets numbers right depends on someone naming the official data source, writing the metric formula, and keeping both maintained. In Anthropic's case, the same model went from under 21% accuracy to over 95% after this work, then fell to about 65% after a month without maintenance.

Ask me 3 questions first:
1. Which number question is it, and what decision will the answer be used for?
2. How many places in the company can this number be found today, and do they calculate it the same way?
3. Does anyone own this metric's definition, and who would notice if tables or rules changed?

Based on my answers, give me:
- The status of the three things, source, formula and exclusions, and maintainer: settled / to confirm;
- A one-line verdict: open to self-service / test only, with human review of answers / keep answering it manually for now;
- What the AI must state with every answer to this question (source, conditions, exclusions);
- One improvement that can be done today.
If information is missing, ask me first. Don't guess the formula or calculate the number yourself.

This card only helps you find what’s missing. The official number must still be confirmed by the person responsible for the metric.

Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References