When teams connect AI to company data, the first expectation is often: “People can just ask AI, so they no longer have to wait for the data team to pull reports.”
That expectation is understandable. Sales wants this month’s conversion rate, support wants to know which issues are increasing, and product managers want retention for a new feature. If every question needs a data teammate to write a query, the process is slow.
But Anthropic’s recent internal case is a useful reminder: whether AI can answer company numbers is not only about whether the model can write queries. The real risk is that if it does not know which table is official and which metric definition the company uses, it may state the wrong number in a very fluent voice.
Anthropic says Claude now automatically handles about 95% of internal business analytics queries, with roughly 95% accuracy. More importantly, when the same Claude lacked prepared analytics process and background knowledge, internal evaluation accuracy did not exceed 21%. After metric definitions, data sources, and analysis steps were organized into operating instructions that AI reads before answering—Anthropic calls them skills—accuracy became stable above 95%.
This article is not asking you to copy Anthropic’s data platform. It turns the case into a more general workflow question: before AI answers company numbers, first fix where it should look, how it should look, and who confirms the answer.
Self-service analytics does not mean “let everyone ask anything”
Self-service analytics means people outside the data team can check operating numbers themselves. Sales, support, marketing, or product teammates do not need to open a ticket every time.
Before AI, this often meant dashboards or reporting tools. Once AI is added, people can ask in natural language: “Which channel brought new customers last week?” or “Which support issues increased?” That sounds much more convenient.
The problem is that company data is rarely a clean dictionary. Different teams may calculate “active users” differently. Old and new tables may coexist. The same customer may have different fields in the CRM, or customer relationship management system, and the billing system. If AI simply searches the data warehouse—the place where many data tables are centrally stored—it can easily find a source that looks relevant but is not the official answer.
So the first question in AI self-service analytics is not “how strong is the model?” It is: “Have we made the official answer sources discoverable and usable by AI?”
Three common failure modes
When AI is connected to company numbers, the error is usually not simple arithmetic. These three failures are more common:
| Failure type | What it looks like | What you actually need to fix |
|---|---|---|
| Wrong source | AI answers an official question with an old table, test table, or private team-maintained table. | Do not just tell AI to be more careful; mark which table is official. |
| Wrong metric definition | It mixes trial accounts, paid accounts, and cancelled accounts as “customers.” | Do not only fix one answer; write the metric definition as a reusable rule. |
| No review trail | AI answers quickly, but does not say where it looked, what it excluded, or what remains uncertain. | Require source, query path, and human sampling review, not prettier wording. |
These failures have one thing in common: they do not disappear just because you ask again. You need an internal “answer map” first, so AI does not have to guess among many similar tables.
Four release conditions before AI answers numbers
If your team wants AI to answer revenue, usage, customer, or product numbers, use this table before opening access.
| Release condition | What to verify | If you cannot answer, do not open it yet |
|---|---|---|
| Single source of truth | Does each common question have an official table, report, or semantic layer? A semantic layer can be understood as the layer where company metric definitions are centrally managed. | If AI may choose among three similar tables, do not let it answer official numbers. |
| Metric definitions | Are “active user,” “churn,” “conversion,” and “usage” defined with algorithms and exclusions? | If different managers use different formulas for the same word, AI will amplify the disagreement. |
| Query steps | Does AI know where to look first, how to respond when data is missing, and when to ask for more context? | If the instruction only says “answer the question,” it will likely force an answer from the most similar data. |
| Review responsibility | Who samples high-risk answers? Which answers must include source, query conditions, and uncertainty notes? | If no one can trace how an answer was produced, errors move into meetings, decks, and decisions. |
The point of this table is not to add paperwork. It prevents a team from confusing “data lookup” with “reliable conclusion.” AI can speed up queries, but it cannot decide which number is the company’s official version.
Which questions should not be handed directly to AI?
Not every question should be available to every teammate through AI. Keep human confirmation when:
- The number affects budget, performance review, layoffs, pricing, or customer promises.
- A metric was recently changed and old and new definitions still coexist.
- The question involves personal data, sensitive customers, contract terms, or permission limits.
- The user asks too vaguely, such as “How did we do last week?” without product, region, segment, or time range.
- AI cannot find an official source but still tries to answer from similar data.
These cases do not mean AI can never be used. They mean AI should stay in a drafting and prompting role. It can list possible metrics, point out missing conditions, and draft a better query, but the final number should be confirmed by a data owner, metric owner, or business owner.
A small-team version
You do not need a large data platform before reducing risk. A small team can start with three steps:
- List the ten numbers people ask for most often. For example: new customers this month, cancellation reasons, support volume, conversion rate, and feature usage.
- Assign one official source to each question. Write down the table, report, system, or responsible person. Do not make AI guess.
- Require AI to say where it looked. If it cannot explain the source, time range, and exclusions, treat the answer as a draft, not a decision input.
After these three steps, more advanced automatic queries, semantic layers, or dedicated AI analytics workflows become much safer.
The lesson
The appeal of AI self-service analytics is that more people can get answers without waiting for the data team. The danger is exactly the same: more people can get seemingly reliable answers much faster.
So do not ask first, “Can AI help everyone check numbers?”
Ask instead: “Have we fixed the answer sources, metric definitions, query steps, and review responsibility?”
If those four things are not fixed, faster AI only spreads errors faster. Fix the source of truth before opening self-service queries. That is how AI reduces data-team load instead of creating more numbers that need cleanup later.
Everyday four-panel comic
- A teammate wants to ask AI directly for an operating number, while too many similar data cards surround the question.
- AI finds several seemingly related tables, so the team pauses before it answers.
- The team fixes official sources, metric definitions, and review checkpoints on one workflow board.
- AI follows the fixed path to prepare an answer, and a human reviewer confirms the source and conditions.
AI handoff card
Ask AI to organize this article's specific situation
Copy this into your own AI chat tool to turn this mini class into a personal checklist. BMC will not see what you paste into your AI tool.
References
- Anthropic: How Anthropic enables self-service data analytics with Claude — https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude
- InfoQ: Anthropic Reports Claude Now Handles 95% of Internal Analytics Queries — https://www.infoq.com/news/2026/06/anthropic-claude-analytics/
- Kenta Yamamoto (note): From 21% to Over 95% Accuracy with the Same Claude: How Anthropic’s Internal Self-Service Analytics Revealed the Importance of Data Infrastructure — https://note.com/pn_yamaken/n/nd3ae714ce54a?hl=en
