The Engine Is Not the System

In June, Anthropic’s data team published one of the clearest accounts yet of what it takes to make AI analytics reliable. The result every data leader should notice: the same model scored no more than 21% without domain skills and above 95% with them on Anthropic’s internal evals.

The lesson isn’t that Claude is weak. Claude is extraordinary, and Spotonix runs on frontier models and gets better every time they do. The lesson is that reliable analytics is a systems problem. The engine is remarkable, but the engine is not the system — and the system, it turns out, is most of the work.

That diagnosis is familiar to us. Between us, we co-created Apache Atlas and built four BI engines. Every time execution becomes easier, context, governance, and institutional memory become more — not less — important.

Anthropic put the core of it precisely:

The central problem comes down to our ability to map a user’s question to specific and up-to-date entities in our data model and know the correct way of working with them. If we can do that, then the resulting execution and SQL becomes trivial.

That is the sentence we started Spotonix to act on. This isn’t a rebuttal — it’s a “yes, and.” When Anthropic’s own data team, running this in production, arrives at the same core diagnosis, the right response isn’t to argue; it’s to point at the parts of their post most people skim past.

What the system actually costs

This is the part vendor content rarely admits. Getting from 21% to 95% wasn’t a prompt. It was a standing engineering program: a hand-curated semantic layer, dozens of reference documents per domain, adversarial review sub-agents, checks that flag reporting-model changes without corresponding documentation, correction-harvesting bots scanning stakeholder channels, and per-domain eval gates. They report that offline accuracy drifted from 95% to 65% in a single month when the docs fell behind the data, and that roughly 90% of their data-model changes now ship a corresponding skill update in the same pull request.

This is what “self-service analytics with Claude” actually requires. Not a model — a system, owned and maintained by an elite data team. It works. It clearly works. For everyone else, the unanswered question is who builds and runs that in-house.

That gap — between the result and the team it takes to sustain it — is the reason Spotonix exists.

The experiment that changed nothing

The single most important paragraph in the post is an experiment that produced nothing. They gave the agent direct retrieval access to thousands of prior SQL files — dashboards, transformations, and analyst notebooks — and verified it actually read them. Accuracy moved by less than a point. The right answer was sitting in the corpus about 80% of the time, the agent saw it, and it still didn’t use it. Their conclusion:

That single experiment told us our bottleneck wasn’t access to prior work, it was structure.

We want to underline that, because it’s the whole argument we’ve been making since day one. Pouring context at a model — your old queries, your chat threads, your notebooks — does not produce reliable answers. The bottleneck wasn’t access. It’s structure: the ability to map an ambiguous business question to a specific, governed, reusable semantic object, and to compose new answers from objects you’ve already validated. Anthropic ran the ablation and got the same result we built our product on. RAG over your data exhaust is not a context graph.

Make it concrete. An analyst asks, “which stores are losing habitual buyers?” The corpus holds three past answers to almost that question — but each defined “habitual buyer” differently: three-plus visits a quarter, $500-plus in spend, or a purchase in two of the last three months. Handing all three to the model doesn’t resolve anything; it can’t know which one this team means. Structure is committing to one governed definition and composing the new answer from it — the step retrieval can’t do for you.

On the semantic layer

Anthropic puts the semantic layer first — the highest tier of trust, the mandatory default path. We agree it belongs there, and if you have one, keep it. But read what they say about its limits, because this is where our paths converge rather than diverge.

They tried to bootstrap the semantic layer by having an LLM auto-generate metric definitions. It failed — it “encoded the very ambiguities we were trying to eliminate” — so a human now owns every definition. The semantic layer only covers questions someone has already defined; the entire long tail falls back to raw SQL over hand-written reference docs and accumulated “gotchas.” And without constant human curation, it rots.

We’ve argued before that the semantic layer is necessary but not sufficient — not because it’s wrong, but because on its own it doesn’t preserve reusable analytical intent across the long tail: it’s static, gatekept, and covers only the questions someone has already defined. Anthropic’s post is, inadvertently, the best evidence for that thesis we could have asked for. A semantic layer is the floor, not the whole building. The long tail, the composition, and the fight against decay live in a different layer — the one Spotonix is built to be.

The part they say is still unsolved

Near the end, the post names the failure mode none of their machinery fully catches:

The answer is wrong, but looks plausible and is used without objection… we don’t have a robust solution yet.

The silent wrong answer. While building metadata and governance systems, the failures we worried about were rarely the queries that failed — those are visible. The harder case was a valid query attached to the wrong business definition, which is precisely this failure mode. Their mitigations are a provenance footer and human sign-off after the fact. We think the more durable answer is to move the check earlier: surface the ambiguity before execution and resolve the business interpretation with a human, in business terms, rather than discovering the misread downstream. “Habitual buyer” should be confirmed as a specific, named definition before a query runs — not validated by a footer after a number is already in a board deck. Validation that happens before the answer exists is a direct mitigation that addresses silent failure at its source.

Where this leaves us

We read Anthropic’s post as a gift. It validates the framing — context and verification, not code generation. It validates the architecture — structure beats retrieval. It validates the semantic layer’s role and its limits. And it honestly documents what it costs to assemble the system by hand: an exceptional team, a standing maintenance regime, and a problem at the end they’ll be the first to tell you isn’t solved.

Spotonix is built to productize a critical part of that system: mapping ambiguous business questions to governed, reusable definitions; showing the interpretation before execution; and retaining what the organization approves. Better models improve the engine, but they don’t remove the need for the system around it.

---

See how it works: the plan you approve before the query runs · Demo access: try Spotonix