Are LLMs Too Expensive for Analytics? I Scored a Platform Against Anthropic's Playbook to Find Out
I recently attended a webinar hosted by Dave Langer, and he made an argument worth taking seriously: a well-built data warehouse is a prediction engine, not just a rear-view mirror, and the better-ROI path to those predictions is traditional machine learning, not large language models. LLMs, in his view, are simply too expensive for what they return in analytics.
That is a direct challenge to the current AI hype. Claude has helped me immensely in building out a data platform for a client: a Microsoft Fabric lakehouse with a medallion architecture, automated pipelines from the client’s ERP and CRM systems, and Power BI semantic models on top. It let me deliver something massive in a fraction of the time it would normally take. But I had noticed something too: when I asked it for factual insights from the data, it sometimes struggled.
Dave also pointed the audience to Anthropic’s recent article on how they automated around 95% of their internal business analytics queries using Claude. It is one of the most honest write-ups I’ve seen on what LLM-driven analytics actually costs to make reliable, including what didn’t work. Which makes it the perfect instrument for testing his argument.
So I did the obvious thing: I scored the platform against Anthropic’s playbook, layer by layer, keeping a running tally of where the accuracy actually comes from and what it costs.
The result was a 7 out of 10, one finding that changed how I work, and a grudging concession that Dave has more of a point than I expected.
The Four-Layer Stack
Anthropic’s article describes an “agentic analytics stack” with four layers:
- Data foundations: canonical, governed datasets instead of forty plausible candidates; everything colocated in one repository; metadata treated as a product.
- Sources of truth: a semantic layer the agent must consult first, plus lineage and business context it can navigate.
- Skills: procedural knowledge files that encode how a senior analyst actually works. Their headline stat: 21% accuracy without skills, over 95% with them.
- Validation: offline evaluation suites, adversarial review of answers, and monitoring for drift.
Their diagnosis of why AI analytics goes wrong is sharp. It is almost never the SQL. It is mapping a user’s question to the right entities in the data model: which table is canonical, which filters are mandatory, what “active” actually means in this business.
The Report Card
| Layer | Score | Verdict |
|---|---|---|
| Data foundations | 8.5 / 10 | Strongest layer |
| Sources of truth | 5 / 10 | Biggest structural gap |
| Skills | 8 / 10 | Strong, with one hidden liability |
| Validation | 6 / 10 | Strong online, absent offline |
Foundations held up well. The medallion architecture does exactly what Anthropic prescribes: the bronze layer is for investigation only, and every report and model reads from a governed silver layer with a single canonical table per concept. Pipelines, semantic models, documentation, and AI skill files all live in one repository, which is their exact recommendation and the one most organisations miss. A live-queried data dictionary covers grain, exclusions, and column semantics.
Skills scored high too. This layer was earned the hard way. Early in the build I kept catching incorrect results: plausible-looking queries, wrong numbers. What turned it around was deploying the guidance from an earlier Anthropic article on agent best practices: encode the workflows, project knowledge, and verification steps the AI needs as explicit instruction files, rather than hoping it infers them. The improvement was major and almost immediate. Today, over a dozen encoded workflows cover pipeline health scans, table profiling, revenue reconciliation, and loader builds, each with the hard-won gotchas baked in. Given Anthropic measured a jump from 21% to 95% accuracy when they added this layer, it is not optional.
The gaps were just as instructive. There is no enforced “semantic layer first” rule: the metric definitions exist in the Power BI models, but for ad-hoc questions the agent goes straight to SQL and re-derives business rules each time. That re-derivation is exactly where Anthropic says subtle wrongness creeps in. And while every piece of work gets adversarially verified against live data before it ships, there is no offline evaluation suite: no bank of question-and-known-answer pairs that would catch silent drift in the numbers.
To be clear, I knew this was a no-no when I made the call. The brief was speed with accuracy, delivering a trusted reporting layer fast to an organisation that had lost confidence in its numbers, and something had to give. Skipping the enforced semantic-layer discipline was the deliberate trade-off: the accuracy was protected by adversarial verification on every piece of work instead, including iteration with subject-matter experts who interrogated the numbers in the Power BI models, which costs more effort per question but needs no upfront build. Now that the platform has bedded in, I am circling back to close exactly that gap, routing ad-hoc questions through the semantic layer where the business rules are already defined once.
The Finding That Surprised Me
The most valuable output of the exercise wasn’t a score. It was this:
The AI’s private memory had quietly become the platform’s gotcha corpus.
Months of daily work had taught the agent dozens of hard-won facts about the client’s data. Things like: the ERP’s ledger table keeps a stale “remaining amount” on closed invoices, so you can only trust it on open ones. One source system’s headline balance column goes negative and the pro-rata column is the one that matches reality. Every CRM query needs the test-data exclusion applied, without exception.
Exactly the “gotchas” a senior analyst carries in their head. And exactly as fragile: locked inside one agent’s memory, invisible to the team, unversioned, unreviewed. If a new analyst or a second AI agent started on this warehouse tomorrow, they would rediscover every one of those lessons the hard way, probably by shipping a wrong number first.
Tribal knowledge is tribal knowledge, whether it lives in a person or an AI.
The Fix List
Three fixes, in value order:
- Move the gotchas into the repository. Per-domain analyst reference docs following Anthropic’s template: business context, entity grain, mandatory hygiene filters, key tables, and the wrong-answer modes a senior analyst would warn you about. Each claim re-verified against live data before it is written down, because memory reflects when it was learned, not necessarily what is true today.
- A thin offline evaluation set. A few dozen question-and-answer pairs pinned to settled periods, run periodically. When a pipeline or a business rule quietly drifts, this is what catches it before a stakeholder does.
- Semantic layer first. Routing ad-hoc metric questions through the existing Power BI models rather than re-deriving logic in SQL. The measures already encode the business rules; the discipline is making the agent use them.
The first fix costs about half a day, and I expect it to be the best value-per-hour improvement this platform gets all quarter.
So Is Dave Right About the Cost?
Partly, and Anthropic’s own numbers make his case for him. Their adversarial review step (a second AI agent challenging the first one’s answer) bought a 6% accuracy gain for 32% more tokens and 72% higher latency. Their 95% accuracy did not come from the model; it came from curated reference docs, evaluation suites, and skill files that humans built and maintain. Strip the scaffolding and their accuracy was 21%. The LLM is the cheapest line item in that stack; the discipline around it is the real cost.
And that scaffolding is not a one-off purchase. To run generative BI at the scale Anthropic describes, genuine org-wide self-service, you need the canonical datasets, the semantic layer, the curated reference documentation, the evaluation suites, and the people to keep all of it current as the business changes. The upfront investment is huge, the ongoing maintenance is real, and the payback arrives slowly: the costs land now while the value takes time to realise. Read honestly, Anthropic’s article confirms this rather than refutes it. They could afford to build all of it at once because they are Anthropic. For any other organisation looking at this undertaking, the way in is not one sweeping build: start with a handful of canonical datasets, prove the value on the questions that matter most, and grow the layers step by step. That is Anthropic’s own getting-started advice buried at the end of their article, and it is the only version of this that survives contact with a real budget.
Where I land after this exercise is that Dave’s framing and Anthropic’s are answering different questions:
- For prediction, Dave wins on ROI. Forecasting revenue or scoring risk is a job for traditional ML: cheap to run, deterministic, evaluable, retrainable on a schedule. Running an LLM for that would be paying language-model prices for arithmetic. The next phase on this platform is exactly that, classic ML models built directly on the governed silver layer.
- For the analyst workflow, the LLM earns its keep. The ad-hoc questions, the investigation, the “why did this number move” work that used to consume analyst days. That is language and judgement work, and no gradient-boosted tree does it.
But here is a question worth sitting with: even where the LLM earns its keep, this is not cheap, and the spend curve is pointing the wrong way. Not because per-token prices are rising; the opposite, the major providers are in a price war. The catch is that agentic workflows consume multiples of the tokens anyone budgeted for, so total bills are climbing even as unit prices fall. Uber’s CTO recently described burning the company’s entire 2026 AI budget in four months as coding-agent adoption spread, at $500 to $2,000 per engineer per month. At that cost profile, an AI analyst is viable for a handful of power users, not the entire organisation. Whether the economics ever support genuine org-wide self-service, the way Anthropic runs it internally, or whether this remains a specialist’s tool is the sharpest version of the question Dave’s argument raises.
The part nobody should miss: both paths run on the same foundation. Every layer in Anthropic’s playbook (canonical tables, a semantic layer, written-down business rules, evaluation against known answers) is exactly the groundwork a warehouse needs before it can predict anything, by any method. The scorecard showed precisely which foundations were ready for the ML phase and which needed shoring up first.
You cannot forecast reliably from data you cannot even report on reliably.
Score Your Own Setup
If you are putting AI anywhere near your analytics, Anthropic’s four layers make a genuinely useful audit checklist:
- Does every business concept have one canonical table, or forty plausible candidates?
- Do your pipelines, models, docs, and AI instructions live in one place, or scattered across tools?
- Is your metric logic defined once in a semantic layer, or re-derived in every query?
- Are your data gotchas written down and versioned, or living in someone’s head (or something’s memory)?
- Would you notice if a number silently drifted, or only when the board does?
A platform built deliberately around these principles scored 7 out of 10. Most organisations I meet would score 3. The good news is that every layer is buildable incrementally, and the foundations layer, the one that matters most, is classic data engineering discipline. No AI required to start.
One last thing, because the current hype cycle keeps promising otherwise: throwing AI at messy data does not fix the data. It just produces wrong answers faster, and with more confidence. Every layer in this post exists to earn the right to put AI anywhere near the numbers.
If you want to start untangling the mess, that is where I come in. Book a free consultation and we’ll find the first thread to pull.
A note on data privacy: all analysis described in this post is undertaken on a Claude Enterprise account, under which data is not used to train Claude’s models. Data governance is part of the platform design, not an afterthought.