This project grew organically out of a mandate focused on building an AI-driven alpha-seeking research operation within a client hedge fund.
The client was running roughly $24 million a year through LLMs and didn’t feel like they had the cost under control. MoM spend growth was intense. Their researchers benchmark models obsessively, they ship their own evals, and the prevailing instinct, that compute is cheap relative to alpha, is not questioned. They were also, it turned out, carrying about a third of their LLM spend as pure waste that produced no research and no alpha.
My team started on this side-project in early May and wrapped up in little over a week. The result was a $8.7 million saving on the yearly bill, with no degradation in research-output quality on a single workload. The fund ended up running more than twice the effective research per dollar. This article is the engineering of that, showing the dials and how each one was turned, measured against their own evals, and signed off. There are a lot of dials.
You cannot improve what you cannot see
Before any optimization we built the measurement substrate, because every number downstream depends on it and because its absence is the first and largest finding.
We reconciled the provider billing exports to the dollar against the invoices, then joined them to the gateway and observability logs, which carry what billing does not: per-request model, input and output token counts, cached-token counts, latency, retry flags, and tags for research team, strategy, and environment. The result is a single spend cube, one row per team, application, model, environment, and day, with columns for requests, input and output tokens, cost, cache-hit rate, retry rate, and average output length. Every lever query in the engagement runs off that cube.
The join produced the first finding immediately. Twenty-two percent of spend could not be attributed to a team or a research process at all. A fund that can attribute every basis point of P&L to a strategy could not attribute a fifth of its AI spend to a research purpose. In the scoring model this is the “visibility” lever. It scores low not because it saves much directly but because nothing else can be governed without it, and we closed the gap to under five percent before touching anything else.
We then scored the estate against the twelve canonical levers across four tiers. The fund came in at 26 out of 100, the band we came to call Tokenmaxxer: spend running with no one accountable for what it returns. For a fund this happens for a specific and almost admirable reason. Research leadership had told the team, correctly, that the worst outcome is a researcher throttled by cost while chasing a real edge. So everyone defaulted to the most capable model for everything, nobody owned the bill, and the waste accumulated in the gap between “compute should never block alpha” and “most of this compute was not producing alpha.”
Tier 0: stop the bleeding
You start with control, not with the biggest dollar lever, because savings you cannot trust or contain are not savings.
The looping research agent. The single most expensive thing we found was an agentic research loop stuck retrying a failing data-fetch step. It had run for weeks at a roughly seven-figure annualized rate, producing no signal and invisible because it looked like ordinary research traffic. The agentic tier ran an 11 percent retry rate overall. The fixes are unglamorous and decisive: exponential backoff with jitter, hard per-run token ceilings, max-retry caps, and circuit breakers that halt an agent that is plainly not converging. We also found two research pilots still billing daily that had been quietly abandoned, and development and backtest traffic running on production keys at production prices. Removing the dead weight and separating non-prod returned $1.5 million, verified as frozen run-rate eliminations because they are one-time removals that will not regrow with volume, each guarded by a persistence window so a temporary dip is never booked as a permanent win.
The caps. There was a gateway-level spend cap but no per-key or per-agent token budgets, which is how one research agent can run an unbounded bill before anyone notices. We instrumented enforced budgets at the key and agent level with spike alerts wired to a named owner. Modest dollars, but it is what ensures the next looping agent trips a wire in an hour rather than a quarter.
Tier 1: right-size
This is where most of the money is, and where a fund’s standards bite hardest, because a fund will reject any change that costs it research quality.
Routing, the heaviest lever, $3.4 million. Fifty-eight percent of the fund’s token volume ran on the top, reasoning-grade model tier regardless of the task. A great deal of it was extraction, pre-screening, classification, and short summarization, work that a model a fraction of the price handles at identical quality. The price ladder makes the prize plain: within the approved set the spread from the reasoning tier to the small tier is roughly five to one, and across the market it runs from a few cents to well over a hundred dollars per million tokens. Defaulting everything to the top of that ladder is the most expensive habit in enterprise AI.
For this client the eval gate was the entire condition of the work. Each workload has a golden set and a research-quality floor, defined with the fund’s own researchers, and a cheaper model is promoted only if it clears that floor offline and then holds it in a live A/B holdout on real traffic. Two constraints shaped the lever. First, security: the fund’s prompts, data, and signals are its alpha, so inference is confined to a tightly vetted set of approved, zero-retention, no-train models running in their own environment, and vetting a new model is slow because the bar is so high. That set the ceiling on routing, and we routed hard within it while opening the vetting of one additional small model to widen the lever later, which we did not book because it was not yet realized. Second, the tail: on workloads with a long tail of difficulty we used a cascade, serving a cheap model first, measuring confidence, and escalating only the hard cases to a stronger model. That captures most of the saving while protecting quality on exactly the requests that need it. The result was $3.4 million with the eval gate and the holdout standing behind every promoted route, and not one research workload dropped below its quality floor.
Caching, $1.3 million. Prompt caching is the highest-return mechanical change available, and the economics are stark: writing a cache entry costs a small premium over base input tokens, roughly a quarter more, and reading it back costs about a tenth of the base price. Anything resent more than a couple of times pays for itself many times over. The fund’s cache-hit rate was 6 percent on workloads with a fixed system prompt. They were paying full freight to resend the same instructions and the same retrieved context on nearly every call. We cached the static research prompts, the standing context block that prefaced the signal-extraction calls, and the retrieved context on the document-extraction and internal-research workloads. The detail that separates a practitioner from a deck: cacheable content must sit at the prompt prefix, so we restructured prompts to put the stable material first and the variable query last, maximizing the cacheable prefix and the hit rate.
Batch, $0.7 million. The Batch API discounts asynchronous work by about half. The dial is an eligibility classifier: any workload with no real-time requirement is a candidate. The overnight backtest summarization and the scheduled evaluation suites both qualified and had been running on the synchronous, full-price API. Moving them captured roughly half their cost, and a research team does not care whether last night’s batch finished at 2am or 4am.
Arbitrage, $0.3 million. A pricing-model mismatch, found by a crossover analysis: heavy users of the researcher coding assistant whose metered bills exceeded a flat seat were moved to seats, and a pool of idle paid seats was reclaimed.
Tier 2: engineer the waste out
These compound across millions of calls.
Hygiene, $0.6 million. Output length on the summarization workloads was uncapped and running about three times longer than the task required, which matters because output tokens are the expensive side of the ledger. We set per-task output caps, pruned redundant few-shot examples now duplicated by the cache, and trimmed oversized retrieved context. None of it is clever but multiplied across the fund’s call volume it is real money.
Distillation, $0.5 million. One workload was the textbook distillation candidate: a high-volume signal pre-screening and document-extraction task that ran every filing and transcript through a general frontier model to decide what merited deeper analysis. The pipeline is the technical centerpiece of this tier. We collected labeled traces from the frontier model, fine-tuned a small approved model on them, evaluated it against a held-out golden set to a quality floor the researchers set, vetted it through the fund’s security and model process, and deployed it behind the same eval gate as the routing work. Cost per screened item fell sharply, and because the pre-screen is now cheap, the fund screens more, which is the effectiveness story in miniature. Highest rate of return of any lever, narrowest applicability, and a project rather than a quick win, which is why most funds never reach it without help.
Semantic caching, $0.15 million. Repeated near-identical research queries, the same question about the same filing asked many ways, were collapsed with a semantic cache that serves a stored answer when a new query’s embedding is within a similarity threshold of a previous one.
Tier 3: the strategic levers, and where the effectiveness came from
The first two tiers cut the cost of doing the work. The third decides whether the work should be done at all, and for a fund it is the part that matters most.
Triage. We did, for AI spend, what a fund already does for every strategy: tie it to a measured return. Working with the research leads we asked of each workload whether it fed a live signal or a strategy that reached production. Two research pilots had never produced anything that informed a live position and were cut. Small direct dollars, $0.2 million, but this is the lever that did the heavy lifting on effectiveness. Before the engagement, only about 22 percent of AI spend was tied to research that reached a live signal; the rest was exploration that had quietly become permanent, or pilots no one had the heart or the time to kill. Afterward, with the dead exploration cut and the surviving spend concentrated on what reached production, that figure was roughly 80 percent. Combined with a cost base a third lower, effective research per AI dollar more than doubled. On the flagship workload, evaluating a candidate signal end to end, cost per evaluated hypothesis fell more than 40 percent at held quality, so the team backtested more than twice as many ideas for the same money. For an alpha shop that is a research-capacity metric, and research capacity is the input to alpha.
FinOps for AI. Savings creep back because no one owns them. We stood up a standing FinOps-for-AI function: a named owner, a monthly cadence, anomaly detection on the spend cube, and per-team budgets, so a researcher is never throttled but a runaway agent is caught within the hour. This is the durability layer, and the natural anchor for an ongoing relationship.
RoI
The measure the research desk agreed to track with us was cost per validated signal: the all-in inference cost to carry one hypothesis from idea, through backtest and evaluation, to a signal that clears the fund’s eval bar and is cleared for a live book.
It is computed straight off the pipeline telemetry. Every inference call is tagged to the hypothesis it serves, so the numerator is the total inference cost spent across the whole funnel in a period, including the calls that led nowhere, and the denominator is the number of signals that reached the validated stage in that period. The dead ends stay in the numerator deliberately. A metric that counted only the cost of the signals that worked would hide most of the waste, which is exactly the spend we were there to find.
Over the engagement, cost per validated signal fell by about 57%. Two forces drove it. The pipeline that produces a signal got cheaper at every stage, through routing, caching, and the distilled pre-screen. And the compute that used to die in dead ends, looping agents, and abandoned pilots left the denominator, because we switched those off.
For an alpha shop this is the figure that matters more than the bill. Validated signals are the raw material of strategies, and the rate at which a desk can manufacture them sets a ceiling on how much alpha it can pursue.
How a fund lets you book the number
None of this matters to a COO / Head of Research without the part that makes it real and defensible, and a quant fund is the most demanding possible auditor of a savings claim.
The baseline is co-defined in writing on the fund’s own usage data before any optimization, then frozen, with one-off anomalies normalized out. Savings are measured three ways, chosen per workstream: unit-cost reduction as the default, frozen run-rate for the one-time eliminations like the zombie agents, and the A/B holdout, the gold standard, for the quality-sensitive levers like routing and distillation, where a concurrent control on comparable traffic measures the counterfactual directly rather than arguing it. A fund understands a holdout better than anyone; it is how they evaluate their own signals.
Every saving is gated by a quality floor, and for this client the floor is sacred: the research-output quality metrics, the eval pass rates and signal-quality measures defined and owned by the researchers, must hold or improve, or the affected saving does not count. This is the crux. The reason a fund tolerates a cost engagement at all is the guarantee that it will never trade research quality for a lower bill, and the holdout plus the eval gate is what makes that guarantee checkable rather than promised. Then the carve-outs, agreed up front, exclude what we did not cause: provider price cuts accrue to the fund, organic volume changes neither create nor destroy a fee, new use cases after the baseline are out of scope, and changes the fund makes independently are excluded or attributed through the holdout. Each month the fund’s finance and research leads review the measured savings, computed reproducibly on their own data, and sign off before any fee is invoiced. We never touch the ledger. Each increment of verified saving is credited for twelve months and then rolls into the baseline at no further fee.
We were paid out of that verified number, a fixed fee for the audit and a minority share of the savings the fund itself confirmed, which left them with enough to redeploy the way they saw fit.
Where the fund ended up: Token Efficiency Score moved 26 → 71, out of the Tokenmaxxer band into Disciplined, with headroom still left toward Yield-optimised.
What this means
None of these layers is a secret. Routing, caching, batching, distillation, the rest, these are published, vendor-neutral practice. The “moat” is purely in being able to execute against a fund’s real constraints, the verification rigor that lets CXOs sign, the absolute respect for the confidentiality of the alpha, and the benchmark that sharpens with every engagement.
The waste in a sophisticated AI estate is almost always large, almost always buys no alpha, and can be removed without taking away a single thing the research actually depends on. But implications go beyond the financial sector. The first phase of enterprise AI in a lot of places was tokenmaxxing, spend as a proxy for progress. The phase that separates the businesses that compound from the ones that bleed capacity is valuemaxxing: every token should buy something useful, or it should not be spent.
If you run a serious AI research operation and cannot say cleanly what it returns, it’s time for a rethink. Two pieces of good news. First, you may be in the most common position at the frontier right now. Second, you live exactly where the largest, fastest, and most defensible gains are to be had.
Be in touch if this is interesting.
Were these issues tied to internal systems or when the agents "left" and tried to read research on third party sites?