A five-person team tries out a new AI agent, an AI assistant that carries out several steps in a row on its own. They ask it to go through twenty competitor announcements and write the key points into an internal doc. The model’s price looks great: two dollars per million input tokens, the unit models use to measure text and charge for it. A week later, the bill is several times what they estimated. Nobody misused it, and the task ran only a dozen or so times.

Where did the money go? Mostly not into the model reading those twenty announcements. It went into two kinds of repetition that long tasks produce: resending the entire context at every step, and starting over after a failure. Neither shows up on a price sheet, yet together they decide what a long task costs. That’s why a long task’s limit can’t live only in the prompt. It has to be enforced by code outside the model, as a single budget shared by the whole task.

Prices fall, but the bill doesn’t follow

In July, SpaceXAI, the Elon Musk company formerly known as xAI, released its new model Grok 4.5 at $2 per million input tokens and $6 per million output tokens. TechCrunch compared it with Anthropic’s flagship Opus 4.7, priced at $5 and $25. SpaceXAI also stressed that the model is more token-efficient, using fewer output tokens than rivals on the same kind of coding task.

Numbers like these make many long tasks that used to look too expensive seem worth a try: digesting a large pile of material, changing code across many files, reading a batch of support tickets and sorting them. The catch is that a price sheet answers “how much per million tokens read,” while a long task’s bill depends on “how many times everything was read.”

Where the money goes: every call rereads the whole context

A long task is work an agent can only finish by taking many steps in a row: read a file, call a tool, look at the result, decide what to do next. Anthropic’s documentation puts it plainly when describing agents that use tools: each step typically requires a new API call, meaning one new request from your program to the model. The model doesn’t remember the previous call, so every call has to send everything so far again: the conversation, the files it has read, the results tools returned.

So cost doesn’t grow evenly with the number of steps. Take a hypothetical example. If each step adds 2,000 tokens of context, step 1 sends 2,000 tokens, step 10 sends 20,000, and by step 30 a single call sends 60,000. Add up all thirty steps and the model has read roughly 930,000 input tokens, of which only 60,000 were actually new.

Vendors offer caching to soften this repetition. Using Anthropic as an example, tokens read from the cache cost one tenth of the base input price, while writing to the cache costs 25% more. But the cache lasts only five minutes by default, and the timer resets each time it’s used. That detail matters in practice. If the agent pauses for someone to confirm a step and that person spends twenty minutes in a meeting, the cache has expired by the time they’re back, and the next step has to write the whole context again.

Advertisement

The second cost: starting over after a failure

The second kind of repetition is the rerun. A long task fails at step 25 because one file has an unexpected format, and the usual response is to start from scratch. Every input accumulated over the first 24 steps gets paid for again. If you’ve set “retry twice on failure,” the worst case is paying three times for the same work and failing all three times.

The trouble with this cost is that you can’t see it. The bill shows how many tokens were used, not how many went to redoing work. The team sees “this month feels a bit expensive” and can’t tell which kind of task burned the money, or at which step.

Why a limit written in the prompt doesn’t count

Many teams’ first instinct is to add a line to the agent’s system prompt: “Stop if spending exceeds $5.” The problem is that the model can’t see the bill. It doesn’t know how many tokens it has used so far or what the current call costs. At best it can remind itself to be frugal. It can’t actually stop when the budget runs out.

Only the layer of code that calls the model can enforce a limit reliably. After every call, the API reports how many input and output tokens it used. Your code adds them up and checks the remaining budget before sending the next call. If there isn’t enough left, it doesn’t send the call. It hands the current results, and the reason for stopping, to the person responsible.

That budget should cover the whole task. A retry doesn’t get a fresh allowance; it shares the same money as the first attempt. Otherwise “retry at most twice” quietly triples the limit.

What a hard limit costs

A hard limit has a cost of its own. If a task hits the limit when it’s 90% done, everything spent so far buys a half-finished result. To reduce that waste, pair the limit with checkpoints along the way: each time a stage is finished, save its cleaned-up intermediate result, and on a rerun resume from the last checkpoint instead of rereading from the start. A failure then costs one stage, not the whole task.

Also, not every task needs this machinery. A short task of three to five steps builds up little context and is cheap to rerun, so designing budget controls for it isn’t worth the effort. Limits are worth setting for work with many steps, work that reads lots of files, and work that may retry automatically.

Measure for a week, then set the limit

There’s no universal answer to how high the limit should be. Tasks differ in step count, context size, and failure rate. Borrow someone else’s numbers and you’ll either set it so tight that tasks keep stopping halfway, or so loose that it might as well not exist.

A more practical approach is to pick a long task your team already runs and track it for a week: how many steps each run took, how many tokens each step sent, how many retries happened, and how often the cache was hit. After a week you’ll know roughly what a normal run of this task costs. Set the limit a little above that. The runs that exceed it are exactly the anomalies worth a person’s attention.

The same record tells you where to save. If resent context makes up most of the cost, first check that caching is actually working and that the context isn’t stuffed with whole files nobody needs. If the cache keeps expiring while the agent waits for confirmation, group confirmations into fewer stages or consider a longer cache lifetime. With Anthropic, for example, a one-hour cache write costs twice the base input price, which can still beat rewriting the context over and over on tasks with long waits. If retries make up most of the cost, fix the cause of the failures; raising the limit only pays for the same failure more times.

This article is about the budget for a single task. How a whole team should divide its monthly AI spending, and who can use how much, is a separate question; see Measure an AI Bill by the Cost of Each Accepted Result, Not the Price per Million Tokens.

If the long task is coding, check whether it can be split up before you hand it over; see Can a Coding Agent Take This Issue? Only If You Can Describe the First Diff. The smaller the pieces, the shorter each one’s context, and the less there is to redo when something fails.

AI handoff card

Ask AI to estimate what a long task really costs This prompt first asks how the task runs, then works out how much goes to resent context and how much to retries, and finally suggests a first budget limit and where to save checkpoints.

I want to estimate what a long AI task really costs and set a first budget limit for it.
Background comes from this article: https://boosterminiclass.com/en/posts/long-running-ai-task-token-cost-limits/
Its view: a long task's bill comes mainly from resending context at every step and rerunning after failures, so the limit must be enforced by the code that calls the model, as one budget shared by the whole task, not just written in the prompt.

Ask me 3 questions first:
1. How many steps does this task take on average, and roughly how much new content does each step read in?
2. What happens on failure today: a full rerun, resuming from the middle, or a set number of automatic retries?
3. Does the task pause midway for a person to confirm something, and how long does that usually take?

Based on my answers, give me:
- A simple calculation estimating the total input tokens for one complete run, marking how much of it is resent context;
- How much one failed run and rerun adds;
- A suggested budget limit for the whole task, and at which stages to save checkpoints;
- Which numbers I should record over the next week to adjust this limit.
State every assumption behind your estimates. If information is missing, ask me instead of filling in numbers yourself.

The estimate is only a starting point. The real limit should be based on a week of actual records.

Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References