Call Ceilings for AI Workloads: What They Do and What They Don't

An AI workload rarely consumes a predictable number of tokens per request. A short classification call and a long generation can have very different cost and resource profiles. But they do share one simple unit: each attempt to call an upstream API is an event an operator can count.

That makes a call ceiling useful. It does not make a credential safe by itself.

The distinction matters because several controls that sound similar answer different questions:

• A rate limit controls how quickly requests can arrive.
• A call ceiling controls how many attempts a credential can make over its life.
• A token or spend limit bounds a different resource dimension.
• An expiry bounds time.
• A provider permission or project policy determines what the upstream credential is authorized to do.

OWASP's guidance on unrestricted resource consumption recommends multiple resource controls, including timeouts, payload bounds, rate limits, operation limits, and provider spending limits. A lifetime call ceiling complements those controls; it does not replace them.

Start with the workload, not a round number

Suppose a batch job is expected to process 120 documents, with at most two model calls per document and a small allowance for retries. A ceiling derived from that work plan is more meaningful than an arbitrary “500 calls” default.

A practical estimate is:

expected units × maximum calls per unit + explicit retry allowance

The estimate should be conservative enough to avoid routine interruption but low enough to be useful if the credential escapes its intended workload. When the work is open-ended, use a shorter review interval or issue a replacement credential rather than assigning a very large ceiling and forgetting it.

Define what counts before relying on the number

Counting policy is part of the security contract. In Till, an activation represents an upstream attempt. A request rejected locally—for example, because authentication fails or a configured budget cannot safely cover it—is refunded. Once Till dispatches the request upstream, the activation counts even if the provider or network later fails.

That policy prevents a failing upstream from creating an unlimited retry path. It also means clients should use bounded retries and treat ambiguous failures carefully. A retry is another upstream attempt and can create a duplicate operation when the first request actually reached the provider.

Call ceilings are not concurrency controls. Ten parallel requests can consume ten activations at once. They are also not a substitute for request-size limits: one activation can still represent a large, expensive generation. Pair them with token and spend ceilings when those dimensions matter.

Remember that the scoped token is still a bearer secret

A Till token is easier to revoke independently than a provider credential shared by many workloads, but possession is still sufficient to use it. The security considerations in RFC 6750 are a useful model even though Till tokens are not OAuth tokens: bearer credentials must be protected in storage and transit, kept out of URLs and logs, and limited in scope and lifetime where possible.

That leads to a straightforward handling policy:

1. Deliver the token once through a protected channel.
2. Store it in a secrets mechanism appropriate to the workload, not in source code.
3. Avoid printing authorization headers in logs or error reports.
4. Revoke the token when the job ends or compromise is suspected.
5. Keep the upstream provider credential subject to the provider's own controls.

Google Cloud's API key management guidance recommends restrictions, isolation, monitoring, deletion of unused keys, and rotation. Those provider-side measures remain valuable when a proxy token sits in front of the provider key.

Use layered bounds

For a bounded agent task, a defensible setup might combine:

• a per-workload Till token with a documented call ceiling;
• an expiry aligned to the expected job window;
• a token or spend ceiling where model pricing and output bounds make enforcement reliable;
• an IP allowlist where the workload has stable egress;
• provider-side project separation, billing alerts, and supported permission restrictions;
• monitoring for unexpected volume, route choice, or source address;
• a tested revocation procedure.

No single line item covers every failure mode. A call ceiling limits the number of upstream attempts available to a stolen token. It does not prove who made those attempts, prevent disclosure, guarantee a maximum dollar cost, or reduce the authority of the underlying provider credential.

That is the right way to evaluate activation limits: as a simple, inspectable bound on one dimension of exposure. Use them because the workload has a countable shape—not because counting calls makes the rest of credential security unnecessary.

← Back to blog