Serverless vs long-running hosts for agent workloads

The workload is mostly waiting

An agent turn spends almost all of its elapsed time idle. The process issues a model call, blocks for seconds while tokens stream back, runs a tool for a few milliseconds, then calls the model again. CPU work is a rounding error next to the wait. That single property is what makes the serverless-versus-host question different for agents than it is for a web API, and it drives four of the five criteria below.

The two options under comparison: a function-as-a-service platform (AWS Lambda, Cloud Run, Cloudflare Workers, Vercel Functions) where the unit of deployment is an invocation, and a long-running host (a VM, a container that stays up, a process under a supervisor) where the unit is a machine that outlives any individual turn.

Billing: you pay for the model’s latency

Serverless platforms bill for wall-clock execution time. For an agent, that is the wrong shape: you are charged for the seconds your function sat blocked on a socket waiting for a model response, not for work done. A turn that computes for 200 ms and waits for 12 seconds is billed as 12 seconds of a provisioned memory slice. The billing meter is measuring the model provider’s latency and charging you for it.

A long-running host bills for the machine by the hour regardless of what the process is doing, so idle waiting is free at the margin. One host can hold hundreds of concurrently blocked agent turns — they are all just file descriptors — and the bill does not move. The cost model inverts: serverless cost scales with total turn duration, host cost scales with peak concurrency and how well you pack it.

The practical consequence is that the crossover point depends on utilisation, not on request count. A host that is busy a few hours a day is worse value than functions; a host that is busy most of the time is dramatically better, because you stop paying a premium for wait states. If you cannot tell which you are, that is a measurement problem before it is an architecture problem — see attributing agent cost and latency to the work that caused it.

The duration ceiling is a correctness problem, not a tuning knob

Every serverless platform enforces a maximum execution duration, and the enforcement is a kill. An agent turn that exceeds it is terminated mid-run with no partial result: no final message, no tool results written back, no chance to checkpoint on the way out. The published ceilings and how they are configured are documented per platform — AWS’s Lambda quotas page and Cloud Run’s request timeout configuration are the two to read before committing — and both are subject to change, so treat the current number as something to look up rather than something to remember.

What matters more than the specific ceiling is that agent turn duration has a long tail you do not control. A turn that usually takes 40 seconds takes six minutes when the model decides to run twelve tool calls, or when the provider is degraded and every call retries. The ceiling turns that tail into a cliff. Under a host there is no cliff — a slow turn is just slow, and you decide when to give up.

This is the criterion that most often forces the decision, and it is also the one with a real escape hatch: if you break the agent loop into steps that each fit comfortably under the ceiling and persist state between them, the duration limit stops binding. That is exactly what durable execution for long-running agent workflows buys you, at the cost of writing the loop as a resumable state machine rather than as ordinary code.

Warm state and connection pools

A long-running host keeps warm process state and a connection pool across turns. The HTTP connections to the model provider stay open, TLS sessions stay established, the prompt cache or embedding index you loaded at boot is still in memory, and the database pool is already sized. A function invocation has to rebuild all of that or externalise it — a connection proxy, a cache service, a state store — each of which is another dependency, another failure mode, and another hop of latency on the critical path.

Externalising is not a defeat; it is a real design with real benefits, chiefly that any instance can serve any turn. But it converts in-process reads into network reads, and for an agent that touches its working state repeatedly within a turn, those add up. The honest summary: hosts get this for free, functions get it by paying for it somewhere else.

Cold start under a bursty access pattern

Cold start matters far more under an agent’s access pattern than under a web request pattern. Web traffic arrives as a broad, fairly smooth stream, so a platform keeps instances warm and cold starts land on a small fraction of requests. An agent’s calls arrive in bursts separated by idle gaps: nothing for twenty minutes, then a fan-out of parallel tool calls or sub-agent turns, then nothing again. Both halves of that pattern hurt. The idle gaps let instances be reclaimed, so the burst arrives at a cold pool; and because the burst is wide, the cold start is paid many times over simultaneously rather than once.

A host has no equivalent — the process was already running, and the burst costs only scheduling. You pay for that availability continuously instead of paying for it in latency at the worst moment.

Concurrency: throttling versus resource exhaustion

Concurrency limits are per-account on serverless platforms and per-machine on hosts, so fan-out fails differently in each. On serverless, exceeding the account concurrency limit produces throttling: invocations are rejected or queued, and — this is the part that surprises people — the limit is shared with everything else in the account, so an agent fan-out can throttle an unrelated production service, or be throttled by one. On a host, the limit is whatever the machine has: file descriptors, memory, event-loop headroom. Exceeding it produces resource exhaustion, which degrades everything on that box at once and is usually less abrupt but harder to attribute.

Throttling is the more legible failure. It is explicit, it is per-invocation, and it maps cleanly onto a retry policy. Exhaustion is the more dangerous one because it arrives as general slowness before it arrives as errors. Either way, the fix is to bound fan-out at the point where you issue it rather than to discover the ceiling empirically — the same discipline needed for agent execution under provider rate limits and concurrency caps, and the reason AWS’s builders’ library argues for constant-work designs: a system that does the same amount of work regardless of load has no load-dependent failure mode to discover at 3am.

Standing on each criterion

Criterion	Serverless functions	Long-running host
Billing shape	Wall-clock, so you pay for model wait	Per machine-hour; idle waiting is free at the margin
Duration ceiling	Hard cap; overrun kills the turn with no partial result	None imposed by the platform
Warm state and pools	Rebuilt or externalised per invocation	Kept across turns
Cold start	Costly: bursts after idle gaps pay it many times at once	Not applicable
Concurrency limit	Per-account; fan-out fails as throttling, shared with other services	Per-machine; fan-out fails as resource exhaustion

Which to pick when

Pick a long-running host if your agent turns are open-ended in duration. If you cannot state a number that the 99th-percentile turn stays under, the duration ceiling will terminate real work with no partial result, and every other criterion also favours the host. This is the default for anything doing multi-step research, code editing, or long tool chains.

Pick a long-running host if the agent runs steadily. Once utilisation is high, paying per machine-hour beats paying wall-clock for time spent blocked on a model socket, and you get warm connections and no cold starts as a side effect rather than as a project.

Pick serverless if the work is short, bounded, and genuinely spiky. A single-shot classification, a routing decision, a webhook that kicks off one model call — these fit under any ceiling, and paying nothing between bursts beats paying for an idle box. Per-request routing logic of the kind described in routing requests between models is a good fit.

Pick serverless if isolation per invocation is the point. Running untrusted tool code benefits from a fresh, disposable execution context, and the cold start you were trying to avoid is the isolation boundary you wanted.

Pick serverless plus durable execution if you want both. Decompose the agent loop into steps that each finish well inside the ceiling, persist state between them, and the duration cliff and the warm-state problem both go away — at the cost of writing the loop as a resumable state machine and accepting per-account concurrency as your fan-out limit.

One caveat on the last option: it is the most engineering effort of the three, and it only pays back if you were going to need checkpointing and resumability anyway. If the only thing pushing you toward it is the duration ceiling, a host is the cheaper answer to that specific problem.