Background Jobs on the Modern Stack: Queues, Cron, and Long-Running Work

The four options

Managed queues (SQS, Cloud Tasks, QStash). Durable execution (Inngest, Trigger.dev, Temporal). Cron endpoints on public routes. Dedicated worker fleets for CPU-heavy work.

• Managed queues for fan-out
• Durable execution for workflows
• Cron for schedule triggers
• Dedicated workers for CPU-heavy

Reference architecture

A durable implementation separates four layers, and keeping them separate is what lets you swap any one of them later without a rebuild. The ingestion layer normalises inbound events — webhooks, form posts, inbox messages, database change feeds — into a single typed envelope. The context layer resolves that envelope against your systems of record so every downstream decision sees the same customer, account and history. The decision layer applies policy and, where appropriate, model reasoning. The action layer writes back through the same APIs a human would use, so nothing bypasses your existing validation.

Between the decision and action layers sits the part most teams under-build: the control plane. Confidence thresholds, value caps, allow-lists of permitted tools, idempotency keys, and a full audit record for every attempt including the ones that were blocked. Without it you cannot answer the two questions leadership will ask in month two — what did it do, and what stopped it doing something worse.

We build these systems on boring, well-understood infrastructure: a typed API surface, a Postgres system of record with row-level security, queue-backed workers for anything long-running, and structured event logs streamed into whatever observability stack you already pay for. The interesting part of background jobs should be the domain logic, not the plumbing. Our engineering team and our product partners at Evron Studio use the same stack across every build for exactly this reason.

• Ingestion — normalise every trigger into one typed envelope
• Context — resolve identity and history before deciding
• Decision — policy first, model reasoning second
• Control plane — thresholds, caps, allow-lists, idempotency
• Action — write back through existing validated APIs

A realistic rollout sequence

Week one and two are discovery and instrumentation. You are not building yet — you are shadowing the people who do the work today, recording how they actually decide, and wiring the measurement so you can prove change later. Skipping this is the single most common reason background jobs projects stall: without a baseline, every subsequent conversation about impact becomes an argument about anecdotes.

Weeks three and four produce a suggest-only version. The system runs on live traffic, produces its recommended handling, and a human accepts, edits or rejects it. Every one of those judgements becomes a labelled example. You are simultaneously delivering value — the operator's draft is written for them — and building the evaluation set you will need to justify autonomy.

Weeks five and six graduate the highest-confidence, lowest-blast-radius slice to autonomous handling behind a kill switch, with the remainder still routed for review. From there, expansion is a data-driven negotiation: each category earns autonomy when its accuracy on held-out cases clears the threshold you agreed in week one. Most teams reach 40–70% autonomous handling within a quarter, and the remaining cases stay human because they should.

• Weeks 1–2: shadow the process, instrument the baseline
• Weeks 3–4: suggest-only on live traffic, capture every correction
• Weeks 5–6: autonomous on the safest slice, behind a kill switch
• Quarter 2: expand category by category on measured accuracy

Measurement that survives a board review

Activity metrics — messages handled, tasks executed, hours "saved" — are the metrics of a project that is about to be cancelled. They rise regardless of whether the work was any good. Replace them with outcome metrics measured against a holdout: a slice of traffic deliberately handled the old way so you always have a live control group rather than a historical one.

The four numbers we hold ourselves to on background jobs engagements are resolution or completion rate without human touch, quality as judged by a blind human review of a weekly sample, cycle time from trigger to resolved, and cost per case fully loaded including inference and engineering amortisation. Report all four together. Any one of them in isolation can be gamed, and the combination cannot.

Publish the numbers weekly to a channel that includes sceptics. Programmes die in silence, not in criticism — and the fastest way to earn the budget for phase two is a four-week chart that a CFO can read without a translator. Our product and application development team ships this dashboard as a deliverable in week one, before the first line of production logic is written.

• Autonomous resolution rate against a live holdout
• Blind quality score on a weekly random sample
• Cycle time, median and 90th percentile
• Fully loaded cost per case, including inference
• Escalation reasons, grouped and trended

Governance, risk and the things auditors ask

Assume from day one that someone will ask you to reconstruct a specific decision from six months ago. That single requirement drives most of the design: immutable logs of every input, every retrieved source, every tool invocation with its arguments, every output, and the identity of any human who reviewed it. Retention should match your existing records policy, not a default someone picked in a console.

Access control is the second pillar. The system should hold the narrowest possible credentials, scoped per tool, rotated on a schedule, and never shared with a general-purpose account. Anything that moves money, changes entitlements, deletes records or communicates a legal position belongs behind an explicit human approval regardless of how confident the model is. Confidence is not authority.

Third, write down what the system is not allowed to say or do, and test it. A short adversarial suite run on every deployment — prompt injection attempts, out-of-scope requests, hostile inputs, edge-case identities — catches regressions that unit tests never will. This is standard practice on every build we ship, and it is the reason our clients pass procurement security reviews without a remediation round.

• Immutable, queryable audit trail per decision
• Least-privilege, per-tool, rotated credentials
• Mandatory human approval for money, entitlements and deletions
• Adversarial regression suite in the deployment pipeline
• Documented data retention aligned to existing policy

Cost model and total ownership

Budget in three buckets and you will not be surprised. Build is a one-off: discovery, integration work, evaluation harness, and the control plane. Run is monthly: inference or platform fees, infrastructure, and observability. Improve is the bucket teams forget — the standing allocation for prompt and policy maintenance, new categories, and responding to upstream API changes. A programme with no improve budget degrades quietly within two quarters.

On the run line, the largest controllable cost is almost never the headline model price. It is unnecessary context. Retrieving twelve documents when three would do, replaying full conversation history on every turn, and re-deciding cases that a cache could answer are the three habits that inflate bills by an order of magnitude. Caching, tiered routing to smaller models for classification, and tight retrieval budgets typically cut spend 60–80% with no measurable quality loss.

Compare against the honest alternative, not against zero. The counterfactual for background jobs is usually additional headcount, an outsourced team, or continued lost revenue from slow response — all of which carry their own ramp, management and quality costs. When you price it that way, the payback window on a well-scoped wedge is normally two to four months.

• Build: discovery, integration, evals, control plane
• Run: inference, infrastructure, observability
• Improve: standing budget for policy and coverage growth
• Optimise: caching, tiered routing, retrieval budgets
• Compare to headcount and lost revenue, not to zero

Frequently asked questions