Backpressure & Resilience

Backpressure is a capacity contract: when consumers cannot keep up, the system must slow admission, bound queued work, reject load, or deliberately shed it. An unbounded queue is not backpressure; it merely converts overload into memory growth, stale responses, and a later outage.

Model capacity, queues, and deadlines together

Measure the sustainable service rate before selecting a queue size. If a worker completes 500 requests per second and receives 700, a queue grows by 200 requests every second until something fails. Little's Law relates average concurrency to throughput and time in the system, so a larger queue also means older work. If a response is useless after 300 milliseconds, a ten-second queue is data loss disguised as reliability.

Set an end-to-end deadline at ingress and propagate the remaining budget across services. gRPC's deadline guidance explains why clients should not wait indefinitely and why downstream calls need the same time budget. Cancel work after the caller leaves; otherwise abandoned requests continue consuming scarce capacity.

Choose an explicit overload policy

Use bounded queues and define what happens when each one is full. Interactive APIs commonly reject new work with a retryable status because old queued work has already consumed most of its deadline. Telemetry pipelines may drop low-priority samples. Event processors may pause partition consumption. Batch systems can block producers. The right policy follows the consequence of delay or loss, not a framework default.

• Block or signal demand: best when the producer can actually slow down.
• Reject: preserves healthy latency and tells callers capacity is exhausted.
• Drop: appropriate only for explicitly lossy, prioritised data.
• Spill to durable storage: useful for valuable asynchronous work, with a bounded retention policy.

In asynchronous Rust, a bounded Tokio MPSC channel makes the limit visible: send().await participates in flow control, while try_send() lets the caller implement rejection or shedding. The channel size still needs a workload-derived justification.

Do not confuse retries with resilience

Immediate retries amplify overload. Retry only operations that are safe to repeat, cap attempts, honour the original deadline, and use exponential backoff with jitter. The AWS Builders' Library guidance shows how retries across several layers can multiply work. Select one layer to retry and include an idempotency key when duplicate effects would be harmful.

Circuit breakers stop calls to a dependency that is consistently failing; rate limits protect a resource before it saturates; bulkheads reserve capacity between workloads. These mechanisms complement backpressure but cannot replace it. A breaker with an unbounded local queue still fails under load.

Propagate demand across pipeline boundaries

Pull-based protocols naturally let consumers request more work. Push-based systems need an acknowledgement window, credit count, pause signal, or bounded broker partition. The Reactive Streams specification formalises asynchronous demand so a publisher sends no more elements than a subscriber requested. Across HTTP or message brokers, teams must design the equivalent contract explicitly.

Priorities require isolation. If health checks, control traffic, and account recovery share the same saturated worker pool as bulk exports, the service can become impossible to operate. Separate pools or reserved concurrency make the desired priority real.

Test overload as a state, not a spike

Monitor queue depth and age, admission and rejection rates, in-flight work, deadline exhaustion, downstream latency, retry volume, memory, and successful throughput. Run a sustained test above capacity, then reduce load and verify that latency and queue age recover promptly. Also stop a dependency, slow one consumer, and cancel callers mid-request. A resilient design has a known, observable degradation mode and a bounded recovery time.