Skip to content
←System Design

// System Design · 06

Fault Tolerance

A component failsleads toSystem keeps running

Designed to survive failure, not just avoid it.

How it fits together

Fault Tolerance structureCheckout APItimeout 2sCircuit breakeropens at 50% failureshalf-open after 30sretry x3, backoff + jitterSupplier API503 or timeoutFallback cache1 call2 call times out3 open: serve
uses / callsfails / error path· numbered steps replay in order
  • Checkout API uses / calls Circuit breaker (1 call)
  • Circuit breaker fails / error path Supplier API (2 call times out)
  • Circuit breaker uses / calls Fallback cache (3 open: serve)

Spot it

  • One slow dependency stalls every request
  • A third-party outage becomes your outage
  • Retries make an incident worse

It costs

Every fallback is a second code path that must be tested and can serve stale data.

In code

services    .AddHttpClient<ISupplierClient, SupplierClient>(client =>    {        client.BaseAddress = new Uri("https://supplier.example.com");    })    .AddStandardResilienceHandler(options =>    {        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);​        options.Retry.MaxRetryAttempts = 3;        options.Retry.BackoffType = DelayBackoffType.Exponential;        options.Retry.UseJitter = true;​        options.CircuitBreaker.FailureRatio = 0.5;        options.CircuitBreaker.MinimumThroughput = 10;        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);        options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);    });
Go deeperThe full explanation and what to ask for · 1 min readExpand

Assume every network call can hang, fail or lie. A timeout is the minimum; without one, a slow dependency borrows your threads until you fall over too.

Retries without backoff and jitter synchronise clients into a thundering herd. A circuit breaker stops you hammering something that is already down.

Bulkheads isolate resources per dependency so one bad neighbour cannot drain the shared pool. Graceful degradation decides what the user sees when part of the system is gone.

Ask for it

  • timeouts
  • retry with backoff and jitter
  • circuit breaker
  • bulkhead
  • graceful degradation
  • fallback

Wrap calls to the supplier API with a 2s per-attempt timeout, 3 retries with exponential backoff and jitter, and a circuit breaker that opens when half of the calls fail within 30s and stays open for 30s. Serve cached results as a fallback while open.

Connects to