Retry Helpers

Retry (sillo.helpers.retry)

Some failures are permanent. A 404 will still be a 404 in five seconds. Some failures are not: a connection reset, a DNS blip, a database failover, a rate limit that expires, a service restarting behind a load balancer. Those succeed if you ask again.

Retrying the second kind converts a user-visible error into a slightly slower success. Retrying the first kind wastes time and, if the operation had side effects, causes damage. This module gives you the mechanism; deciding which failures qualify is on you, and it is the part that matters.

from sillo.helpers import retry

Stdlib-only.

Exponential backoff, and why the delay grows

Retrying immediately is usually worse than not retrying. A service that just failed is often overloaded, and hammering it with instant retries from every client is how a brief blip becomes a sustained outage.

Backoff means waiting longer after each failure. With the defaults, delays double:

attempt 1 fails -> wait  2s
attempt 2 fails -> wait  4s
attempt 3 fails -> wait  8s
attempt 4 fails -> wait 16s
attempt 5 fails -> wait 32s
attempt 6 fails -> wait 60s   (capped by max_delay)

The cap matters. Without it, doubling reaches hours, and a background job sleeps past any deadline you cared about.

Jitter randomizes each delay. It is on by default and you should leave it on. Without jitter, a thousand clients that fail at the same instant all retry at exactly the same instant, then again two seconds later, and again four seconds later. The service gets synchronized waves of load precisely when it is trying to recover. Randomizing spreads them out:

_compute_delay(3, base=1.0, factor=2.0, cap=10.0, jitter=True)
# 4.98, 6.98, 2.49, 7.83, 0.85 ...  different every call, never above the cap

The decorator

@retry(max_attempts=3, base_delay=1.0, max_delay=60.0, backoff_factor=2.0, jitter=True, retryable_exceptions=Exception)

Wraps a function so every call retries on failure. Works on both sync and async functions — it inspects the target at decoration time and returns the matching wrapper:

from sillo.helpers.retry import retry

@retry(max_attempts=5, base_delay=1.0)
async def fetch_rates():
    return await http_client.get("https://api.example.com/rates")

@retry(max_attempts=3)
def read_config():
    return open("/mnt/shared/config.json").read()

A call that eventually succeeds returns normally, and the caller cannot tell it took three attempts. A call that exhausts every attempt raises RetryError:

from sillo.helpers.retry import RetryError

try:
    rates = await fetch_rates()
except RetryError as exc:
    logger.error("rate service unreachable: %s", exc)
    # 'Retry failed after 5 attempts'
    rates = cached_rates()

The original exception is wrapped, not re-raised

This is the detail that surprises people. After the last attempt you get RetryError, not the ConnectionError that actually happened:

@retry(max_attempts=3, base_delay=0)
def always_fails():
    raise ConnectionError("connection refused")

always_fails()   # raises RetryError, not ConnectionError

Existing except ConnectionError: handlers upstream stop catching it the moment you add the decorator. Audit callers when you apply @retry to an existing function.

The upside is that RetryError distinguishes “failed once” from “failed persistently after N attempts”, which is a genuinely different operational signal. If you need the underlying error, catch it inside the retried function and re-raise something you control.

Choosing what to retry

retryable_exceptions narrows which failures trigger a retry. Everything else propagates immediately:

@retry(max_attempts=3, retryable_exceptions=(ConnectionError, TimeoutError))
async def call_api():
    ...

The default is Exception, which retries everything.

The default retries bugs in your own code

retryable_exceptions=Exception catches TypeError, KeyError, AttributeError, and ValueError alongside the network errors you meant.

A typo that raises AttributeError on every call now takes three attempts and fourteen seconds to surface, instead of failing instantly with a clean traceback. Under load, that turns one broken deployment into a thread pool full of sleeping retries.

Always name the exceptions you actually expect:

# Good — only transient transport failures retry.
@retry(max_attempts=3, retryable_exceptions=(ConnectionError, TimeoutError, OSError))
async def call_api(): ...

# Risky — a KeyError in your parsing code retries three times first.
@retry(max_attempts=3)
async def call_api(): ...

For HTTP work, the status code matters more than the exception type. Retry 429, 502, 503, and 504. Do not retry 400, 401, 403, 404, or 422 — the request is wrong and repeating it verbatim cannot help.

One-shot forms

When you cannot decorate the function — it belongs to a library, or the retry policy varies per call site — use the functional forms.

sync_retry(func, *args, **kwargs)

from sillo.helpers.retry import sync_retry

result = sync_retry(
    requests.get,
    "https://api.example.com",
    max_attempts=3,
    base_delay=0.5,
    retryable_exceptions=(ConnectionError,),
)

async_retry(coro, *args, **kwargs)

from sillo.helpers.retry import async_retry

result = await async_retry(
    http_client.get,
    "https://api.example.com",
    max_attempts=3,
    base_delay=0.5,
)

Positional and keyword arguments pass straight through to the wrapped callable. The retry-control keywords (max_attempts, base_delay, max_delay, backoff_factor, jitter, retryable_exceptions) are consumed by the retry wrapper.

A name collision between your kwargs and the retry kwargs is silent

If the function you are retrying takes its own timeout or max_attempts argument, the retry wrapper claims the ones it recognises. There is no error; your value goes to the wrapper instead of your function.

Wrap the call in a lambda or functools.partial to keep the argument namespaces separate:

result = await async_retry(
    lambda: client.request(url, max_attempts=10),   # client's own parameter
    max_attempts=3,                                  # the retry policy
)

Idempotency: the rule that decides everything

Retrying is only safe when performing the operation twice has the same effect as performing it once. That property is called idempotency, and it is not a detail — it is the whole question.

The dangerous case is a request that succeeded on the server and failed on the way back. The server charged the card, then the connection dropped before the response arrived. Your client sees a ConnectionError, retries, and charges the card again.

Not safe to retry

@retry(max_attempts=3)
async def charge_customer(customer_id: str, amount: int):
    return await payments.charge(customer_id, amount)   # may charge three times

Safe to retry, because the key deduplicates server-side

from sillo.helpers.strings import random_token

async def charge_customer(customer_id: str, amount: int):
    key = random_token(16)   # generated ONCE, outside the retry

    @retry(max_attempts=3, retryable_exceptions=(ConnectionError, TimeoutError))
    async def attempt():
        return await payments.charge(customer_id, amount, idempotency_key=key)

    return await attempt()

The key must be created outside the retried function. Generating it inside means a fresh key per attempt, which is exactly the duplicate-charge bug with extra steps.

A rough guide:

Operation	Retry?
GET / read query	Yes
PUT with a full resource body	Yes, it is idempotent by definition
DELETE	Usually — the second one 404s, which you can treat as success
POST that creates something	Only with an idempotency key
Payment, email send, SMS	Only with an idempotency key
Database write inside a transaction	Retry the whole transaction, never one statement

Retrying in a request handler

The most common mistake with this module is putting it somewhere it makes the user experience worse.

Do not use long backoff inside a request handler

Default settings give up after roughly 60 seconds of sleeping. The client’s connection is held open the whole time, one of your workers is occupied, and the user is staring at a spinner. Most browsers and load balancers time out first, so the user gets an error and you paid the full cost.

Inside a request, keep the total retry budget under your own timeout — two attempts with a short delay at most:

@app.get("/dashboard")
async def dashboard(request, response):
    try:
        stats = await async_retry(
            fetch_stats,
            max_attempts=2,
            base_delay=0.1,
            max_delay=0.5,
            retryable_exceptions=(TimeoutError,),
        )
    except RetryError:
        stats = None   # degrade, do not hang
    return await response.template("dashboard.html", {"stats": stats})

Long, patient retries belong in Background Tasks and the Job Queue, where nobody is waiting on the socket. If an operation can afford a minute of retries, it can afford to be a job.

Where retrying makes an outage worse

Retries add load exactly when a system is least able to take it. Three specific failure modes to know about.

Retry amplification. With three services chained and three attempts each, one user request becomes up to 27 calls at the bottom of the stack. A partial failure at the leaf multiplies into a load spike that guarantees a total one. Retry at one layer, ideally the outermost, and let inner layers fail fast.

The synchronized herd. Covered above, and the reason jitter defaults to on. Never turn it off in production; the only reason to disable it is deterministic tests.

Retrying past the point of usefulness. A service that has been down for a minute is not coming back within your backoff window. Continuing to retry keeps your workers occupied and adds load to a system already in trouble. A circuit breaker — fail fast for a cooldown period after N consecutive failures, then probe with a single request — is the standard complement to retries, and this module does not provide one. For a dependency you call constantly, add one.

A retry budget helps too. Cap retries as a fraction of total requests (say 10%), so a broad failure cannot triple your outbound traffic no matter how many individual calls want to retry.

Observability

A silent retry is a hidden latency problem. An endpoint that “works” but succeeds on the third attempt has a p99 measured in seconds, and nothing in your metrics says why.

Log when retries happen, not just when they are exhausted:

Making retries visible

import logging
from sillo.helpers.retry import async_retry, RetryError

logger = logging.getLogger(__name__)

async def fetch_with_logging(url: str):
    attempts = 0

    async def attempt():
        nonlocal attempts
        attempts += 1
        return await http_client.get(url)

    try:
        result = await async_retry(attempt, max_attempts=3, base_delay=0.2)
        if attempts > 1:
            logger.warning("succeeded on attempt %d for %s", attempts, url)
        return result
    except RetryError:
        logger.error("exhausted %d attempts for %s", attempts, url)
        raise

Emit a counter for retries and a separate one for exhaustions. Retries climbing while exhaustions stay flat is an early warning; exhaustions climbing is an incident.

Configuration guidance

Scenario	max_attempts	base_delay	max_delay	Retryable
Inside a request handler	2	0.1	0.5	Timeouts only
Background job, external API	5	1.0	60	Connection, timeout, 5xx
Queue worker, database	3	0.5	5	Operational errors only
Startup dependency check	10	1.0	30	Connection errors
Webhook delivery	5	2.0	300	Connection, timeout, 5xx

Startup checks are the one place a long total budget is right. Waiting 30 seconds for a database to accept connections during a rolling deploy is better than crash-looping the container.

What not to do

Do not retry with the default retryable_exceptions. It retries your own bugs.

Do not retry non-idempotent operations without an idempotency key. Double charges, duplicate emails.

Do not generate the idempotency key inside the retried function. A new key per attempt defeats it.

Do not use long backoff in a request handler. The user is waiting and your worker is blocked.

Do not disable jitter in production. Synchronized retries are how a blip becomes an outage.

Do not retry at every layer. Amplification. Pick one layer.

Do not retry 4xx responses. The request is wrong; repeating it verbatim cannot help.

Do not retry silently. Log and count them, or your latency problem is invisible.

Do not assume except ConnectionError: still fires. After @retry, it is RetryError.

Do not retry without a timeout on the underlying call. Retrying an operation that hangs forever means hanging forever, three times.

Performance

The retry wrappers themselves cost nothing measurable — a loop, an exception check, and a delay computation.

The cost is the waiting. With the defaults, five attempts spend up to 30 seconds sleeping. In async code that sleep is awaited and the event loop serves other requests, so the cost is latency for that one caller. In sync code the thread is blocked, and a thread pool full of retrying threads serves nobody.

That asymmetry is worth remembering: async_retry and the async decorator are cheap under concurrency. sync_retry is not, and its cost scales with how many threads you have.

Function reference

Name	Signature	Notes
retry	(max_attempts=3, base_delay=1.0, max_delay=60.0, backoff_factor=2.0, jitter=True, retryable_exceptions=Exception)	Decorator; handles sync and async
async_retry	(coro, *args, **retry_kwargs, **kwargs)	One-shot for coroutine functions
sync_retry	(func, *args, **retry_kwargs, **kwargs)	One-shot for sync callables
RetryError	Exception	Raised when attempts are exhausted

Parameter	Default	Effect
max_attempts	3	Total attempts, including the first
base_delay	1.0	Seconds before the first retry
max_delay	60.0	Ceiling on any single delay
backoff_factor	2.0	Multiplier per attempt
jitter	True	Randomizes delay. Leave on.
retryable_exceptions	Exception	Which failures retry. Narrow this.

Related

• Background Tasks — where patient retries belong
• Job Queue — durable retries that survive a restart
• HTTP Client — has its own retry configuration
• Error Handling — turning exhausted retries into responses
• Async Helpers — sync and async detection, which the decorator uses