Automatic retries

Websites fail now and then: a server under load answers 503, a rate limiter answers 429, a connection drops. Ask the API to retry, and these transient failures are absorbed before they reach you.

{
  "target_url": "https://example.com",
  "fields": ["h1"],
  "options": { "retry": 3 }
}

retry is the number of extra attempts after the first one, from 0 (the default) to 5.

See it workCopy Anchor Link

The two requests below target a demo page that answers 503 Service Unavailable twice before it succeeds.

Without retry, the first 503 is final and the API returns an error:

With "retry": 3, the API tries again until the page answers (usually on the third attempt), and you get the data. meta.attempts tells you how many requests it took:

{ "status": 200, "data": ["It worked"], "meta": { "attempts": 3, … } }

What is retriedCopy Anchor Link

Exponential backoffCopy Anchor Link

The API waits longer after each failure, so a struggling site gets time to recover. By default the first wait is about 0.5 s and each following one doubles, up to 8 s, with some randomness so many clients don't retry at the same instant:

If the site sends a Retry-After header, the API waits that long instead, up to the maximum delay.

Tune it with options.backoff:

"options": {
  "retry": 4,
  "backoff": {
    "initial_delay": 1000,
    "max_delay": 10000,
    "multiplier": 3,
    "jitter": false
  }
}

With the example above, the waits are 1 s, 3 s, 9 s, then 10 s.

Each attempt has its own timeout, so a request with retries can take longer to answer.

Choose which statuses to retryCopy Anchor Link

By default the API retries 429 and every 5xx. Set options.retry_on to your own list of status codes, from 400 to 599. It replaces the default list:

"options": { "retry": 3, "retry_on": [403, 429, 503] }

Network errors and timeouts are always retried.

When every attempt failsCopy Anchor Link

You get the error from the last attempt, with attempts set to the number of requests made:

{
  "target_url": "https://example.com",
  "error": {
    "code": "HTTP_ERROR",
    "message": "HTTP 503 for https://example.com",
    "status": 503,
    "attempts": 4,
    "strategy": 0
  }
}