Fallback strategies

Some sites refuse a request because of how it looks, not because they're down: a bot User-Agent or fingerprint, a missing header, the wrong cookies. Retrying the same request won't help. A fallback is a different way to ask, tried only if the main request fails.

{
  "target_url": "https://example.com",
  "fields": ["h1"],
  "options": {
    "fallback": [
      { "headers": { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" } }
    ]
  }
}

See it workCopy Anchor Link

The demo page below answers 403 Forbidden to anything that doesn't look like a browser.

Requests look like a browser by default, so to see a refusal, the main request below drops its fingerprint (fingerprint: false sends the API's own User-Agent) and turns off the automatic escalation to a browser, which would otherwise get the page by itself.

Without fallback, the request is refused:

With a fallback that looks like a browser again, the main request is refused, then the fallback gets the page. meta.strategy tells you which one worked:

{
  "status": 200,
  "data": ["Welcome, browser"],
  "meta": { "attempts": 2, "strategy": 1, … }
}

How it worksCopy Anchor Link

1. The API sends the main request, with its retries if you set any.
2. If it still can't get the page, it tries fallback[0], with that strategy's own retries.
3. Then fallback[1], and so on, up to 3 strategies. The first one that gets the page wins.

A fallback is used when a strategy ends with an HTTP error (any non-2xx status, 403 and 404 included), a network error, a timeout, too many redirects, no proxy available, a browser that can't start, or a failed action. It isn't used for errors another strategy can't fix: an invalid URL or selector, a redirect to another domain, a page over the size limit.

What a strategy containsCopy Anchor Link

A strategy accepts the same options as the main request, except cache_ttl: method, headers, cookies, body, timeout, follow_redirects, max_redirects, proxy, fingerprint, browser, actions, retry, retry_on and backoff (see Custom requests, Timeouts & redirects and Automatic retries).

It starts from the main options and changes only what it sets:

• headers and cookies are merged: set just a User-Agent and every other main header is still sent.
• backoff is merged too.
• Everything else replaces the main value: a strategy with "retry": 3 retries 3 times even if the main request doesn't retry.

"options": {
  "headers": { "Accept-Language": "fr" },
  "retry": 1,
  "fallback": [
    { "headers": { "User-Agent": "Mozilla/5.0" } },
    {
      "headers": { "User-Agent": "Mozilla/5.0" },
      "cookies": { "consent": "yes" },
      "retry": 3
    }
  ]
}

Here the first fallback sends Accept-Language: fr with a browser User-Agent and retries once. The second one adds a cookie and retries up to 3 times.

Drop what you don't want to inheritCopy Anchor Link

Set an option to false in a strategy to stop inheriting it:

"options": {
  "method": "POST",
  "body": "q=shoes",
  "headers": { "X-Api-Version": "2" },
  "proxy": true,
  "fallback": [
    { "method": "GET", "body": false, "headers": false, "proxy": false }
  ]
}

The fallback sends a plain GET: no body, none of the main headers, no proxy. "cookies": false and "retry": false work the same way.

A strategy can't contain its own fallback.

In the responseCopy Anchor Link

When every strategy fails, the error has the same two properties, for the last strategy tried:

{
  "target_url": "https://example.com",
  "error": {
    "code": "HTTP_ERROR",
    "message": "HTTP 403 for https://example.com",
    "status": 403,
    "attempts": 3,
    "strategy": 2
  }
}