Timeouts & redirects
Timeout
The API gives up on a page that takes longer than 15 seconds to answer. Change it with options.timeout, in milliseconds, from 1000 to 60000:
"options": { "timeout": 5000 }The timeout covers the whole request: connection, redirects and download of the page. It applies to each attempt, so with retries a request can take longer in total.
The page below takes 3 seconds to answer. With a 1-second timeout, the API gives up with 504 TIMEOUT:
Actions
Data
With the default timeout, the page arrives:
Actions
Data
A timeout is retried like a network error. Combine a short timeout with retry to skip slow answers fast, or set a longer timeout in a fallback strategy.
Redirects
By default the API follows up to 5 redirects and extracts the data from the page at the end of the chain. final_url in the response tells you where that was:
Actions
Data
{
"target_url": "http://localhost:8787/demo/redirect?hops=3",
"final_url": "http://localhost:8787/demo/redirect?hops=0",
"status": 200,
"data": ["End of the chain"]
}Limit the number of redirects
options.max_redirects sets how many redirects the API follows, from 0 to 10. One more fails with 502 TOO_MANY_REDIRECTS:
Actions
Data
Don't follow redirects
With "follow_redirects": false, a redirect is the answer: you get its status, the fields extracted from the redirect page itself, and redirect_to, the address it points to:
Actions
Data
{
"final_url": "http://localhost:8787/demo/redirect?hops=3",
"status": 302,
"redirect_to": "http://localhost:8787/demo/redirect?hops=2",
"data": ["Redirecting (3 left)"]
}Use it to check where a link leads without loading the destination, or to read a page a site shows before redirecting.
Rules that always apply
- Redirects stay on the target's domain and its subdomains. A redirect to another domain fails with
403 HOST_NOT_ALLOWED. - After a
POST, a301,302or303continues as aGETwithout body, like a browser. See Custom requests.