Custom requests
By default the API fetches the target page with a plain GET. Use options to send the request the site expects: another method, headers, cookies or a body.
{
"target_url": "https://httpbin.org/anything",
"fields": ["body"],
"options": {
"method": "POST",
"headers": {
"Content-Type": "application/json",
"User-Agent": "my-app/1.0"
},
"cookies": { "session": "abc123" },
"body": "{\"query\": \"shoes\"}"
}
}Try it
httpbin.org/anything answers with a description of the request it received, so you can see exactly what the API sent. Open Additional request parameters to change the method, User-Agent, headers, cookies or body:
Actions
Data
Method
"method" is one of GET (default), POST, PUT, PATCH or DELETE.
Headers
"headers" is an object of header names and values. A User-Agent header replaces the one of the default browser fingerprint; its other headers stay. With fingerprint: false, it replaces the API's own (scraping-engine/0.1).
A few headers are set by the API itself and can't be overridden: Host, Content-Length, Connection, Transfer-Encoding, Keep-Alive, Upgrade, TE, Trailer and Proxy-*.
Cookies
"cookies" is an object of cookie names and values, sent in a Cookie header. If you also send a Cookie header, both are combined.
Body
"body" is a string, sent as is. It needs a method other than GET. The API doesn't set a Content-Type for you: add the one that matches your body.
| Body type | body | Content-Type header |
|---|---|---|
| JSON | "{\"q\": \"shoes\"}" | application/json |
| Form | "q=shoes&page=2" | application/x-www-form-urlencoded |
Redirects
If the site answers a POST with a 301, 302 or 303 redirect, the API follows it with a GET and no body, like a browser does after a form submission. 307 and 308 keep the method and body.