SE Home
Help
  • Documentation
  • Help centre
SEScrape Engine

Getting started

DocumentationQuickstartAuthenticationPlayground

Extracting data

Selecting valuesTransformsLists & tablesPage dataAI extractionMCP server

Fetching pages

Custom requestsTimeouts & redirectsCachingAutomatic retriesFallback strategies

Browsers & actions

BrowsersActionsRecipesScreenshots & PDFStored files

Getting past blocks

Anti-bot protectionCaptchasProxiesFingerprintsSessions

Workflows

Multi-step requestsVariablesPaginationBatchCrawlingTemplatesSchedules & monitoringAsync jobs & webhooksExport to S3, Sheets, databases & webhooks

Account & billing

CreditsSecretsPlansLimitsLogs & request idsNotificationsAccountAffiliate programmeHelp centre

Reference

POST /scrapeGET /sessions/:idGET /browsersGET /proxies/countriesGET /healthErrors
Documentation

Getting started

DocumentationQuickstartAuthenticationPlayground

Extracting data

Selecting valuesTransformsLists & tablesPage dataAI extractionMCP server

Fetching pages

Custom requestsTimeouts & redirectsCachingAutomatic retriesFallback strategies

Browsers & actions

BrowsersActionsRecipesScreenshots & PDFStored files

Getting past blocks

Anti-bot protectionCaptchasProxiesFingerprintsSessions

Workflows

Multi-step requestsVariablesPaginationBatchCrawlingTemplatesSchedules & monitoringAsync jobs & webhooksExport to S3, Sheets, databases & webhooks

Account & billing

CreditsSecretsPlansLimitsLogs & request idsNotificationsAccountAffiliate programmeHelp centre

Reference

POST /scrapeGET /sessions/:idGET /browsersGET /proxies/countriesGET /healthErrors

Variables

In a multi-step request, a step can use what an earlier step got: a link to follow, a token to send, an ID to search for. Write {{ … }} anywhere in a step's strings, and it's replaced by the value right before the step runs.

{
  "steps": [
    {
      "name": "list",
      "target_url": "https://example.com/products",
      "fields": { "first": { "selector": "a.product", "attribute": "href" } }
    },
    { "target_url": "{{list.data.first}}", "fields": ["h1", ".price"] }
  ]
}

Follow a link found on the page

Step shop reads the "Next page" link (made absolute, see URLs); the second step opens it:

Actions

Data

Actions

Data

Name a step, then point at a value

Give a step a name (letters, digits, _ or -), then write the path to a value in its result:

VariableValue
{{shop.data.next}}The field next of the step named shop
{{shop.data[0]}}The first value, when fields is an array
{{shop.data.links[2]}}The third value of a multiple field
{{shop.final_url}}Any part of the step's response: final_url, status, captured[0].json.id…
{{steps[0].data.title}}Steps by position, from 0, named or not

Filters

FilterEffectExample
| urlEncodes the value for a URL"https://example.com/search?q={{shop.data.title | url}}"
| jsonInserts the value as JSON, arrays and objects included"body": "{\"items\": {{shop.data.names | json}}}"

Without a filter, text, numbers and true/false are inserted as they are. An array or an object needs a position ([0]) or | json.

Where variables work

Everywhere a step has a string: target_url, field selectors, and every option: headers, cookies, body, action values and URLs:

{ "fill": "#q", "value": "{{search.data.term}}" },
{ "navigate": "{{list.data.detail_url}}" }

Numbers and booleans (timeout, retry…) can't be variables.

When a value is missing

A variable must point at a value of a step that ran before and succeeded. Otherwise, the step fails with 400 INVALID_VARIABLE and the request stops, unless continue_on_error is set:

{
  "target_url": "https://example.com/{{list.data.nope}}",
  "error": {
    "code": "INVALID_VARIABLE",
    "message": "{{list.data.nope}} has no value: no earlier step has it (check the step name, the field and that the step succeeded)"
  }
}

A field that matched nothing is null, which counts as missing.

On this page

  • Follow a link found on the page
  • Name a step, then point at a value
  • Filters
  • Where variables work
  • When a value is missing