Crawling

POST /crawl starts from one page, follows its links, and scrapes every page it reaches with the same fields and options as /scrape:

curl -X POST "$SCRAPE_ENGINE_URL/crawl" -H "Authorization: Bearer $SCRAPE_ENGINE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "target_url": "https://example.com/blog", "fields": { "title": "h1" }, "crawl": { "max_depth": 2, "max_pages": 50 } }'

Each page is a full scrape: browsers, proxies, actions, anti-bot escalation all apply. The whole crawl shares one session, so cookies carry from page to page: without options.session, a session is opened for the crawl (same IP and identity on every page) and dropped after 10 minutes; name your own to keep it longer. Pages fetched within a session aren't cached.

Each page runs as its own job in the queue, concurrency at once (at most your plan's concurrent requests), so a big crawl never holds a slot for long and other requests go between its pages. Your plan's longest run covers the whole crawl: past it, the pages not visited yet are skipped and a warning says how many.

How pages are foundCopy Anchor Link

The crawl is breadth-first: the start page (depth 0), then the pages it links to (depth 1), then theirs, up to max_depth, and never more than max_pages pages.

Only links on the start page's site are followed: its domain and subdomains, www. or not. Each address is visited once: fragments (#…) and tracking parameters (utm_*, fbclid, gclid…) are ignored, and query parameters are sorted. A page whose HTML is identical to one already seen is dropped too.

OptionsCopy Anchor Link

Under crawl:

fields is optional: without it, each page returns its title.

PaginationCopy Anchor Link

"pagination": true follows each page's "next page" link: rel="next", or a link whose class or label says next. A CSS selector picks it yourself: "pagination": ".pager a.forward".

Pagination doesn't spend depth: page 2 of a list is at the same depth as page 1. With "max_depth": 0, a crawl goes through a list's pages and nothing else:

With "max_depth": 1 and links set to the list's item links, it reads every page of the list and every item on them.

SitemapsCopy Anchor Link

"sitemap": true adds the URLs of the site's sitemap to the crawl, at depth 1: the sitemaps listed in robots.txt, or /sitemap.xml. Sitemap indexes are followed. "sitemap": "only" visits the start page and the sitemap's URLs, and follows no links:

/demo/private is in the demo sitemap, but robots.txt disallows it: it's counted in skipped.robots instead of visited.

robots.txtCopy Anchor Link

By default, the crawl reads the site's robots.txt and skips what it disallows for all robots (User-agent: *). "robots": false ignores it. The start page is always scraped, since you asked for it, with a warning if it's disallowed.

The responseCopy Anchor Link

{
  "start_url": "http://localhost:8787/demo/shop",
  "pages": [
    { "target_url": "…/demo/shop", "status": 200, "depth": 0, "data": { … }, … },
    { "target_url": "…/demo/shop?page=2", "status": 200, "depth": 1, "data": { … }, … },
    {
      "target_url": "…/demo/missing",
      "depth": 1,
      "error": { "code": "HTTP_ERROR", "status": 404, "message": "…" }
    }
  ],
  "skipped": { "robots": 1, "filtered": 12, "duplicates": 7, "over_limit": 0 },
  "warnings": []
}

pages are in crawl order. Each one is a scrape response plus its depth, or its error: a page that fails doesn't stop the crawl. skipped counts the links not followed: disallowed by robots.txt, outside the site or the filters, already seen, or over max_pages.

A crawl answers when it's done, so large ones take a while: up to 100 pages per request.