Templates

A template is a request you keep, with parameters, and what it learned. It's the cheapest way to run the same kind of scrape again and again.

How a template learnsCopy Anchor Link

Give a variant a prompt instead of fields:

1. First run: the AI reads the page and writes the fields (ai.config). The template then checks them without AI and without the cache, on up to three of the request's pages (a batch's targets), twice a few seconds apart: a selector that only works once (rotating content, an A/B test) shows up there. If some come back empty, the AI gets a second try, and the steadiest config wins: the one whose worst check filled the most fields. Two first runs at the same time learn once: the second waits for the first one's config.
2. Next runs: the stored config is a plain request. No model call, no AI cost.
3. When the site changes: a field that comes back empty is sent to the AI alone (ai.repair). It gets a new selector, the config is updated (only that field), and the next runs are free again. For /batch and /crawl, a field empty on every page that worked is repaired once, on one of them, and the next runs use the new selector. One repair runs at a time per variant.

Every change of a config is kept in its history (learned, repaired), with the fields that changed.

Parameters and variantsCopy Anchor Link

{{params.name}} anywhere in the request is replaced by the value given at run time. A template can hold several variants, one per feature: an IMDb template could have person and movie, and you run only the one you need.

{
  "name": "Product pages",
  "category": "ecommerce",
  "variants": [{
    "key": "product",
    "name": "Product",
    "request": { "target_url": "https://shop.example/products/{{params.slug}}" },
    "params": [{ "name": "slug", "example": "oak-chair" }],
    "prompt": "product name, price as a number, stock, and the reviews (author, stars)"
  }]
}

A variant with its own fields (and no prompt) runs them as written. endpoint can be /scrape (default), /batch or /crawl.

Running oneCopy Anchor Link

From the dashboard, or with an API key:

curl http://localhost:8788/v1/templates/<id>/run \
  -H "Authorization: Bearer $SCRAPE_ENGINE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "variant": "product", "params": { "slug": "oak-chair" } }'

<id> is the template's id, or for official templates its name in the URL: /v1/templates/hacker-news/run.

The response is the usual scrape response, plus template: its id, the variant, the config version, learned: true on the run that learned it, and repaired with the fields fixed on that run. Runs are billed like any request: AI only when learning or repairing.

The marketplaceCopy Anchor Link

Publish a template to share it. Publishing is final: a published template stays on the marketplace, can't be edited or deleted, even if you delete your account. Each submission is reviewed by an AI first (templates that collect personal data of private individuals, bypass logins or paywalls, or try to manipulate the review are refused), and an admin has the last word.

Anyone can run a published template, or fork it to get their own copy, learned config included. Official templates are made by us, shown first, and may use code of their own for what plain requests can't do.

Official templatesCopy Anchor Link

Each one is a complete API for a site, a variant per use:

Charts take a date (any day: that week's chart). Lists take page and per_page (pagination); Hacker News pages with its own page and returns next_page.

A template's pageCopy Anchor Link

Clicking a template on the marketplace opens its page: what it does, its variants and their parameters. Try in the playground opens a variant's request there, filled with example parameters. Edit opens its edit page: yours to change, or read-only with Fork when it's someone else's.

Ratings and reportsCopy Anchor Link

Anyone can browse published templates at /marketplace, without an account. Signed in, rate a template from 1 to 5 stars (rating again changes your rating; not on your own), or Report it if it's broken, illegal, collects personal data or is spam: an admin looks at each report.