Map a site, crawl it under an explicit page budget, and receive signed webhooks as each page lands: markdown that’s ready to chunk, with a bill that can’t run away.
POST /api/v1/map discovers a site’s
URLs before you commit to crawling any of them: sitemaps first, page links second. A browser
is only opened when the sitemaps came up short, and each returned URL says which of the two found
it. Filter with search to keep only paths and
titles containing a term, widen with includeSubdomains, and cap the
result set with limit.
Billing follows what actually comes back: 2 credits per started block of 10 returned URLs. Knowing the size of the job is the cheap part.
curl -X POST https://api.wayfern.com/api/v1/map \
-H "Authorization: Bearer wf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"search": "guide",
"limit": 1000,
"includeSubdomains": false
}'{
"success": true,
"data": {
"links": [
{
"url": "https://docs.example.com/quickstart",
"title": null,
"description": null,
"source": "sitemap",
"lastModified": "2026-07-02"
},
{
"url": "https://docs.example.com/guides/webhooks",
"title": "Webhooks",
"description": null,
"source": "links",
"lastModified": null
}
]
},
"credits": 2
}A crawl that can’t be bounded is a crawl you can’t budget. POST /api/v1/crawl takes its limits
in the same body as the job itself, and returns a job id immediately, so the crawl runs in the background
while you get on with something else.
The hard ceiling on pages this job will ever scrape, and therefore the hard ceiling on what it can cost. The job record carries it back to you as `limit`.
How many link hops from the start URL the frontier is allowed to follow. Depth 0 crawls the seed page and stops.
Regular expressions a path MUST match to be crawled. An unparseable pattern is rejected at submit time rather than degrading into a prefix that quietly changes your scope.
Regular expressions that drop a path from the frontier. Same budget, same up-front validation: you find out before the job runs, not after.
curl -X POST https://api.wayfern.com/api/v1/crawl \
-H "Authorization: Bearer wf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"limit": 500,
"maxDepth": 3,
"includePaths": ["^/docs/"],
"excludePaths": ["^/docs/changelog/"],
"delay": 250,
"concurrency": 3,
"webhook": {
"url": "https://api.example.com/hooks/wayfern",
"events": ["page", "completed"],
"metadata": { "index": "docs-v3" }
},
"scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true }
}'{
"success": true,
"data": {
"id": "4f1c9b2a-6d3e-4a17-9c88-0b5e2f7a1d34",
"status": "queued",
"completed": 0,
"failed": 0,
"limit": 500,
"creditsUsed": 0,
"createdAt": "2026-07-20T09:14:11.002Z",
"updatedAt": "2026-07-20T09:14:11.002Z",
"error": null,
"expiresAt": "2026-07-27T09:14:11.002Z"
},
"credits": 0
}Attach a webhook and each page
is posted to you as it completes, so chunking and embedding start while the crawl is still
running. The URL must be https — an http:// URL is rejected at submit
time, because deliveries carry your crawl results and any header you attached. Your metadata object comes back verbatim
in every delivery, which is how a payload knows which index it belongs to.
Pick the ones you want with events, or omit it to
receive all four. Delivery is best-effort, up to 3 attempts with exponential backoff, and a 4xx
other than 408 or 429 stops the retries, since an endpoint that rejected
the payload will reject an identical one. A webhook that never lands never fails the crawl: the
pages stay durable and readable through the polling API.
{
"type": "page",
"crawlId": "4f1c9b2a-6d3e-4a17-9c88-0b5e2f7a1d34",
"at": "2026-07-20T09:14:22.481Z",
"completed": 12,
"failed": 1,
"data": {
"markdown": "# Webhooks\n\nEvery delivery carries a timestamp.",
"metadata": {
"title": "Webhooks - Example Docs",
"statusCode": 200,
"sourceUrl": "https://docs.example.com/guides/webhooks"
},
"cached": false
},
"metadata": { "index": "docs-v3" }
}A webhook URL is usually reachable by anyone who learns it, so an unverified endpoint is an
open door into your index. Every delivery carries X-Wayfern-Timestamp, and — when
a signing secret is configured — X-Wayfern-Signature: sha256=<hex>, an HMAC-SHA256 over `${timestamp}.${body}`. That
construction is pinned in our test suite by a known-answer digest and re-checked against a
signature the service actually sent, so the recipe below is the one we sign with.
import { createHmac, timingSafeEqual } from "node:crypto";
import express from "express";
const app = express();
// The signature covers the RAW body. Parsing to JSON and re-serializing it
// changes the bytes, and the HMAC stops matching.
app.post("/hooks/wayfern", express.raw({ type: "application/json" }), (req, res) => {
const body = req.body.toString("utf8");
const timestamp = req.get("X-Wayfern-Timestamp") ?? "";
const provided = req.get("X-Wayfern-Signature") ?? "";
const digest = createHmac("sha256", process.env.WAYFERN_WEBHOOK_SECRET)
.update(`${timestamp}.${body}`)
.digest("hex");
const expected = Buffer.from(`sha256=${digest}`);
const actual = Buffer.from(provided);
// timingSafeEqual throws on a length mismatch, which is itself a mismatch.
const ok = expected.length === actual.length && timingSafeEqual(expected, actual);
if (!ok) return res.status(401).end();
const event = JSON.parse(body);
if (event.type === "page") {
// event.metadata.index is your own value, echoed back verbatim.
void chunkAndUpsert(event.metadata.index, event.data.markdown);
}
res.status(200).end();
});scrapeOptions is the full
scrape option set, applied to every page in the crawl — so a crawled page and a scraped page
come back identical. onlyMainContent is on by
default and drops navigation, headers, sidebars, footers and cookie notices; the page title is
put back at the top when the extractor took it as metadata, so no document arrives without
saying what it is.
<head><title>Example Docs</title></head>
<body>
<nav class="navbar">
<a href="/">Home</a><a href="/pricing">Pricing</a>
</nav>
<header>Example Docs</header>
<article>
<h1>Installing the client</h1>
<p>Add the package to your project and set your API key.</p>
</article>
<aside class="sidebar">On this page…</aside>
<footer>© Example, Inc.</footer>
</body># Example Docs
# Installing the client
Add the package to your project and set your API key.CSS selectors to drop, for the chrome that survives a generic pass: a promo rail, a breadcrumb bar, an inline signup form.
CSS selectors to keep; everything else goes. An explicit include list is treated as your definition of main content, so the automatic extractor is not consulted at all.
Every href and src is rewritten absolute, because markdown is read far from the page it came from — and inline base64 images are replaced with their alt text by default.
Failures never disappear into the result set. GET /api/v1/crawl/:id/errors returns the pages the job could not fetch, up to the first 50 recorded, while the job’s failed count keeps climbing past
that — each with the URL, a typed reason, a message and the instant it happened. Reading it is
free, and a page that failed was never billed: the charge only happens once a document exists.
{
"success": true,
"data": {
"errors": [
{
"url": "https://docs.example.com/legacy",
"error": "not_found",
"message": "not_found: https://docs.example.com/legacy",
"at": "2026-07-20T09:14:19.774Z"
}
]
},
"credits": 0
}# Stop a running job. Pages already crawled stay readable, and stay billed.
curl -X DELETE https://api.wayfern.com/api/v1/crawl/4f1c9b2a-6d3e-4a17-9c88-0b5e2f7a1d34 \
-H "Authorization: Bearer wf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"Cancelling is honest about what already happened: the pages crawled up to that point stay
readable through the polling API and stay billed, and the job comes back with status cancelled and its running creditsUsed.
GET /api/v1/crawl/:id returns
the job summary plus a page of documents, paginated with skip and limit. Use it instead of webhooks,
or alongside them as a backstop.
Because pages are charged one at a time, a balance that runs out ends the crawl with that reason on the job record — the pages already crawled stay readable.
The job record, its pages and its errors stay readable until the expiresAt returned when you started
it — seven days later by default. Pull the corpus into your own store before then.
1 credit per successful page, nothing at the start, nothing for the pages that failed. Here is the whole bill for mapping and crawling a 500-page documentation site into markdown.
runs of this crawl per month
1,520 credits a month at $19, and the same balance covers browser sessions, search and every other Web Data endpoint.
runs of this crawl per month
9,000 credits a month at $99, and the same balance covers browser sessions, search and every other Web Data endpoint.
runs of this crawl per month
46,000 credits a month at $499, and the same balance covers browser sessions, search and every other Web Data endpoint.
Markdown is the default format and carries no add-on. Asking a crawl for a screenshot, PDF or JSON output adds +4 credits a page — 2,000 across this job — and an LLM summary adds +2 a page. The full rate card is on the pricing page.
Map a domain, set a page budget, and let the markdown come to you. Start with a credit pack and size the crawl before you commit to it.