// Use case · RAG knowledge base
For teams feeding a retrieval index: map a domain, crawl it under a page budget, and receive each page as markdown through a signed webhook while the crawl runs.
credit per crawled page
1
pages per crawl, max
5,000
credits for a failed page
0
{ "type": "page", "completed": 12, "failed": 1, "data": { "markdown": "# Webhooks\n\nEvery delivery…", "metadata": { "statusCode": 200, "sourceUrl": "https://docs.example.com/…" } }, "metadata": { "index": "docs-v3" }}// Why
The problem
What Wayfern does
limit (1 to 5,000), maxDepth (0 to 10) and path patterns cap the job before it runs.onlyMainContent is on by default: page chrome is dropped, links are made absolute, the title stays on top.// How it works
The crawl runs in the background. Chunking and embedding start while it is still going, because each page is posted to you as it completes.
Step 1 of 4
Discover URLs from sitemaps first and page links second, at 2 credits per 10 returned.
POST /api/v1/map
Step 2 of 4
Set limit, maxDepth and path patterns. You get a job id at once; nothing is charged at the start.
POST /api/v1/crawl
Step 3 of 4
Each page event carries the markdown and your own metadata, signed with HMAC-SHA256.
X-Wayfern-Signature: sha256=…
Step 4 of 4
Verify, chunk, embed. Poll the job or read its errors list any time; both are free.
GET /api/v1/crawl/:id
// Code
The map call is the dry run: it tells you the page count before you commit. The last two tabs check the webhook signature on the raw body.
curl -X POST https://api.wayfern.com/api/v1/crawl \
-H "Authorization: Bearer wf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"limit": 500,
"maxDepth": 3,
"includePaths": ["^/docs/"],
"excludePaths": ["^/docs/changelog/"],
"delay": 250,
"concurrency": 3,
"webhook": {
"url": "https://api.example.com/hooks/wayfern",
"events": ["page", "completed"],
"metadata": { "index": "docs-v3" }
},
"scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true }
}'// Budget
A crawl that cannot be bounded cannot be budgeted. POST /api/v1/crawl takes its limits in the same body as the job, and an invalid pattern is rejected before anything runs.
1 – 5,000, default 50
limitThe hard ceiling on pages the job will scrape, and so on what it can cost. The job record carries it back as limit.
0 – 10, default 3
maxDepthHow many link hops from the start URL the crawl may follow. Depth 0 crawls the seed page and stops.
up to 50 patterns
includePathsRegular expressions a path must match to be crawled. A pattern that does not parse is rejected at submit time.
up to 50 patterns
excludePathsRegular expressions that drop a path. Same up-front validation: you find out before the job runs.
| Option | What it does |
|---|---|
| allowSubdomains | Off by default: the crawl stays on the host you named. Turn it on to follow links onto subdomains. |
| useSitemap | On by default. The frontier is seeded from the site’s sitemaps before a link is followed. |
| delay | 0 to 60,000 ms of pause between requests to the site, default 0, applied before each batch. |
| concurrency | 1 to 5 pages fetched in parallel within one job, default 3. |
| scrapeOptions | The full scrape option set, applied to every page, so a crawled page and a scraped page come back identical. |
allowSubdomains
useSitemap
delay
concurrency
scrapeOptions
// Webhooks
Attach a webhook and pick events, or omit them to receive all four. Your metadata object comes back verbatim on every delivery, which is how a payload knows which index it belongs to.
| Event | When it fires |
|---|---|
| started | The job has begun. |
| page | One page finished. The whole document, markdown and metadata, is in data. |
| completed | The job finished on its own terms. |
| failed | The job aborted. The reason is in error: a duration ceiling, an exhausted balance or a restart. |
started
page
data.completed
failed
error: a duration ceiling, an exhausted balance or a restart.// Clean output
onlyMainContent is on by default and drops navigation, headers, sidebars, footers and cookie notices. The page title is put back at the top, so no document arrives without saying what it is.
<head><title>Example Docs</title></head>
<body>
<nav class="navbar">
<a href="/">Home</a><a href="/pricing">Pricing</a>
</nav>
<header>Example Docs</header>
<article>
<h1>Installing the client</h1>
<p>Add the package to your project and set your API key.</p>
</article>
<aside class="sidebar">On this page…</aside>
<footer>© Example, Inc.</footer>
</body>excludeTagsCSS selectors to drop, for chrome a generic pass misses: a promo rail, a breadcrumb bar, an inline signup form.
includeTagsCSS selectors to keep; everything else goes. An include list is your definition of main content, so the automatic extractor is skipped.
Every href and src is made absolute, because markdown is read far from its page. Inline base64 images become their alt text by default.
// What it costs
Nothing at the start, nothing for the pages that failed, nothing for polling. Here is the whole bill for mapping and crawling a documentation site.
Markdown is the default format and carries no add-on.
| Item | Credits |
|---|---|
| map, 500 URLs returned | 100 |
| crawl started (job id returned) | 0 |
| 500 pages land | 500 |
| polling, errors list, cancel | 0 |
Total
600credits
about $7.50
at $0.0125 / credit
About 2 runs a month on Developer ($19 for 1,520 credits).
Runs of this crawl per month, by plan
The same balance covers browser sessions, search and every other endpoint. See the pricing page.
// FAQ
1 credit per successfully crawled page. Starting a job, polling, the errors endpoint and cancelling are free, and a page that fails is not billed. Asking for a screenshot, PDF or JSON output adds 4 credits a page; an LLM summary adds 2.
Every delivery carries X-Wayfern-Timestamp and, when a signing secret is set, X-Wayfern-Signature: sha256=<hex>: an HMAC-SHA256 over ${timestamp}.${body}, computed on the raw body. The Node.js and Python tabs above show the check.
Delivery is best-effort: up to 3 attempts with exponential backoff, and a 4xx other than 408 or 429 stops the retries. A webhook that never lands never fails the crawl: the pages stay readable through the polling API. The URL must be https; an http:// URL is rejected at submit time.
GET /api/v1/crawl/:id/errors lists them, up to the first 50, each with the URL, a typed reason, a message and a timestamp, while the job’s failed count keeps climbing past that. Reading it is free, and a failed page is never billed.
DELETE /api/v1/crawl/:id cancels it. Pages crawled up to that point stay readable and stay billed, and the job reports status cancelled with its running creditsUsed. If the balance runs out mid-crawl, the job ends with that reason and the pages already crawled stay readable.
The job record, its pages and its errors stay readable until the expiresAt returned when you started it: seven days by default. Pull the corpus into your own store before then.
// Keep reading
// Next step
Size the site with one map call, set a page budget, and let the markdown come to you. Start with a credit pack if you only need one corpus.