// Use case · Authenticated scraping
For jobs where the data sits behind a sign-in, a consent overlay or a captcha: drive the login over CDP, hold one exit IP for the run, and solve challenges when they appear.
captcha task types
8
credit per solved captcha
1
credits per open-page hour
8
// task.type: TurnstileTaskProxyless{ "solution": { "token": "0.Rk9P…" }} // Write the token into the form field// cf-turnstile-response, then submit.// Why
The problem
What Wayfern does
POST /api/captcha/solve and write the answer into the form: 1 credit, only on success.session=<id>, so the login and the last page come from the same address.// How it works
A session is created when the connection opens. There is no endpoint to call first and nothing to install beyond the driver you already use.
Step 1 of 4
Open the CDP socket with country and a session id: one residential IP for the whole run.
?token=…&country=us&session=run4711
Step 2 of 4
Fill, click, wait for the URL. Credentials and second factors come from your own secret store.
page.fill() · page.click()
Step 3 of 4
If a Turnstile, reCAPTCHA or hCaptcha widget appears, send its sitekey to the solve endpoint and write back the token.
POST /api/captcha/solve
Step 4 of 4
Every later navigation reuses the same cookies and the same exit until you close the socket.
page.goto("/reports/q1")
// Code
Both endpoints in this script are real: the CDP gateway and POST /api/captcha/solve. The second tab skips the browser when you already hold a session cookie.
import { chromium } from "playwright";
// One CDP connection is one browser session. Exit selectors ride on the query
// string: "country" turns on a residential exit, "session" pins it for the run.
const browser = await chromium.connectOverCDP(
`wss://api.wayfern.com/ws?token=${LINK_TOKEN}&country=us&session=run4711`,
);
const context = browser.contexts()[0];
const page = await context.newPage();
await page.goto("https://app.example.com/login");
await page.fill("#email", process.env.TARGET_EMAIL);
await page.fill("#password", process.env.TARGET_PASSWORD);
// A Turnstile widget guarding the form: solve it, then write the token into
// the field the form actually posts.
const widget = page.locator(".cf-turnstile[data-sitekey]").first();
if ((await widget.count()) > 0) {
const websiteKey = await widget.getAttribute("data-sitekey");
const solved = await fetch("https://api.wayfern.com/api/captcha/solve", {
method: "POST",
headers: {
// An API token (wf_...) carrying the "captcha" scope.
Authorization: `Bearer ${WAYFERN_API_TOKEN}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
task: {
type: "TurnstileTaskProxyless",
websiteURL: page.url(),
websiteKey,
},
}),
});
// { "solution": { ... } }: the fields depend on the task type.
const { solution } = await solved.json();
await page.evaluate((token) => {
const field = document.querySelector('input[name="cf-turnstile-response"]');
if (field instanceof HTMLInputElement) field.value = token;
}, solution.token);
}
await page.click("button[type=submit]");
await page.waitForURL("**/dashboard");
// Logged in. Every later navigation reuses this context (same cookies, same
// sticky exit) until you close the connection.
await page.goto("https://app.example.com/reports/q1");
console.log(await page.title());
await browser.close();// Walls
A logged-in page fails for one of four reasons. None of them is a flag you flip and hope: each is a surface you call.
Wall 01
A form, a redirect, a cookie set three hops later. Over CDP the login is ordinary Playwright, and the context keeps its cookies for the rest of the run.
wss://api.wayfern.com/ws
Wall 02
On screenshot and PDF captures, Wayfern clicks the accept control it recognises, hides what is left with CSS and unlocks scrolling. On by default.
capture.blockCookieBanners: true
Wall 03
reCAPTCHA, hCaptcha, FunCaptcha, GeeTest, Turnstile or a plain image. Post the task, get the solution, write it into the form.
POST /api/captcha/solve
Wall 04
In auto mode a 401, 402, 403 or 407, or a redirect onto a known challenge path, is retried once through a residential exit. Set residential to start there.
proxy: 'auto' | 'residential'
// Identity
If the run is logged in, the exit has to hold still.
Add country and a session id (letters and digits) to the gateway URL. The id is encoded into the upstream credentials with a 24-hour TTL, so a multi-step flow keeps one exit IP. Narrow it with region, city or isp.
?token=…&country=us&session=run4711
A one-shot call takes a residential exit without a session id, so the IP rotates per connection. Right for independent fetches; for a logged-in flow, drive a session or send your credential on every call.
proxy: 'residential', country: 'us'
// Captcha
POST /api/captcha/solve takes an anti-captcha task object and returns the solution. Call it from inside a session, or from anywhere you hold a sitekey, with a token that has the captcha scope.
| task.type | What it solves | Common fields |
|---|---|---|
| RecaptchaV2TaskProxyless | reCAPTCHA v2, checkbox or invisible | websiteURL, websiteKey, isInvisible |
| RecaptchaV2EnterpriseTaskProxyless | reCAPTCHA v2 Enterprise | websiteURL, websiteKey, enterprisePayload |
| RecaptchaV3TaskProxyless | reCAPTCHA v3, score-based | websiteURL, websiteKey, minScore, pageAction |
| HCaptchaTaskProxyless | hCaptcha, including invisible | websiteURL, websiteKey, isInvisible |
| FunCaptchaTaskProxyless | FunCaptcha / Arkose Labs | websiteURL, websitePublicKey, data |
| GeeTestTaskProxyless | GeeTest v3 and v4 | websiteURL, gt, challenge, version |
| TurnstileTaskProxyless | Cloudflare Turnstile | websiteURL, websiteKey, action |
| ImageToTextTask | Plain image or text captcha | body (base64), phrase, numeric, math |
RecaptchaV2TaskProxyless
RecaptchaV2EnterpriseTaskProxyless
RecaptchaV3TaskProxyless
HCaptchaTaskProxyless
FunCaptchaTaskProxyless
GeeTestTaskProxyless
TurnstileTaskProxyless
ImageToTextTask
// What it costs
Session time, captcha solves and residential bytes, all drawn from the same credits. Datacenter egress is free.
One page open for half an hour on a sticky residential exit, with two challenges on the way.
| Item | Credits |
|---|---|
| 1 page open for 30 minutes | 4 |
| 2 captcha solves | 2 |
| 0.05 GB residential traffic | 16 |
Total
22credits
about $0.28
at $0.0125 / credit
About 69 runs a month on Developer ($19 for 1,520 credits).
No session: /scrape carries your cookie on each call. Residential bytes are metered on top when used.
| Item | Credits |
|---|---|
| 40 scraped pages | 40 |
Total
40credits
about $0.50
at $0.0125 / credit
About 38 runs a month on Developer ($19 for 1,520 credits).
Each /api/v1 response returns the
exact charge in its credits field.
Every rate is on the pricing page.
// FAQ
No. The login runs in your own Playwright or Puppeteer code over CDP. Credentials stay in your environment and your script types them into the page.
No. There are no persistent customer profiles: cookies and storage live in the browser for as long as your socket is open. Log in again on the next run, or send the session cookie you hold with /scrape.
Nothing is debited. A solve costs 1 credit once the provider returns a solution; one that errors or times out costs nothing. Only proxyless task types are accepted (proxy fields and softId are rejected), and websiteURL must be an absolute http(s) URL.
Yes. The Web Data API carries up to 20 custom headers per call, a cookie among them, at 1 credit per scraped page. It rejects the four headers it must own (Host, Content-Length, Connection, Transfer-Encoding) instead of dropping them silently.
With proxy: "auto" (the default on the Web Data API) a navigation that reads as blocked is retried once through a residential exit. Other failures surface at once, because retrying a timeout on a slower path only spends the budget twice. On a session, you choose the exit in the URL.
Three meters on one balance: session time pro-rated from 8 credits per open-page hour (60-second minimum per page), 1 credit per successful solve, and 320 credits per GB of residential traffic. Datacenter egress is free.
// Keep reading
// Next step
Hand challenges to an endpoint and keep one exit for the whole run. The same balance covers the browser, the solves and the bytes.