Scrapers

High-value scrapers for the sites people actually scrape, built on Playwright and a transparent mouse/typing/scroll humanizer, with typed structured output. Each is a faithful port of the corresponding Scrapfly reference scraper, re-implemented to run on a real browser you control.

Install

shell

npm install wayfern playwright

Scrape

launch() returns a humanized Chromium session. Hand each scraper the page and a URL or query.

node

shell

import { launch, amazon } from "wayfern/scrapers";

const session = await launch();               // humanized Chromium
try {
  const products = await amazon.scrapeSearch(
    session.page,
    "https://www.amazon.com/s?k=kindle",
    { maxPages: 3 },
  );

  const detail = await amazon.scrapeProduct(
    session.page,
    "https://www.amazon.com/dp/B0BCNKKZ91",
  );

  const reviews = await amazon.scrapeReviews(session.page, detail[0].url);
  console.log({ products, detail, reviews });
} finally {
  await session.close();
}

The scraper contract

The scrapers are standardized at the convention level, not forced into one identical interface: every function's first argument is a Playwright Page, each site exposes a scrape* (navigate + extract) and a lower-level parse* (extract from a page you drove yourself), and all return typed structured data. The specific methods differ per site (a Google SERP and an Amazon product page don't share a meaningful method name), so the table below is the source of truth for each module's API.

node

shell

import * as google from "wayfern/scrapers/google";

// Every scraper takes a Playwright `page` as its first argument, so it works
// with ANY Playwright-drivable browser. scrape*() navigate + extract; parse*()
// extract from a page you've already navigated yourself.
const serp = await google.scrapeSerp(page, "anti detect browser", { maxPages: 2 });
const onThisPage = await google.parseSerp(page); // page already on the results

What's included

Site	Module	Functions
Amazon	`amazon`	scrapeSearch · scrapeProduct · scrapeReviews
Google	`google`	scrapeSerp · scrapeKeywords
LinkedIn	`linkedin`	scrapeProfile · scrapeCompany · scrapeJob · scrapeJobSearch
TikTok	`tiktok`	scrapePost · scrapeProfile · scrapeComments · scrapeChannel · scrapeSearch
Instagram	`instagram`	scrapeUser · scrapePost
Walmart	`walmart`	scrapeProduct · scrapeSearch
Zillow	`zillow`	scrapeSearch · scrapeProperty
eBay	`ebay`	scrapeProduct · scrapeProductWithVariants · scrapeSearch
Indeed	`indeed`	scrapeSearch · scrapeJobs
Yelp	`yelp`	scrapeBusiness · scrapeReviews · scrapeSearch

Each module also exports its parse* functions for a page you've already navigated.

Works with any browser, better on the Wayfern engine

Each scraper takes a Playwright Page, so it runs on vanilla Chromium, Firefox or WebKit. Plain Chromium is fine for light use; for production pipelines, scheduled jobs, or scraping at volume, drive the Wayfern anti-detect engine; it ships a consistent humanized fingerprint and rotating residential exits that keep these targets from blocking you. The scraper code is identical; switch by changing one line.

node

shell

import { chromium } from "playwright";
import * as amazon from "wayfern/scrapers/amazon";

// Point Playwright at the Wayfern anti-detect engine (CDP over WebSocket) for a
// consistent humanized fingerprint + rotating residential exits; the scraper
// code is identical to vanilla Chromium.
const browser = await chromium.connectOverCDP(WAYFERN_WS_ENDPOINT);
const page = await (await browser.newContext()).newPage();
const items = await amazon.scrapeSearch(page, "https://www.amazon.com/s?k=kindle");

Framework helpers

When you drive the browser yourself, the ./_framework exports the Scrapfly equivalents:

Scrapfly concept	Wayfern framework
render_js (headless render)	`a real browser, always on`
wait_for_selector	`gotoPage(page, url, { waitForSelector })`
auto_scroll / scroll scenario	`autoScroll(page, { maxScrolls, delay })`
parse __NEXT_DATA__	`scriptJson(page, "script#__NEXT_DATA__")`
application/ld+json	`jsonLd(page)`
regex over result.content	`jsonFromHtml(page, /…json…/)`
xhr_call capture	`captureJson(page, urlPattern)`
authenticated backend request	`browserFetch(page, url, opts)`
concurrent_scrape	`mapLimit(items, limit, fn) + session.newPage()`

Concurrency

Each scraper drives a single tab. To scrape many URLs at once, open multiple pages and bound the concurrency with mapLimit.

node

shell

import { launch, mapLimit } from "wayfern/scrapers";
import * as zillow from "wayfern/scrapers/zillow";

const session = await launch();
const urls = [/* many property URLs */];
try {
  const results = await mapLimit(urls, 4, async (url) => {
    const page = await session.newPage();      // one tab per task
    try {
      return await zillow.scrapeProperty(page, url);
    } finally {
      await page.close();
    }
  });
} finally {
  await session.close();
}

Caveats

These are educational ports of a fast-moving target, so expect to maintain them. Respect each site's Terms of Service and robots.txt, and only scrape data you're permitted to.

Selector drift

Amazon, Google, LinkedIn, eBay and Yelp parse rendered HTML with class/attribute selectors the sites rotate periodically. Expect to maintain them.

Private-API rotation

Instagram rotates its GraphQL doc_id and x-ig-app-id; Yelp rotates its documentId; TikTok signs its XHR. These break when the site changes them.

Auth / consent walls

LinkedIn, Instagram and Google may redirect anonymous traffic to login or consent pages before data renders. A logged-in, humanized session avoids most of this.

Bot challenges

TikTok and Instagram aggressively serve challenges, so the Wayfern engine with residential exits is strongly recommended for these.

API reference → Get an API token