Turn any URL into clean markdown.

One-click install for AI agents

scrapefold ships a CLI (scrapefold) and an MCP server (scrapefold-mcp) with four tools — scrape_url, crawl_site, list_engines, classify_url. All four definitions cost your agent ≈750 tokens (budget-tested). Need one fact from a long page? scrape_url(focus="query") returns only the relevant blocks. Failures come back structured — never error-page HTML posing as content. One command registers it in your client:

$ pip install "scrapefold[mcp]"
$ scrapefold install claude   # Claude Code
$ scrapefold install codex    # Codex CLI
$ scrapefold install cursor   # Cursor
$ scrapefold install vscode   # VS Code

Any other client: paste the config JSON above into its MCP settings. Full agent instructions live at scrapefold.com/install.md — fetchable by any agent.

Why Scrapefold?

Every scraping vendor has trade-offs. Scrapefold lets you switch between them with one line — and escalates from free local engines to paid APIs only as far as a site forces it.

30 engines, one interface

Local engines are free and fast; SaaS engines add premium proxies and stealth. The router picks the cheapest tier that works. Ratings: ★★★ excellent · ★★☆ good · ★☆☆ basic.

How to choose

Or skip the decision entirely — call scrape(url) and let the router pick.

• Static blog or documentation siterequests — zero deps, sub-second
• JS-rendered SPA, no anti-botscrapling_fast (free) or Jina Reader (free tier)
• Cloudflare / Datadome / PerimeterXscrapling_stealth (free) → Firecrawl / ScrapingBee (paid)
• Site that emits clean markdown via APIJina Reader — direct markdown, no parsing
• Visual layouts, tables, charts, or screenshotsPixelRAG — local pixelshot tiles + reader markdown / JSON
• LinkedIn / niche socialExa public people/company search + Apify (LinkedIn) actor fallback
• Structured fields straight from a pageScraperAPI — AI Parser fills the json slot
• IP-geofenced targetsOxylabs — residential pool + geo_location
• Page is gone (404) or paywalled nowwayback — archive.org snapshot, marked source=archive.org
• Need an MCP server for AI agentsscrapefold-mcp — built-in; register with scrapefold install claude

Quickstart

Install one extra per vendor, or scrapefold[all] for everything.

import asyncio
from scrapefold import scrape, crawl_site, ScrapeOptions

async def main():
    # Single URL, auto-engine — router picks the cheapest tier that works
    result = await scrape("https://example.com")
    print(result.markdown)        # always populated
    print(result.engine)          # which engine actually fetched it

    # Cloudflare-protected site — same call, router auto-escalates
    result = await scrape(
        "https://protected.example.com",
        opts=ScrapeOptions(render_js=True, stealth=True),
    )

    # Whole-site crawl with disk cache
    crawl = await crawl_site(
        "https://docs.example.com",
        opts=ScrapeOptions(max_pages=50, max_depth=3),
        output="site.md",
    )

asyncio.run(main())

# CLI
$ scrapefold scrape https://example.com
$ scrapefold scrape https://example.com --focus "pricing"  # only relevant blocks — saves tokens
$ scrapefold crawl https://docs.example.com --max-pages 50 --output site.md
$ scrapefold list-engines
$ scrapefold doctor            # health check: engines, MCP extra
$ scrapefold update --check    # self-update via PyPI

Built by & ecosystem

Scrapefold is built and maintained by Mike Sadofyev (CEO, Datatera.ai) — the scraping engine behind Datatera — alongside a small ecosystem of AI-data tooling. Connect on LinkedIn, X, or GitHub.