Turn the Web into LLM ready data

Scrape, crawl, and convert any website into clean Markdown, HTML, images, or structured data with a single API call. Built for RAG pipelines, AI agents, and knowledge bases.

How it works

From raw web data to grounded AI in three steps

Request

Pass any URL or domain to retrieve HTML, Markdown, images, a full sitemap, or crawl an entire site in one API call.

const data = await client.web.scrape({
  url: "example.com"
})

Context

Include the scraped content in your system prompt or context window alongside your user query.

messages: [{
  role: "system",
  content: data.markdown
}]

Output

Your model answers with real web context — no hallucinations about outdated or missing information.

// Grounded in real data

"Revenue grew 23% YoY to $4.2B, driven by cloud adoption..."

Endpoints

Five ways to extract web content

GET /v1/web/scrape/html

Raw HTML

Full HTML source of any URL. Feed it into a custom parser or extract specific DOM elements before handing off to your model.

API Docs

<html>
  <body>
    <h1>Pricing</h1>
    <div class="plan">...

GET /v1/web/scrape/markdown

Markdown

Clean GitHub Flavored Markdown. Strips noise, preserves semantic structure. Cuts token usage dramatically versus raw HTML.

API Docs

# Pricing

Compare plans and find the right fit for your team.

## Starter
- 10k requests/mo

GET /v1/web/scrape/images

Images

Every image from a page — img tags, inline SVGs, base64 URIs, picture elements, video posters. Returns src, format, and alt text.

GET /v1/web/scrape/sitemap

Sitemap

Discover up to 500 URLs by crawling sitemaps recursively. Build a URL list before batch-scraping a site into your vector store.

  1. /pricing
  2. /docs/quickstart
  3. /blog/launch-week
  4. /changelog
  5. /api-reference

POST /v1/web/crawl

Crawl Website

Crawl up to 500 pages starting from a single URL and extract every page as Markdown. Only follows links within the same domain.

Built for

Teams that need live web data in their AI stack

RAG Pipeline Engineers

Challenge

Keeping LLM knowledge current requires constant re-indexing. Models hallucinate when their training data is outdated relative to live web content.

Solution

Use the Markdown endpoint to fetch and embed fresh web pages on demand. Combine with the Sitemap endpoint to bulk-ingest entire domains into your vector store.

Impact

Always-current RAG context without manual scraping infrastructure.

AI Agent Developers

Challenge

Agents need to read web pages, but running a browser per-agent is expensive, slow, and hard to scale.

Solution

Replace headless browser calls with single API requests. Get HTML or Markdown with automatic proxy escalation built in.

Impact

Agents that browse the web reliably at scale with zero browser infra.