# Turn the Web into LLM ready data

Scrape, crawl, and convert any website into clean Markdown, HTML, images, or structured data with a single API call. Built for RAG pipelines, AI agents, and knowledge bases.

## How it works

### From raw web data to grounded AI in three steps

#### Request

Pass any URL or domain to retrieve HTML, Markdown, images, a full sitemap, or crawl an entire site in one API call.

```javascript
const data = await client.web.scrape({
  url: "example.com"
})
```

#### Context

Include the scraped content in your system prompt or context window alongside your user query.

```javascript
messages: [{
  role: "system",
  content: data.markdown
}]
```

#### Output

Your model answers with real web context — no hallucinations about outdated or missing information.

// Grounded in real data

"Revenue grew 23% YoY to $4.2B, driven by cloud adoption..."

## Endpoints

### Five ways to extract web content

GET `/v1/web/scrape/html`

#### Raw HTML

Full HTML source of any URL. Feed it into a custom parser or extract specific DOM elements before handing off to your model.

#### API Docs

```html
<html>
  <body>
    <h1>Pricing</h1>
    <div class="plan">...
```

GET `/v1/web/scrape/markdown`

#### Markdown

Clean GitHub Flavored Markdown. Strips noise, preserves semantic structure. Cuts token usage dramatically versus raw HTML.

#### API Docs

```markdown
# Pricing

Compare plans and find the right fit for your team.

## Starter
- 10k requests/mo
```

GET `/v1/web/scrape/images`

#### Images

Every image from a page — img tags, inline SVGs, base64 URIs, picture elements, video posters. Returns src, format, and alt text.

GET `/v1/web/scrape/sitemap`

#### Sitemap

Discover up to 500 URLs by crawling sitemaps recursively. Build a URL list before batch-scraping a site into your vector store.

1. /pricing
2. /docs/quickstart
3. /blog/launch-week
4. /changelog
5. /api-reference

POST `/v1/web/crawl`

#### Crawl Website

Crawl up to 500 pages starting from a single URL and extract every page as Markdown. Only follows links within the same domain.

## Built for

### Teams that need live web data in their AI stack

#### RAG Pipeline Engineers

##### Challenge

Keeping LLM knowledge current requires constant re-indexing. Models hallucinate when their training data is outdated relative to live web content.

##### Solution

Use the Markdown endpoint to fetch and embed fresh web pages on demand. Combine with the Sitemap endpoint to bulk-ingest entire domains into your vector store.

##### Impact

Always-current RAG context without manual scraping infrastructure.

#### AI Agent Developers

##### Challenge

Agents need to read web pages, but running a browser per-agent is expensive, slow, and hard to scale.

##### Solution

Replace headless browser calls with single API requests. Get HTML or Markdown with automatic proxy escalation built in.

##### Impact

Agents that browse the web reliably at scale with zero browser infra.

---