GemScrape Guide (early access)

GemScrape turns web pages into clean, LLM-ready Markdown — fetched,

extracted, and (optionally) crawled across a whole site. It's the first

product of the MicroGemLabs LLM Suite. Call it over REST, from the

mgl CLI, or as MCP tools your agent can use directly.

Give GemScrape a URL and it:

1. Fetches the page server-side, SSRF-guarded, with redirect

validation on every hop (render: true runs it in a real browser

first for JavaScript-heavy sites).

2. Extracts the readable content and converts it to Markdown —

navigation chrome, scripts, and boilerplate stripped, headings, code,

and tables kept.

Results are returned to you — GemScrape is passthrough and doesn't

store your scraped content. Crawling, sitemap mapping, and chunking are

included; bundling and chunking are free on every plan, not separate

SKUs.

Create a team API key (Account → API Keys), then every call uses it as a

Bearer token:

export MGL_API_KEY=mgl_team_...   # from Account → API Keys
BASE=https://staging.microgemlabs.cloud

Scrape one URL → clean Markdown (charges 1 page):

curl -X POST $BASE/api/v1/scrape \
  -H "Authorization: Bearer $MGL_API_KEY" \
  -d '{"url": "https://docs.example.com/guide", "render": false}'
# → { "markdown": "# …", "meta": { "url", "render", "bytes" }, "remaining": 999 }

Map a site — its sitemap URLs (free, no page charge):

curl -X POST $BASE/api/v1/map \
  -H "Authorization: Bearer $MGL_API_KEY" \
  -d '{"url": "https://docs.example.com"}'
# → { "urls": [...], "source": "sitemap", "count": 42 }

Crawl a site — sitemap-first, else follow same-domain links, 1 page

each. Async: returns a job id to poll.

curl -X POST $BASE/api/v1/crawl \
  -H "Authorization: Bearer $MGL_API_KEY" \
  -d '{"url": "https://docs.example.com", "maxPages": 50}'
# → 202 { "jobId": "…", "status": "queued" }

curl $BASE/api/v1/crawl/<jobId> -H "Authorization: Bearer $MGL_API_KEY"
# → { "status": "done", "pagesDone", "pages": [ { "url", "markdown" }, … ] }

Chunk Markdown for RAG (free):

curl -X POST $BASE/api/v1/chunk \
  -H "Authorization: Bearer $MGL_API_KEY" \
  -d '{"markdown": "# Title\n\n…", "maxChars": 2400}'
# → { "chunks": [ { "content", "headingPath", "approxTokens" }, … ], "count": 7 }

Usage — your page balance this period:

curl $BASE/api/v1/usage -H "Authorization: Bearer $MGL_API_KEY"
# → { "included", "rollover", "topup", "used", "available", "periodEnd" }

Unix-friendly — scrape writes Markdown to stdout, chunk reads stdin,

so they pipe:

mgl login                                   # paste your mgl_team_… key
mgl scrape https://docs.example.com         # clean Markdown to stdout
mgl scrape https://docs.example.com | mgl chunk --json > chunks.json
mgl map https://docs.example.com            # sitemap URLs, one per line
mgl crawl https://docs.example.com --wait   # crawl a whole site
mgl usage                                   # page balance

Point any MCP client (Claude, Cursor, your own agent) at

/api/mcp with the same mgl_team_… token and it gets the tools scrape_url, map_site, and crawl_site. Same engine, same page pool

as REST — your MCP tool list shows only what your plan includes.

Priced by pages per month — one scraped or crawled page = 1 page,

flat, whether static or JS-rendered (no crawl/extract multipliers).

Unused pages roll over month to month (capped), and you can buy $5

top-up packs any time — no surprise overage billing. Higher tiers add

rendering concurrency for JS-heavy workloads. The free tier includes

1,000 pages/month; see the current tiers on your Products page.

• In-app dashboard for crawls and usage
• GemBundle: a whole repo or site as one LLM-ready file

Questions or requests? Use the Feedback link — early-access feedback

directly shapes what ships next.