MicroGemLabs/Docs
Sign inSign up free
Browse docs (19)

GemScrape Guide (early access)

347 words · 2 min read · 5 sections

Overview

GemScrape turns web pages into clean, LLM-ready Markdown — fetched,

extracted, semantically chunked, and stored for retrieval. It's the

first product of the MicroGemLabs LLM Suite and is currently in early

access: the ingestion API is live, and the dashboard UI, site crawling,

and JS rendering controls are rolling out.

What it does

Give GemScrape a URL (or raw HTML, or plain text) and it:

1. Fetches the page server-side, SSRF-guarded, with redirect

validation on every hop.

2. Extracts the readable content and converts it to Markdown —

navigation chrome, scripts, and boilerplate stripped.

3. Chunks the Markdown on semantic boundaries (headings first,

size-capped for embeddings) so your RAG pipeline starts at the

interesting part.

4. Stores the chunks in your team's knowledge store, retrievable

by semantic search and full-text search.

Bundling and chunking are free features on every plan, not separate

SKUs.

Using the ingestion API

Ingestion is a single authenticated endpoint. Create a team API key

first (Account → API Keys), then:

# MGL_API_KEY is your team key from Account → API Keys (mgl_team_…)
curl -X POST https://staging.microgemlabs.cloud/api/ingest \
  -H "Authorization: Bearer $MGL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs/getting-started"}'

Accepted payload shapes:

  • {"url": "...", "title": "...", "render": true} — fetch a page;

render runs it in a real browser first for JS-heavy sites.

  • {"html": "...", "sourceId": "...", "title": "..."} — bring your

own HTML.

  • {"text": "...", "sourceId": "..."} — plain text or Markdown.

The endpoint returns 202 Accepted and processes asynchronously.

Requests are rate-limited per key.

Plans and pages

Plans are priced by pages per month. A page = one ingested

document. Unused pages roll over month to month (capped), and you can

buy top-up packs any time — there is no surprise overage billing.

Higher tiers add rendering concurrency for JS-heavy workloads.

See the current tiers on your Products page; the free tier includes

1,000 pages a month.

What's coming

  • Dashboard UI for sources, chunks, and search
  • Site crawling (sitemap-aware, budget-capped)
  • GemBundle: a whole repo or site as one LLM-ready file

Questions or requests? Use the Feedback link — early-access feedback

directly shapes what ships next.

← Agent Skills