Login

grub 🕷 v1.0.0

Agentic web crawler with bot-bypass and mesh P2P.

One container. Crawl any URL — even Cloudflare, Incapsula, or DataDome — and get clean markdown back.
Reads PDFs. Ghost Protocol vision fallback for hard-blocked pages. Speaks MCP to your coding agent.

Run it Add to your agent API Try it Health GitHub
—
Status
—
Tools
Camoufox
Engine
MCP
Transport

Run it

# Full image: Camoufox (stealth Firefox) + Chromium. linux/amd64 + linux/arm64.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest

# Lite image: Chromium only, half the size. Raspberry Pi 5 and other small hosts.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest-lite

# Check it, then crawl something
curl http://localhost:6792/health
curl -X POST http://localhost:6792/api/markdown -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "customer_id": "me"}'

DISABLE_AUTH=true is for your own machine. Leave it unset and put the service behind nuts-auth tokens when it is reachable by anyone else. Scanned-PDF OCR needs a vision provider: Ollama on the host (benhaotang/Nanonets-OCR-s) or an ANTHROPIC_API_KEY / OPENAI_API_KEY.

Add to your agent

grub is an MCP server (streamable HTTP) at /mcp/ with grub_crawl, grub_screenshot and grub_diagnose. Use http://localhost:6792/mcp/ for your own container, or the hosted https://grub.nuts.services/mcp/ with Authorization: Bearer ahp_… from your dashboard.

Claude Code
claude mcp add --transport http grub http://localhost:6792/mcp/
# hosted
claude mcp add --transport http --header "Authorization: Bearer ahp_yourtoken" grub https://grub.nuts.services/mcp/
Cursor · .cursor/mcp.json
{ "mcpServers": { "grub": { "url": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }
VS Code (Copilot) · .vscode/mcp.json
{ "servers": { "grub": { "type": "http", "url": "http://localhost:6792/mcp/",
                       "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }
Claude Desktop · claude_desktop_config.json (via mcp-remote)
{ "mcpServers": { "grub": { "command": "npx",
    "args": ["-y", "mcp-remote", "https://grub.nuts.services/mcp/", "--header", "Authorization: Bearer ahp_yourtoken"] } } }
Codex CLI · ~/.codex/config.toml
[mcp_servers.grub]
url = "https://grub.nuts.services/mcp/"
bearer_token_env_var = "GRUB_TOKEN"   # export GRUB_TOKEN=ahp_yourtoken
Gemini CLI · ~/.gemini/settings.json
{ "mcpServers": { "grub": { "httpUrl": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

Capabilities

Crawl

Playwright + Camoufox stealth. Cookie injection, residential proxy support, automatic Cloudflare/Incapsula challenge solver.

Ghost Protocol

When a page is blocked, screenshot it and extract content via vision LLM. Fallback survives most anti-bot detection.

Markdown

Rust-backed extraction (grub_md) returning clean markdown. Body/visible char counts, content-quality classification.

Agent (Mode B)

Bounded LLM loop: plan → tool call → observe. Anthropic, OpenAI, Ollama providers. Policy gates on every dispatch.

MCP Server

Streamable-HTTP MCP at /mcp/. Exposes grub_crawl, grub_screenshot, grub_diagnose to any MCP client.

Mesh P2P

Optional peer-to-peer mesh — crawl jobs route to the healthiest peer. Zero-trust HMAC + replay-protected nonces.

API

MethodEndpointDescription
POST/api/crawlCrawl URL → HTML + markdown + screenshot
POST/api/markdownCrawl URL → markdown only (faster; PDFs come back page by page)
POST/api/batchCrawl multiple URLs in one request
POST/api/pdf/pagesPDF → per-page text + rendered PNG pages
POST/api/jobs/crawlAsync job — queue and poll for result
POST/api/agent/runRun bounded LLM agent on a task
POST/api/agent/ghostGhost Protocol — vision extract a blocked URL
GET/mcp/MCP streamable-http transport (tools)
GET/healthStatus, version, tool count

All /api/* endpoints require Authorization: Bearer <ahp_token>. Get a token from your dashboard.

Try it

POSTs to /api/markdown with your bearer token. Returns the first 1KB of markdown.

Hosted API examples

# Crawl → markdown
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# With Camoufox warmup (for Cloudflare-protected sites)
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target.com", "options": {"warmup": true}}'

# Inject pre-solved cookies to bypass challenges
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target.com", "options": {"cookies": {"visid_incap_X": "..."}}}'
Docs Health MCP GitHub
grub · DeepBlue Dynamics —