docpull converts the web into source-linked local context for agent workflows: documentation, reference sites, product pages, changelogs, styleguides, brand profiles, screenshots, schemas, search results, and other crawlable public surfaces.
It is deliberately browser-free by default: async HTTP keeps installs smaller, execution faster, and the crawl surface easier to reason about. Browser rendering, provider-backed search, and paid-capable calls are explicit escalation paths rather than hidden defaults.
pip install docpull
docpull https://example.com
System
- Web HTML becomes Markdown with YAML frontmatter and source URLs
- Crawl profiles support RAG context, full mirrors, fast sampling, and typed local packs
- Pack builders cover brand profiles, styleguides, product pages, schema extraction, images, screenshots, and search
- Depth, page-count, URL-pattern, cache, and robots controls are explicit
- Optional JavaScript rendering and provider-backed search paths for sites and workflows that need them
Runtime
- PyPI CLI for one-off crawls and project setup
- stdio MCP server for agent-driven web context retrieval
- Incremental cache profiles with configurable TTLs
- Markdown, JSONL/NDJSON, SQLite FTS5, OKF bundle, manifest, and context-pack output paths
Proof
- Blocks private IP ranges, CGNAT, and IPv4-mapped IPv6 SSRF paths
- Rejects common binary formats before parsing with magic-byte checks
- Applies URL and file validation before writing crawl artifacts
- Respects robots.txt and HTTPS-only defaults
- Makes local, open, browser, provider, and paid-capable boundaries explicit