@jacewbuilds Good question, so I tested it. Code fences and nested lists come through fine on the Rust Book and MDN, with the language tag kept. Python docs were the exception: their code blocks came out unfenced. Fixed that now, they come through as proper python blocks
I built a web scraper that snarfs a whole website into clean Markdown.
113 pages in 0.34 s. Fastest median latency on the WebCrawlerAPI benchmark (1.15 s per page).
Rust, open source, plugs straight into AI agents over MCP.
https://t.co/A8qyEgBv8n
I built a web scraper that snarfs a whole website into clean Markdown.
113 pages in 0.34 s. Fastest median latency on the WebCrawlerAPI benchmark (1.15 s per page).
Rust, open source, plugs straight into AI agents over MCP.
https://t.co/A8qyEgBv8n
@stretchcloud data access turning into a cloud bill line item is lowkey why i built snarf, rust scraper that runs locally and plugs into gemini cli over mcp the same way, bill stays at zero https://t.co/wUAqo6rNUW
I built a web scraper that snarfs a whole website into clean Markdown.
113 pages in 0.34 s. Fastest median latency on the WebCrawlerAPI benchmark (1.15 s per page).
Rust, open source, plugs straight into AI agents over MCP.
https://t.co/A8qyEgBv8n
@ds_fafa_ messiest source i threw at mine was the whole rust book, 113 pages in 0.34s, snarf is open source if anyone wants the free version of this https://t.co/wUAqo6rNUW
I built a web scraper that snarfs a whole website into clean Markdown.
113 pages in 0.34 s. Fastest median latency on the WebCrawlerAPI benchmark (1.15 s per page).
Rust, open source, plugs straight into AI agents over MCP.
https://t.co/A8qyEgBv8n
4/ Give it to your agent:
cargo install --git https://t.co/A8qyEgBv8n snarf
claude mcp add snarf -- snarf mcp
Your agent gets scrape, batch_scrape, crawl and map as tools.
3/ Why it's fast: no browser unless a page needs one, a single streaming HTML pass, and failures fail fast. Dead hosts and captcha walls never wait on a browser.