I scrapped the old webclaw dashboard and built it again from zero.
now you sign up and you are running in one minute. you see your usage, your runs, you test every feature in the same place.
the engine got faster too, and the web data comes out cleaner even on the pages that block you.
go try it, tell me what breaks 👉 https://t.co/OlNYuYMsgb
+1,500 stars ⭐
I built webclaw to turn any website into clean markdown an LLM can read. It's now a CLI, an MCP server, a REST API, and SDKs for JS, Python, and Go.
Everyone who starred, filed an issue, or sent a PR got it here. Thank you. Plenty left to build.
https://t.co/vAWoCoMQZp
Refer one person to webclaw. Get paid every month they stay.
20% of every payment, no cap, no expiry. Free to join, paid in PayPal, Wise, or bank transfer.
Start earning: https://t.co/w00rA17SUO
I have the entire Product Hunt catalog, from day one to today, enriched with emails and social profiles, and sorted by category. I scraped the whole site with https://t.co/hSOMqHb3zJ. This list could be useful for anyone building tools to generate B2B leads.
https://t.co/OlNYuYMsgb just landed its 6th sponsor: MangoProxy.
Residential, ISP, datacenter, and mobile proxies across 200+ locations. That's the access layer a scraper leans on when sites start blocking you.
Use code 0XMASSI for 8% off their ISP static proxies:
https://t.co/KSQZUUDeMU
One Reddit search gave me 10 people describing a problem my product solves.
Then ~15 lines ranked them by buying intent, so I reply to the ready ones first. One search call, one extract call.
https://t.co/hSOMqHb3zJ + the JS SDK. Code below.
https://t.co/hSOMqHb3zJ just picked up sponsor #5
s/o @nodemaven their proxies are part of what makes the webclaw work, so the support means a lot.
if you scrape anything serious, go check them out 👇
https://t.co/U939YEJ3jZ
AI agents can onboard themselves.
webclaw exposes agent discovery, auth.md, OAuth device flow, MCP, API catalog, OpenAPI, and structured web extraction, so agents can discover, authenticate, scrape, crawl, and extract without a human wiring every step.
https://t.co/pYKADasANG
Today WorkOS is launching auth.md
An open protocol for agents to register for services on the web.
We're partnering with @Cloudflare and @Firecrawl as some of the first providers.
Why did we build this? And why now? 🧵
sitemaps don't show you APIs.
the APIs live in the JavaScript bundles.
new on webclaw:
paste any url, get the full hidden API surface in seconds. graphql, REST, websockets, all of it.
live on Product Hunt today:
https://t.co/nrejfin6L1
Your RAG pipeline does not need 1.8MB of docs app HTML.
It needs clean markdown.
I used https://t.co/ZLepXjmgzE on the LangChain quickstart and turned the page into a compact markdown file that is much easier to chunk, embed, or pass to an agent.
Example:
https://t.co/3bg0Og4BWD
Crawling should feel boring.
Give it a starting URL.
Set the depth.
Let it walk the site and return clean pages.
Recorded a quick webclaw dashboard demo:
25 pages scraped in 1,753ms.
No browser orchestration.
No custom crawler glue.
Just structured web context ready to plug into an agent or RAG pipeline.
https://t.co/gjCMGzEBij
Most scraping demos show one perfect URL.
Real workflows use lists.
I recorded a quick dashboard demo of webclaw v1/batch scraping:
- docs pages
- MCP docs
- changelogs
- pricing pages
Multiple URLs in.
Clean markdown out.
Ready for agents, RAG pipelines, or scripts.
https://t.co/gjCMGzEBij