Crawl4AI

Blazing fast, open-source, and LLM-friendly crawler built for the modern web.

[Crawl4AI @crawl4ai]

🚀 Crawl4AI Cloud API (Closed Beta) is coming soon.
Built for reliable, large-scale web extraction and drastically more cost-effective than existing solutions.

Apply for early access:
https://t.co/LzBRygv3aQ

[Crawl4AI @crawl4ai]

Crawl4AI 0.9.3 is out. 🚀

Security release. Five advisories, reported privately and fixed before disclosure. Four in the PDF path, one in the Docker Playground.

SSRF through PDF redirects. Unbounded PDF downloads. Unescaped PDF text. File write from untrusted request bodies. DOM XSS in the Playground. All closed.

Plus 33 bug fixes since 0.9.2. PDF scraping works out of the box now.

pip install -U crawl4ai docker pull unclecode/crawl4ai:0.9.3

https://t.co/fe4PRmNt6a

Thank you to Jace Sun, Nguyen Tran Thanh Lam, and e1codes. Report it privately, we will treat it right.

Star the repo. A hosted version is coming.

Live long and import crawl4ai 🖖

[Crawl4AI @crawl4ai]

The World Cup final is today. So we built one feed that watches everything: live score, breaking news, what fans are feeling, even shirt prices moving in real time.

Powered by the new Crawl4AI Cloud. Deliberately overfitted on the World Cup, any trend is next.

Check it out: https://t.co/aCPqedHzwH

[Unclecode (Hossein) @unclecode]

Crawl4AI v0.8.7 is live 🚀

Mostly a security hardening release. Patched several critical Docker API issues reported responsibly by the community: pre-auth RCE, SSRF on crawl and webhook endpoints, monitor auth bypass, arbitrary file write, stored XSS, unauthenticated JS execution, and a hardcoded JWT secret. Big thanks to every researcher who disclosed.

Also new: DomainMapper for full-domain URL discovery, and per-URL configs for arun_many in the Docker API.

Plus a batch of fixes across markdown fidelity (mermaid, tables, chunking), deep crawl, dispatcher, LLM extraction, MCP CJK handling, stealth, and logging.

If you self-host the Docker API, upgrade now:

pip install -U crawl4ai

Release Note: https://t.co/dlCjlcvtHT

Star it, use it, break it: https://t.co/6apdISAYzn

Docs:
https://t.co/6mkQsKdRLf

C4AI Hosted version, coming soon. 😎

Live long and import crawl4ai 🖖

[Unclecode (Hossein) @unclecode]

🎉 Update: v0.8.6 is out! And good news on the litellm front, I've published a clean drop-in replacement: pip install unclecode-litellm==1.81.13

crawl4ai now installs from this package until @LiteLLM gets it sorted. Honestly, I like this library and hope they fix & maintain it soon, but in the meantime I'm planning to make this a much leaner version maintained specifically for @crawl4ai and those of you want a tiny version of litellm. Tiny-lite-llm, here we come 😄

pip install crawl4ai==0.8.6

[Unclecode (Hossein) @unclecode]

🚨 PSA: litellm (@LiteLLM) is compromised on PyPI (v1.82.7 & 1.82.8 steal credentials). Can't pip install right now. @crawl4ai v0.8.6 dropping shortly with a clean fork. Check your version: "pip show litellm" my team is on it, stay tuned. Until then: pip install git+https://t.co/J54fJAWH1H
Details: https://t.co/5SWzm7wi8W

[Crawl4AI @crawl4ai]

Hi @peeefour thanks for flagging this and sorry the email didn’t go through.
Could you please email Unclecode directly ([email protected]) and CC ([email protected] & [email protected])? We’ll prioritise reviewing it immediately.
Appreciate you reporting this responsibly.

[Crawl4AI @crawl4ai]

We’re excited to welcome @ThordataTeam as an official sponsor of Crawl4AI!
Thordata delivers a global, high-performance web data platform built for AI teams that need speed + scale.

What you get:
🔹 AI-native crawling infra
🔹 99.9 percent uptime
🔹 Plug-and-play AI/ML integration
🔹 Massive, scalable data access
🔹 1-on-1 expert support

[Onur Oztaskiran @onuro]

Just built this @crawl4ai @n8n_io node for one shot scraper with LLM extractor. Single node, many possibilities. Also supports multiple urls.

Using it to daily scrape all landing page inspiration sites' links and save it to my stash.

Will publish the n8n community node shortly

[Crawl4AI @crawl4ai]

Join us for a special @crawl4ai x @Scrapelessteam Meetup this Friday!

- Live demo script
- Real-world code samples
- Tips for scaling Crawl4AI with Scrapeless

📅 Dec 5
🕒 3PM UTC+8
📍 Discord

Mark your calendar, you don’t want to miss this.
👉 https://t.co/AtZ61UNlsJ

#Crawl4AI #Scrapeless #opensource #AItools #webscraping

[Crawl4AI @crawl4ai]

@nstproxy @nstproxy is also running a major Black Friday campaign with discounts on Residential, ISP, Datacenter, and IPv6 proxy plans.
Details here: https://t.co/MJyWcJ6c0E

[Crawl4AI @crawl4ai]

Crawl4AI is excited to welcome @nstproxy as our latest ecosystem partner.
NSTProxy is a leading proxy provider trusted by global developers for web scraping, automation, and AI data pipelines, delivering 110M+ real residential IPs, city-level targeting, 99.99% uptime, and pricing from $0.1/GB.
A strong fit for developers building with Crawl4AI at scale.

Find integration examples here: https://t.co/s1YeKCVFwI

[Unclecode (Hossein) @unclecode]

🎉 Crawl4AI 0.7.7 is live, Self Hosting is the KING and QUEEN 🤴 🫅

Self hosting just leveled up. This release turns @crawl4ai into a real platform, not a library. You get a monitoring dashboard baked into Docker, open slash dashboard and you see the entire system breathing in real time, CPU, memory, active requests, browser pool state, errors, cleanup cycles, all streaming every two seconds over WebSocket.

There is a full Monitor API now, everything exposed as simple REST. Trigger cleanup, kill frozen browsers, pull performance metrics, plug into Prometheus, wire it into your own infra without fighting the black box problem that every crawler suffers from.

The smart browser pool went through a serious upgrade. Permanent, hot, cold pools that actually learn your usage pattern and cut memory use by an order of magnitude. Infra feels lighter, faster, cleaner.

Critical fixes landed too, async extraction stalls, viewport quirks, DFS traversal edge cases, all hammered down.

This release closes a loop. Crawl4AI now behaves like a self hosting engine with proper observability, privacy first, predictable cost, and operational clarity. You get the power without the enterprise tax.

Next steps stay ambitious, hosted API, distributed crawlers, self contained binaries, and a path toward cheap AI infra for everyone. This version is a foundation, not a peak.

Massive respect to the 56k plus who starred the project, shipped patches, tested weird edge cases, and supported the vision. OSS only works when the community actually shows up.

🔗 GitHub: https://t.co/6apdISAqJP
💬 Discord: https://t.co/waAsHDjpli
📖 Release Notes: https://t.co/TATMms1Zei

Special thanks to sponsors
@am_tambe, @vinayprabhu, @capsolver

And to the individual supporters
Martin Sjöborg
Romek Rozen
Kourosh Kiyani
Max Bodewes

Everyone who filed a bug, fixed a line, or pushed the platform forward left fingerprints on this release. You made 0.7.7 real.

[Crawl4AI @crawl4ai]

We’re excited to announce that @Scrapelessteam is now an official sponsor of Crawl4AI, the #1 trending GitHub repository for blazing-fast, AI-ready web crawling! ⚡
This partnership strengthens our shared mission: to make intelligent, large-scale web crawling faster, more reliable, and accessible to everyone.
Scrapeless provides production-grade infrastructure for Crawling, Automation, and AI Agents, including:

Scraping Browser — a cloud-based browser built for automated workflows and large-scale data extraction, offering high-concurrency performance, low-latency session isolation, and advanced stealth fingerprinting to bypass modern anti-scraping defenses.

4 Proxy Types — Residential, ISP, Datacenter, and IPv6 proxies, giving developers flexible routing and reliable access across regions and network types.

Universal Scraping API — helps you bypass website blocks in real time and fetch data faster, with built-in support for dynamic content and anti-bot handling.

Supports data customization — offering a variety of enterprise data solutions, including tailored approaches for AI chat platforms such as Perplexity and ChatGPT.

Together, Crawl4AI and Scrapeless create a powerful ecosystem for AI-driven data collection — open, extensible, and developer-friendly.
Stay tuned as we release integration examples, best-practice workflows, and ready-to-use templates showing how Scrapeless tools can supercharge your Crawl4AI pipelines.

[Tu Ba Khuym @TuBaKhuym]

Day 4:
Feed @crawl4ai docs to @FactoryAI also with targeted sites. Droid nailed it, extracted the admin procedure documents after 30 minutes on autopilots. Next step would be cleaning up the documents.

[Unclecode (Hossein) @unclecode]

Just added a "Copy Page" button to @crawl4ai docs!
Now you can instantly copy any documentation page as markdown and feed it to your favorite AI coding assistant.

Check it out → https://t.co/6mkQsKdRLf

#crawl4ai

[Crawl4AI @crawl4ai]

CAPTCHA challenges are one of the biggest blockers in scalable web data workflows.

By combining Crawl4AI’s open-source crawling framework with CapSolver’s reliable CAPTCHA-solving, developers now have a more robust tech stack for continuous, scalable web data automation.

We’re excited to partner with @capsolver on this integration! 🎉 Huge thanks to the @capsolver team for sponsoring Crawl4AI’s open-source mission🙌.

[CapSolver @capsolver]

CapSolver × @crawl4ai

🙌🏻Together we bring faster, smarter, and more reliable Captcha solving for AI-driven crawling.

📑Read more about the integration tutorial:
https://t.co/SyTS1vmaW0

[Crawl4AI @crawl4ai]

🚀 New Meetup Recording!

We put @n8n_io + Crawl4AI to work:
✨ Watch for stars on a GitHub repo
🔍 Scrape the starrer’s GitHub profile for contact info
🤝 If they’ve listed LinkedIn → automatically trigger a connection request via n8n and Crawl4AI

Demo built by our intern @sohamkukreti 👏

📺 Watch here: https://t.co/62S00ffeFY

Big thanks to @n8n_io for powering smooth automation 💙

Live long & import crawl4ai 🖖

[Crawl4AI @crawl4ai]

🚀 Exciting news!

We’re thrilled to welcome Kipo AI as a sponsor of Crawl4AI 🎉

Kipo AI helps engineers & buyers find, compare, and source electronic/industrial parts in seconds.

With Crawl4AI, they’re building a massive database of components & specs by crawling thousands of supplier sites and structuring millions of spec sheets.

Here’s to powering innovation together with @am_tambe & @vinayprabhu ⚡️

#opensource #AI #crawl4ai #electronics #sourcing