Scrapling 🕷️

The best Web Scraping framework that handles everything from a single request to a full-scale crawl by @D4Vinci1
Follow for daily tips & tricks.

[Scrapling 🕷️ @Scrapling_dev]

Scrapling MCP extends the limits of current AI models 🥀

[Karim Shoair @D4Vinci1]

Some CAPTCHAs need more than a screenshot. They need motion and precise mouse control while observing the page.

Scrapling MCP v0.5.1 gives AI models both. 🔥

1. See motion

Scrapling records a 2-second video and sends it to the model as timestamped frames. The model can inspect how the page changes, then use chained browser actions to act.

2. Adjust a slider without letting go

The model can move a slider and ask Scrapling to keep the mouse button held down. It can then inspect the page, adjust the position, and release when ready.

That means it can check and adjust during a drag, before releasing the mouse button.

In the video, I demonstrate this on hCaptcha, DataDome, and GeeTest using Codex with Astra at light effort. 🚀

Scrapling v0.5.1 is available now for free. Link below 👇

Let me know what you think.

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: Network history spans page loads. Use search(after_id=cursor) to read only new saved API replies, then advance the cursor with reply.meta["network_id"]. No more reading page 1's replies again on page 2. ⚡

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: page_action lets you read nested frames that response.css() cannot see. Get the frame's HTML with frame.content(), then parse it with Selector. Reuse the loaded browser page without a second HTTP fetch. 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: The adaptive feature has a match cutoff: percentage=80. If a price vanishes, the closest match may be "Sold out". This demo rejects that match. Tune the cutoff on real pages; it is a similarity score, not confidence. 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: Two prices in one tag? Pass ignore_tags=("del", "s") to get_all_text() to skip the crossed-out price. It keeps the text right after </del>, even the lone "$" in this example. You get "$29.95" in one call. 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: Paginating a JSON API with POST? Keep the spider's duplicate filter on. It checks the JSON body as well as the URL. Send page 1, page 2, then page 2 again to the same endpoint. Only two requests reach the server. 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: You can use a regex inside XPath. Bind the EXSLT regex namespace and use re:test() to match a SKU on each row. One query gets Lamp and Desk and skips the Flash sale row. 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: Two GraphQL calls hit /graphql. Which reply has the price? Turn on record_requests=True. Each saved reply carries its POST body in meta["request_body"]. Match the operationName and read the price without sending the query again 🎯

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: RemoteSession(new_context=False) uses an open browser's login state. Connect to a browser with remote debugging enabled, then scrape without exporting cookies. When the session ends, your existing tabs stay open 🎯

[Scrapling 🕷️ @Scrapling_dev]

@AISuperDomain Thanks for writing about us!

Here’s a Chinese version of our docs: https://t.co/krKFSXdi8y

[Scrapling 🕷️ @Scrapling_dev]

@rammcodes Thanks for writing about us!

Here’s the full docs: https://t.co/9kmq7Sx9zP

[Scrapling 🕷️ @Scrapling_dev]

@clayul Thanks for writing about us!

Here’s the full docs in Spanish:

https://t.co/kaiEqm7kAR

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: stream() has a 100-item buffer. If your writer falls behind, item producers wait when it fills. Save each result as it arrives instead of keeping the full result list in RAM. This example writes one JSON record per line ⚡

[Scrapling 🕷️ @Scrapling_dev]

@techNmak Thanks for the mention!

Here’s the full docs: https://t.co/9kmq7SxHpn

[Scrapling 🕷️ @Scrapling_dev]

@midudev Thanks for the mention!

Here’s the Spanish version of the docs: https://t.co/kaiEqm7kAR

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: AutoThrottle accepts a target below 1. Set autothrottle_target_concurrency=0.5 to move each site's delay toward twice its response time, within your delay limits. Slow sites get more breathing room without a fixed sleep. ⚡

[Scrapling 🕷️ @Scrapling_dev]

Now you know 🔥

[Karim Shoair @D4Vinci1]

So what makes Scrapling MCP better than Playwright MCP and other known free MCPs that automate browsers or scrape the web?

Scrapling MCP in v0.5 can achieve the same browser automation as others, but:

- It provides Chainable browser actions. Unlike any other MCP, the agent makes one call to the MCP to take an action in the browser, then thinks, then makes another call for the following action, and so on. In Scrapling, the agent can send a sequence of actions to run in the browser without repeating calls, saving tokens and letting the agent plan ahead.

For example, a website that shows a game to test whether you are human, based on your quick reactions, can be solved this way. The actions cover all waiting conditions, so a sequence can include clicking a button, waiting 5 seconds, ensuring the network is stable, typing in an input field, pressing Enter, waiting for a specific selector to appear, waiting for all JavaScript to execute, and then finishing.
Imagine Scrapling doing all of that for the Agent instead of the agent sending the calls one by one!

- The MCP can see and interact with closed shadow-DOM elements like CAPTCHAs. I don't think this needs explaining, but it means you can rest assured the agent can interact with anything the website throws at you. Previously, you had to give your agent access to computer-use add-ons so it could see the screen and interact with elements like this through coordinates; now it can do it directly through the browser.

- There is a lot of focus on saving tokens! Here, we block ads; let you select elements instead of the whole page for all operations; let you save browser state and repeat it later; and protect against prompt injection better than most tools. For example, we delete all hidden elements and zero-width elements, so we save more tokens while protecting you more!

- Stuff we already had from before:
* We provide a Cloudflare solver that doesn't require AI.
* You can do requests through Scrapling without browsers that fake TLS fingerprints, so you only use browsers when you have to (and with a lot of control).
* Control over what reaches the AI. Extract selected elements as Markdown, HTML, or text. Search page snapshots or read captured API responses. This can reduce unnecessary output.

Other than that, you will find the things that you already see in other tools, like moving and clicking the mouse, using the keyboard and typing slowly, interacting with dropboxes and similar elements, doing drag and drop with the mouse, executing JavaScript on the page, interacting with dialog popups, seeing the requests that are happening in the background and the parts that the Agent is interested in each request, and full control over the type of content returned to the agent to be HTML/Markdown/Text content and so on.

Now tell me, have you tried this update yet, and what do you think? 🔥

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: retry_blocked_request() can switch a blocked HTTP request to a stealth browser. Add that browser with lazy=True and it stays closed until needed. Fast pages use HTTP; blocked pages get one browser retry in this example ⚡

[Scrapling 🕷️ @Scrapling_dev]

A big clap for our new Platinum Partner @nonecapapi
👏
Check them out: https://t.co/uSzMABLaoI

[Scrapling 🕷️ @Scrapling_dev]

🕷️ Scrapling tip: on_scraped_item() can remove duplicate products from different URLs. Track each SKU in a set and return None for repeats. They stay out of results and stream(), while items_dropped counts them. This set lasts for one run 🎯