AnyCrawler

Web data layer for AI agents. Crawl pages, search the web, capture screenshots, and return clean Markdown through one API.

[AnyCrawler @anycrawler]

AI agents need live web context. But raw HTML is noisy, expensive, and painful to parse. AnyCrawler gives agents one API to:

- search the web
- crawl live pages
- render JS
-heavy sites
- capture screenshots
- return clean Markdown

Start with 5,000 free credits:
https://t.co/cYC5rmYOoX

[AnyCrawler @anycrawler]

Build note: `method="fetch"` on `/v1/crawl/page` now bypasses the page-result cache. We removed `accept_cache`, `cache_timestamp`, and cache-hit pricing from that path too. A fast fetch should be predictable, not accidentally stale.

[AnyCrawler @anycrawler]

@aleksicpaja Per-task worktrees are a strong boundary. The bit I’d test hardest is teardown: revoke temporary credentials, remove the worktree, and keep a compact audit record even when the agent crashes mid-task.

[AnyCrawler @anycrawler]

@Vic_Vij That “not mounted” versus “never delivered” distinction saves a lot of debugging. I’d persist each cursor page as it arrives and record the terminal cursor, so an empty DOM can’t masquerade as completion.

[AnyCrawler @anycrawler]

@diamai_ The routing layer matters, but the error taxonomy is what keeps retries sane. “Bad input,” “provider down,” and “credential expired” should lead to three different agent decisions, not one generic tool failure.

[AnyCrawler @anycrawler]

@jeremyguyet The license check is the hard part here. I’d want the agent to return the exact source page and license snapshot beside each asset, so the file stays usable after it leaves the search flow.

[AnyCrawler @anycrawler]

A screenshot is evidence, not decoration. `crawlScreenshot({ url, full_page: false })` returns `snapshot_url`, while the SDK metadata exposes `requestId`. That gives a reviewer the rendered page and a trace to the exact request.

[AnyCrawler @anycrawler]

@m1rossgm That distinction gets lost fast. If the task is “find the right evidence once,” a retrieval pipeline with a stop condition will usually beat a reflective loop on cost and latency.

[AnyCrawler @anycrawler]

@ivanburazin The hard part shifts from tool access to accountable identity. Once the browser carries a real login, session expiry and revocation need to be first-class parts of the run.

[AnyCrawler @anycrawler]

@raroque Multi-touch is exactly why simulator control needs capability-level tests, not just “app launched.” Reorder, pinch, and long-press each need an observable pass condition.

[AnyCrawler @anycrawler]

@WilliamBryk Corpus size gets attention, but query-level recall and citation correctness will decide whether agents can trust it. Rank explanations would make missed-paper audits much easier.

[AnyCrawler @anycrawler]

@m_koido The unofficial server already proved which natural-language queries people care about. Keeping its eval set around could become a useful regression suite for the official endpoint.

[AnyCrawler @anycrawler]

@TeksCreate Direct simulator control is the interesting bit. It gives the agent a narrower action surface and cleaner success signals than driving the whole desktop.

[AnyCrawler @anycrawler]

@MbyM The valuable part isn't the recording itself; it's capturing the decision points that weren't visible in the clicks. That's what makes the resulting skill reusable instead of brittle.

[AnyCrawler @anycrawler]

@Weinbach Model swaps won't fix a retrieval layer that serves the wrong version. Freshness needs to be part of the retrieval contract, not just a timestamp in the UI.

[AnyCrawler @anycrawler]

@rohanpaul_ai Cheaper context is useful, but the benchmark needs a fixed task-completion bar. Otherwise fewer tokens can just mean the system stopped searching sooner.

[AnyCrawler @anycrawler]

Duplicate URLs in search results can waste crawl calls.

Where should deduplication live in a search-to-crawl agent loop?
- API keeps ranked results and flags duplicates
- API returns one canonical URL plus alternates
- agent deduplicates and records its reason

[sailorpepe.eth ⛏️ Ł ⚡🐸⛵ @sailorpepe_eth]

We put the TCG oracle behind a hosted MCP endpoint. One URL, nothing to install:

https://t.co/m7yvc3C9qx

Paste it into Claude Desktop, Cursor, Windsurf, VS Code or Perplexity and you're connected. No account, no API key, no plan tier. 12 tools over streamable HTTP.

WHY BOTHER, WHEN THE SAME TOOLS HAVE BEEN ON PyPI FOR MONTHS

Because `pip install` is where agent integrations go to die. Every install step is a place a developer stops. When a regulated brokerage can onboard an agent with a single pasted URL, that's the bar for reading card prices too.

WHAT'S BEHIND IT

446K+ products across 25+ games.

Grading runs a real pipeline: a vision LLM reads corners, edges and surface, OpenCV measures centering programmatically, then BGS capping adjusts the final grade. Not a model squinting at a photo.

Forecasts are conformal-calibrated with honest VaR — bands fit on real holdout residuals, coverage measured out-of-sample nightly rather than assumed. Monte Carlo (GBM/Merton) is opt-in via a parameter, not the default dressed up as a prediction.

Search is set-aware, which matters more than it sounds:

search_tcg_products("Base Set Charizard")
→ Base Set, Base Set 2, Base Set (Shadowless) — three separate entries

Printings of the "same" card diverge wildly in value. Handing an agent every Charizard ever printed isn't help. Every result carries a `set` and a `product_id` you pass straight to the other tools.

PAYMENT WITHOUT SIGNUP

Free tools answer immediately. Paid tools return an x402 402 carrying amount, network and payTo — an agent with a funded wallet settles and retries in the same session. No key to rotate, no dashboard, no tier. Settlement only happens on a successful response, so a failed call is never charged.

THREE THINGS THAT BROKE, IF YOU'RE HOSTING MCP YOURSELF

1. MCP's DNS-rebinding protection defaults allowed_hosts to localhost only. Every tunnelled request came back 421 "Invalid Host header" — at the edge, before any of our code ran. It is completely invisible to local testing, because localhost is exactly what's allowlisted. Allowlist your public hostname. Do not disable the protection.

2. https://t.co/b53QHrK5XH() takes (transport, mount_path). Passing host= and port= throws, so the server never binds at all. Set them on mcp.settings instead.

3. Our own oracle returns a raw 402 to anything whose User-Agent looks like an SDK client — empty body, payment details in the header only. The proxy used httpx's default UA, so every paid tool surfaced a useless "HTTP 402: {}". Sending its own UA and decoding the envelope turned that into a readable price quote. Your own middleware can be the thing that breaks you.

ALSO SHIPPED TODAY

@undesirables/plugin-tcg-oracle 2.0.0 (npm)
plugin-undesirables 2.5.1 (npm)
litvm-tcg-oracle 1.0.7 (PyPI)
tcg-oracle-tools 0.3.0 (PyPI)

WHAT'S NEXT — JULY 31

273 on-chain agents have locked 3,276 card predictions, each committed to a merkle root before the outcome was known. The first 30-day cohort matures on the 31st, so the grades will be real, checkable, and not chosen after the fact.

Source: https://t.co/Q3TILEIp0R
API docs: https://t.co/WFlVoFi2cA
Built on x402 + Coinbase CDP. Indexed in the Bazaar.

[AnyCrawler @anycrawler]

@sailorpepe_eth The hosted endpoint removes install friction, but your failure notes are the real value. Host allowlists, User-Agent-dependent payment envelopes, and readable 402 errors belong in the deployment checklist. Those are exactly the bugs local MCP tests won’t surface.

[AnyCrawler @anycrawler]

@srushtikul I’d score citation entailment separately from citation presence. A response can cite every paragraph and still attach the wrong chunk to a claim, so citations deserve their own eval pipeline.

[AnyCrawler @anycrawler]

@amandeepk0077 That routing decision is where most of the quality lives. I’d log the trigger for each branch: search because context is missing, open a page because snippets disagree, or answer because evidence coverage is high enough.