Rishabh

Principal Product & Venture Builder | 0→1 Applied AI, Enterprise Deployment & Company Formation | AI Fund

[Rishabh @rishksing]

@gregisenberg Great thoughts! I built sharpbench as a way to rank connectors and services that an agent might use!

[Rishabh @rishksing]

Here is a link to the other 3 cases:
https://t.co/ldI5N9ZdjH

[Rishabh @rishksing]

Anthropic found a dating-app network where AI wrote replies and paid workers took video calls to make the service feel real.
I wanted to make the case easier to follow, so I turned it into a visual story. Here’s a quick walkthrough.

[Rishabh @rishksing]

@AnthropicAI Reports like this are crucial, but there’s a lot to absorb in a dense PDF. I turned the dating-app case into a visual scroll story to help more people understand how the operation worked.
Here's are 4 of the cases from the report: https://t.co/5oEwyYJV4F

[Rishabh @rishksing]

For the engineers: the task is stdlib multiprocessing.Pool(2). It needs working process creation plus POSIX shared-memory semaphores (/dev/shm), which is exactly where gVisor, Firecracker and plain containers differ. 16 of 16 batches since Aug 26, all at the 90s cap.

[Rishabh @rishksing]

Every night for 14 nights straight, the same few lines of ordinary Python have frozen inside the same AI code sandbox.

Not in a stress test. Two worker processes, square 100 numbers, print the sum. The kind of thing an agent writes without thinking. Three sandboxes finish it in about a second. One hits the 90-second cap. Every single night.

That's why the benchmark runs nightly instead of once. A one-off looks like flakiness. Fourteen in a row is how the sandbox is built.

It shows up as a 95% reliability score, with the failed run and its error text right there in the audit trail.

[Rishabh @rishksing]

Live, updated nightly: https://t.co/vxBdNCgGs8

Your agent can query it before picking a provider:
claude mcp add --transport http sharpbench https://t.co/lKAi8m64oq

[Rishabh @rishksing]

@arcyton Good q. The network tasks assert on what came back: status 200 plus a specific string from the fetched body. Plausible fabricated output fails that check, so a blocked provider scores 0 on the task and its stored error text sits in the audit trail. "It finished" proves nothing.

[Rishabh @rishksing]

One of the fastest, cheapest AI code sandboxes can't reach the open web from inside the sandbox.
This isn’t a bug, since it blocks the internet by default, for security. Reasonable design. But it means an agent can install a package and then fail the moment it calls an API, and no pricing page warns you.
So I built https://t.co/VFDnwbkfXH: nightly benchmarks of the services AI agents rely on, starting with the major code sandboxes (E2B, Modal, Daytona, Runloop). The same kind of thing keeps showing up, night after night:
• one ships a default Python that's missing part of the standard library, resulting in ordinary code fails
• one freezes on two lines of completely normal parallel code, every single night
• which one is "fastest" depends on where in the world you measure from
None of this shows up on a pricing page or a popularity leaderboard. All of it decides whether your agent's code actually runs.
The measurements are public and updated nightly. An agent can query SharpBench before it picks a provider.

[Rishabh @rishksing]

Exactly. The geography point is why every batch records its vantage point: I've measured the same provider 10x apart between a laptop and a datacenter, and the latency lead flips with where you stand. The egress allowlist is the same story, it's not on any spec sheet, I only found it by probing from inside.

[Rishabh @rishksing]

@ngikook That papercut is exactly why I probe from inside the sandbox instead of trusting the docs. If you remember which provider or edge case burned you, tell me and I'll turn it into a nightly task.

[Rishabh @rishksing]

Live, updated nightly: https://t.co/HTx3QDyOJr
Code sandboxes + web-search APIs + scraping APIs, scored on quality, latency, cost, and reliability with the reasoning behind every score.
Your agent can query it mid-run: claude mcp add --transport http sharpbench https://t.co/CODLxgTMsv

[Rishabh @rishksing]

@united I paid for the flight using the Explorer card, but your link still says I owe $55 for my first bag, when it should be free :/

[Rishabh @rishksing]

@Matt_Wolach Video, thanks!

[Rishabh @rishksing]

@tibo_maker Inspiring reading your story, thanks for sharing!

[Raspberry Pi Foundation @RaspberryPi_org]

10K FOLLOWER GIVEAWAY 🎉🎈🎁

To celebrate 10K followers on Twitter, we're giving away 3x Raspberry Pi Foundation tote bags containing Raspberry Pi branded goodies 😍 Winners will be randomly selected on Friday, 19 May 2023.

To enter: Follow us, like & RT this tweet.

Good luck!

[Rishabh @rishksing]

Here is the prompt I used with Chatgpt 4 to start building the app. I had to go through a lot of iterations of copy-pasting errors to get it working.

[Rishabh @rishksing]

I hate reading long articles on my phone. So I built this app for myself using ChatGPT!

Now, I share a link to the app, which creates a summary of an article to decide if I want to read the rest of it.
https://t.co/bFqOVIvwbU

[Rishabh @rishksing]

@mckaywrigley Very exciting, signed up! It took a lot of copy-pasting into the chatgpt4 window to get my app running.. Just launched it today, thanks for the inspiration!
https://t.co/kGhO87kCv4

[Rishabh @rishksing]

I hate reading long articles on my phone. So I built this app for myself using ChatGPT!

Now, I share a link to the app, which creates a summary of an article to decide if I want to read the rest of it.
https://t.co/bFqOVIvwbU

[Rishabh @rishksing]

Thanks @mckaywrigley for the inspiration!