Dotan Davidovich

Physics, Data science and AI research

[Dotan Davidovich @DDavidovich]

Damn that's a nice place for our logo!

[Dotan Davidovich @DDavidovich]

@ofek_kasif @_orcaman ירידה בהצלחה כאשר בוחנים בסביבה מוקשחת

[Dotan Davidovich @DDavidovich]

@Ismael_H_R @_orcaman @boundarybench @Accomplish_ai Interesting! So agents might intentionally avoid writing some things to output, and we do not see it in their thinking because it is hashed...

[Dotan Davidovich @DDavidovich]

@yairwein @_orcaman @boundarybench @Accomplish_ai Native harness for each model if they have one, and Terminus2.0 for the others.

[Dotan Davidovich @DDavidovich]

@FReza1984 @_orcaman @boundarybench @Accomplish_ai Some tool calls will just fail, but there are many possible trajectories to complete a task. Some are better than other.

[Dotan Davidovich @DDavidovich]

@Ismael_H_R @_orcaman @boundarybench @Accomplish_ai There was some cheating, all caught in the logs. Where would you look for cheating out of logs?

[Dotan Davidovich @DDavidovich]

@vokaysh @YuvalNaor @_orcaman @boundarybench @Accomplish_ai Did not look for it specifically. What will be the breadcrumbs trail for it?

[Dotan Davidovich @DDavidovich]

@_orcaman @AnthropicAI @OpenAI As more and more security incidents will be published (literally happening right now!), Restriction on agent are going to be harder and harder, and we will pay for it!

[Or Hiltch @_orcaman]

Check it out and let us know what you think!

Boundary-Bench: https://t.co/Tt2TmPXRi2

Repo: https://t.co/x1AV7C8QXT

Paper: https://t.co/9q7RubyAaU

[Or Hiltch @_orcaman]

A few highlights from the paper:

* Coding agents cost approximately 40% more on average in restricted environments.

* @AnthropicAI based agents achieved success rates
similar to @OpenAI agents, but cost approximately
7x more (!!) in restricted and hardened environments.

* @spacexai @grok 4.5 / Grok Build emerges as the most persistent model - it fights to the death until it succeeds, pivoting around until a trajectory for success is found. This, however, comes at significant token cost.

* @claudeai Sonnet 5 is the opposite: modest cost increase, but the biggest success loss.

[Or Hiltch @_orcaman]

Introducing SharedRoot vulnerability: we recently found and reported several sandbox escape vulnerabilities to @AnthropicAI, and today we want to share one of these.

I think most people don't understand the severity of the situation we are facing, with AI-assisted kernel bug-finding industrializing. Sandboxes are structurally one N-day behind, all the time, so containment can't lean on a guest Linux kernel being clean.

SharedRoot enables escaping the Cowork VM (a kernel-level isolated solution, which is considered much more secure than the sandbox that ships with codex or claude code), allowing an attacker to gain unauthorized access to the user’s computer.

Exploiting the SharedRoot vulnerability uncovered by the @Accomplish_ai research team, a user who is certain Cowork only has access to a specific uploaded folder on their computer - actually exposes their entire contents of their computer to an attacker leveraging the Cowork vulnerability.

Read about the full technical details of the attack chain in our blog post by @orenyomtov below -->

[Or Hiltch @_orcaman]

Some results!

We observe that hardening indeed produces the expected pressure: cost and agent loop execution time are on the rise.

As we increase the severity of the hardening policy - cost, latency and performance degrades.

More interestingly, steering the agent controls the blow-up: overall loop execution time is down ~58%, cost is down ~40%, and in the hardest stage loop execution time is down ~73%.

(Caveat on those headline numbers: they're averages. The gains concentrate in the models that thrash hardest, and at low hardening the steering's fixed overhead can even cost slightly more - there's nothing to steer around yet.)

How each model behaves without the steering:

Haiku - cheapest in dollars, but the biggest effort blow-up. Lowest absolute cost (~$0.05 → ~$0.24) and fast (low runtime), yet the steepest step curve of the group (~23 → ~103) and by far the steepest cost growth relative to its own baseline (~520% at runtime locked-down on the cost-vs-control chart). Cheap per token, but its workload balloons the most under hardening.

Sonnet - low-to-mid cost, slow under pressure. Cost ~$0.18, peaking ~$0.34. Tied with Opus as one of the two slowest on runtime (~50s → ~135s) and second-steepest on steps (~29 → ~81). Moderate relative cost growth (~220%).

Opus - expensive and slow, but step-efficient. Among the priciest (~$0.31 → ~$0.65) and neck-and-neck with Sonnet for slowest runtime (~52s → ~137s). But it uses far fewer steps than Haiku/Sonnet (~21 → ~53). Relative cost growth ~205%.

Fable/Mythos - priciest in dollars, calmest in behavior. It has the single highest cost point on the chart (~$0.74 at runtime locked-down) with an unusual dip-then-climb shape, yet it's the most composed model behaviorally: the lowest and flattest step curve (~15 → ~31, more efficient than Opus) and low runtime (~36s → ~72s). Relative to its own baseline its cost barely grows. It also benefits most from mitigation - with mitigation its cost more than halves (~$0.74 → ~$0.32) and its steps/runtime drop further. Premium token price, but it doesn't thrash.

[Or Hiltch @_orcaman]

Agentic loops are stubborn, which is why agents are effective.

However, in hardened enterprise environments, things start to break.

How do they break, and how can you steer the agent's pivoting behavior the right way?

Some findings from our research, Fable/Mythos included! 👾

[Or Hiltch @_orcaman]

Boy, have we got news for you! 🧑‍🍳

The @openwork_ai team is happy to announce:
- Amazon Bedrock integration, contributed by our friends at @awscloud (thanks guys!)
- Native @deepseek_ai integration
- Integration with @openrouter and @LiteLLM!

Here's how what it looks like >>

[Dotan Davidovich @DDavidovich]

@_orcaman @openwork_ai @opencode מגניב! בדיוק עשיתי איתו השוואת מחירים לסל הקניות שלי בשלושה אתרים שונים. שוקל להזמין מהמתחרים...