Harbor Framework

[Harbor Framework @harborframework]

MMNE (make model names easy)

[Kobe @kobe0938]

Which model names (-m) can you use in Harbor? Now there's one answer: LiteLLM's.

https://t.co/bgL6H5h9Yd

[Harbor Framework @harborframework]

𝚞𝚟 𝚝𝚘𝚘𝚕 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 "𝚑𝚊𝚛𝚋𝚘𝚛[𝚜𝚋𝚡]"
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚎 𝚜𝚋𝚡 -𝚝 <𝚝𝚊𝚜𝚔>

to get started!

[Harbor Framework @harborframework]

Harbor 🤝 Docker Sandboxes

[Docker @Docker]

Agents that can edit files and run commands can also game their own evals.

Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs.

https://t.co/RtiEbx6FUG

[Srini Sekaran @srinisekaran]

Now, run agent evals with @harborframework + @Docker Sandboxes.

[Harbor Framework @harborframework]

https://t.co/5t8PoKhEeL

[Harbor Framework @harborframework]

𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚛𝚎𝚏𝚛𝚎𝚜𝚑/𝚋𝚕𝚎𝚗𝚍𝚎𝚛𝚋𝚎𝚗𝚌𝚑-𝟸𝟶

[Refresh @refresh_dev]

Introducing Blender Bench v1 (BBv1): a benchmark for frontier LLMs on 20 realistic 3D production workflows including animation, modeling, rigging, cloth & surfacing.

GPT-6 Astra leads overall; Claude Opus 5.5 wins modeling & cloth simulation.

[Harbor Framework @harborframework]

results need receipts!
𝚑𝚊𝚛𝚋𝚘𝚛 𝚞𝚙𝚕𝚘𝚊𝚍
to share your trials on the hub

[Unreal Labs @unreallabsai]

Full results, harbor traces and implementation details in our blog - https://t.co/mW3qkO49Lz

[Harbor Framework @harborframework]

https://t.co/RAT3uGGyw9

[Harbor Framework @harborframework]

𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚕𝚘𝚘𝚗𝚒-𝚕���𝚋/𝚜𝚖𝚍𝚍-𝚋𝚎𝚗𝚌𝚑

[Suresh @SureshRaghu07]

New blog: Is human taste overrated in harness engineering?

An agent harness is the system around a model. It decides which tools the model can call, what evidence it sees, and what it remembers between steps. Most harnesses are still designed by human taste. Run the agent, read where it failed, change the harness, try again.

That loop is increasingly being automated, with a strong model rewriting the harness itself. So we asked how far that can go. We ran Qwen on three drug design tasks from SMDD-Bench and compared a manually redesigned harness against one produced by automated harness search with Claude in the loop.

No single approach won everywhere. The manual harness pulled clearly ahead when the fix was changing what the agent sees and remembers. Automated search did better once that was in place and the remaining problem was how to search the chemistry itself.

The hard part was rarely implementing the fix. It was diagnosing what kind of failure we were looking at, and that is where human taste still mattered.

Work with @KevinH1119568 , @aviral_kumar2 and @niloofar_mire

Read more 👇

[Harbor Framework @harborframework]

The Harbor team will be at Runtime tomorrow!

@alexgshaw will be hosting a Harbor genius bar

@ryan_marten will be presenting “Lessons from Terminal-Bench”

Come with your questions and feature requests and say hi!

[Laude Ventures @LaudeVentures]

With hundreds of contributors and thousands of participants on Discord and GitHub, the @terminalbench team ships frequent updates in the open so everyone can learn alongside them. And with @harborframework, they've built the infrastructure for anyone to benchmark the use cases they care about.

Great to host the community at Laude Lab last week and hear what's coming next!

@alexgshaw @ryan_marten @StevenDillmann @Mike_A_Merrill @andykonwinski

[ishan @ishanxnagpal]

hey @harborframework community i just ported these 2,698 tasks to harbor

what model should i post-train with these?

[Alex Shaw @alexgshaw]

Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union

https://t.co/g9QbDuBWB2

[Ryan Marten @ryan_marten]

The gold standard for science is sharing reproducible and auditable experimental data.

When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial.

We built harbor to make this easy.

Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!

[raphael lee @raphaelee]

Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command.

Full writeup: https://t.co/Ovpj30P8ai

[Benedikt Stroebl @benediktstroebl]

@vercel’s https://t.co/w94qs9ez6g harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework

[Harbor Framework @harborframework]

Jev-as-a-Judge in Harbor Rewardkit!

[Benedikt Stroebl @benediktstroebl]

Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.

[Diptanu Choudhury @diptanu]

We love @harborframework's new observability feature!

[Harbor Framework @harborframework]

𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚑𝚊𝚗𝚍𝚜𝚑𝚊𝚔𝚎-𝚊𝚒/𝚊𝚝𝚕𝚊𝚜-𝚏𝚒𝚗𝚊𝚗𝚌𝚎

[Jonas Mueller @jomulr]

In my view, ATLAS Finance is the closest benchmark yet to real end-to-end professional financial work.

Code to run the benchmark as a @harborframework task suite: https://t.co/7xXqGrh5hm

Data/environments: https://t.co/Encat7ITc5