Melissa Pan

CS PhD @UCBerkeley Sky Lab 🐻 Systems & AI & Sustainability 🌍 Prev: @google, @ibm, @CarnegieMellon🐕‍🦺, @UofT🇨🇦

[Melissa Pan @melissapan]

Does your Claude model really need Claude Code…? 🤔

We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:

1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.

Millions of people are using coding agents, but the impact of harness choice remains unclear.

(1/n) More details in the thread. 🧵

[Melissa Pan @melissapan]

@arthurcd the broader message is that the community should be more aware about their choice of harnesses as it can have an non-trivial cost implication

[Melissa Pan @melissapan]

Does your Claude model really need Claude Code…? 🤔

We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:

1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.

Millions of people are using coding agents, but the impact of harness choice remains unclear.

(1/n) More details in the thread. 🧵

[Melissa Pan @melissapan]

@MatthewParrott @slumdog_8 This is describing the sponsored Kimi K3 API level setup available to our project, **not** Fireworks AI’s public offerings. Happy to poke around and find out how it correlates with public offering for reference.

[Melissa Pan @melissapan]

@lunkertw thank you Labangram!

[Daisuke Shimamoto @diskshima]

モデルとエージェントの組み合わせによる性能を調べた研究です。

オープンソースのPiが本家のエージェントよりも高得点を出したり、違うエージェントの方が高かったり、と面白いです。

モデルがかなり賢くなってて、もともとエージェントが提供してる機能とかがいらなくて、むしろ邪魔になってたりするのかもしれませんね。

[Melissa Pan @melissapan]

@diskshima Thank you Daisuke for your kind words!

[Melissa Pan @melissapan]

@ziming_mao @TechFollowrazzi @OfficialLoganK by zmmm - someone who already made it since the beginning

[Melissa Pan @melissapan]

@TechFollowrazzi @OfficialLoganK 🙏🫡

[Melissa Pan @melissapan]

@ericzakariasson JIT harness from scratch or on top of existing ones like cursor?

[Melissa Pan @melissapan]

@tobias_pfuetze Thank you Tobias for sharing our work 🙌

[Melissa Pan @melissapan]

@TsubomiHaruno Thank you @TsubomiHaruno for sharing 🙏

[Melissa Pan @melissapan]

@mltmnd_ Thank you @mltmnd_ 🙏

[Melissa Pan @melissapan]

Learn from my friend that HarnessTax was trending on hacker news as well 🫡🙌

[Melissa Pan @melissapan]

Does your Claude model really need Claude Code…? 🤔

We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:

1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.

Millions of people are using coding agents, but the impact of harness choice remains unclear.

(1/n) More details in the thread. 🧵

[Melissa Pan @melissapan]

@SP3NC3R5395 🫡

[Melissa Pan @melissapan]

@lucasbzrs Thanks Lucas!

I haven't use VSCode + Copilot for almost 3 years now at this point. But imo it can also be a UI vs. CLI decision.

[Melissa Pan @melissapan]

And really appreciate the community's discussion and for sharing our work on Hacker News 🙏

[Melissa Pan @melissapan]

@jimmy_longbow_ thank you Jimmy for sharing our work 🙌

[Melissa Pan @melissapan]

@JeanMar61773039 thank you for sharing our work!

[Melissa Pan @melissapan]

@renesugar @OpenRouter 😀

[Melissa Pan @melissapan]

@diegoale_06 Thanks for sharing Diego!