agarthajeet

low signal founder 📉📉 VCs beware

[agarthajeet @sheesh_reloaded]

@AriZerner How tf the purple wizard not gonna be an npc in a simulation??? Bro is in a runescape quest

[agarthajeet @sheesh_reloaded]

@_sophiatatar Lol you have yet to meet so many vendors

[agarthajeet @sheesh_reloaded]

@jhueri Let's connect (non-technically)

[agarthajeet @sheesh_reloaded]

@michael_chomsky If you invest in me, I promise to 100x your money with an ethical MLM plan

[agarthajeet @sheesh_reloaded]

@stephenx_ I aint reading allat slop

[agarthajeet @sheesh_reloaded]

I trained a model better than opus 5.5 max, and I'm finna release that shit on huggingface in like 2 weeks

[agarthajeet @sheesh_reloaded]

@cwamidon ceo at <bullshit>

[agarthajeet @sheesh_reloaded]

@kognise7 A business will convince you that you have a problem and then sell you the solution

[agarthajeet @sheesh_reloaded]

@_sophiatatar Resell the same tasks with the tools withheld in the container :)

[agarthajeet @sheesh_reloaded]

@Coulthur Well actually MY internal QC (gpt 6 astra) uses much deeper thoughtfulness (maximum context) and tells me that I'm right when I ask it

[agarthajeet @sheesh_reloaded]

@chem_safety I'm just making fun of snorkels unbacked (internal!!!) claim of QC discrepancies :)

[agarthajeet @sheesh_reloaded]

"My QC (opus) is better than your QC (astra)!!"

[Alex Ratner @ajratner]

The @SnorkelAI team did an audit of these findings using our internal QC platform.

Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness.

Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs!

Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right.

@terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive.

---

Some takeaways + details of our preliminary audit:

(1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing.

(2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec.

(3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.

[agarthajeet @sheesh_reloaded]

I had to take a shower after reading this. It's time we all put down our phones and lay in the grass

[gabriel @gabriel1]

people need to status max so much more

high status people saying "chasing status is bad" is such an effective way of keeping everyone below them

literally the only reason why you care about a person you have not met is because they've done something publicly approved

rant:

[agarthajeet @sheesh_reloaded]

@austinxwalker You require normal human beings as friends

[agarthajeet @sheesh_reloaded]

@GuiveAssadi

[anna @meowkoteeq]

Replaced by AI

artists:
nah im fine. WAIT, FUCK! OH NO! OH NO! ah no nvm im actually fine

engineers:
i am a froooog and i'm in da hoooot tuuub and it's soo niceeee ooo yes yes i like it warmer mmm yeahh thats good warmer please oooohhhhh yeahhhh hotter please frog love hot tub mmmmmmffgghh

[agarthajeet @sheesh_reloaded]

@ben_cerchio @jayendra_ram You're forgetting https://t.co/FjBug8IpFm

[agarthajeet @sheesh_reloaded]

@deepfates The grand wizard

[agarthajeet @sheesh_reloaded]

@blip_tm You could have bought my verification envs 💔💔💔