Felix Dorrek

Co-Founder & CEO at Earthscale

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Tomorrow morning, I’ll release a blog that outlines a detailed roadmap for setting up evals on subjective / open-ended tasks. The blog also covers a variety of stats / ML tricks for reducing noise and increasing confidence in your evals.

Here is a list of blogs, papers, and other resources (written both by me and by others) I often return to when thinking about this topic:

- High-quality human data: https://t.co/5o8flZLzzX
- Human evaluation guidelines: https://t.co/BwgTwv26e3
- LLM-as-a-Judge core concepts: https://t.co/ZVVIqezskV
- LLM-as-a-Judge practical guide: https://t.co/Qsv78WlBfB
- LLM-as-a-Judge best practices: https://t.co/S76u1d4NS1
- Rubric-based judges (RL focus): https://t.co/rNOAiEL59D
- Rubric-based judges (eval focus): https://t.co/jUrQIgQrxz
- Stats for LLM evals: https://t.co/Qsu4vBZ1DH
- Adding error bars to evals: https://t.co/hYMrnaiGxb
- Anatomy of a benchmark: https://t.co/7nFvsMCyR0
- Agent evals: https://t.co/LkdsoLXYGu
- Full ecosystem for agent evals: https://t.co/jkA8DT8t8W

Would highly recommend reading all of these if you're interested in building better evals. Each of these blogs also link to further papers / resources on each topic that are incredibly useful.

[Naval @naval]

The best way to pace the frontier is to hold the labs fully liable for the behavior of their models.

[Felix Dorrek @FDorrek]

@scaling01 no

[Felix Dorrek @FDorrek]

@GregHBurnham Abstraction = Compression = Intelligence

You can do a lot with the current set of abstractions but much less than you could do with entirely novel abstractions. Just think about the models having to do this math without relying on the abstractions so far developed by humans.

[Felix Dorrek @FDorrek]

@scaling01 Data bound -> Verification bound

[Felix Dorrek @FDorrek]

@gabriel1 Dependent Origination

[じゃがりきん @jagarikin]

なんと動いてません

[Felix Dorrek @FDorrek]

@davis7 This is great. Thank you! Any reason you didn't try 5.6 Pro on these problems? It's very smart. Not exactly the same domain but for solving math conjectures it seems to be the strongest publicly available model.

[Felix Dorrek @FDorrek]

@wtgowers As I say there I think a main missing ingredient is something like coming up with deep explanations relative to pre-existing experience.

[Felix Dorrek @FDorrek]

@wtgowers Thanks for sharing this. Recently had an LLM agent team find a Lean-verified proof for bodies of revolution of petty's conjecture (my former field) but it matches your description exactly. Many paths tried, no novel methods in the result: https://t.co/5dqURubGF9

[Felix Dorrek @FDorrek]

@AcerFur Had GPT 5.6 and Fable solve a significant special case of a conjecture from my former field of convex geometry (petty's conjecture). Observed the same pattern there. Wrote up my thoughts here: https://t.co/5dqURubGF9

[Zeke Hausfather @hausfath]

I'm not sure folks have realized just how crazy the second half of 2026 and 2027 will be for global temperatures – on the back of a record-smashing El Niño event. Here is my latest estimate of where both years will end up compared to global temperatures since 1850.

[Felix Dorrek @FDorrek]

@emollick Sonnet 4.6 > Fable in this eval ...

[Felix Dorrek @FDorrek]

@danshipper Counterexamples are more grindable. Also relatively under-explored because of incentive structures / bias for positive results.

[Felix Dorrek @FDorrek]

Used to switch back and forth between Codex and Claude Code upon major releases. I now use both on a daily basis. The products/models feel increasingly complementary and differentiated.

[Nick @nickcammarata]

I think it’s okay to be textbook rather than poetic about the path bc the truth is already caricaturely poetic

you can have everything you’ve always wanted but have to give up craving, you’re not everything but never separate, to be happy in any state you need to act selflessly

[spor @sporadica]

eventually you learn there is no winning or losing, there is just living truthfully or untruthfully

[Felix Dorrek @FDorrek]

@alevizio We love it thank you @alevizio !!

[Ruben Laukkonen @RubenLaukkonen]

You can just stop doing stuff

[Nature is Amazing ☘️ @AMAZlNGNATURE]

Ants making a smart maneuver