Sumuk

research @google / prev @PrimeIntellect @huggingface, phded for a while at @siebelschool | opinions my own and do not reflect current or previous employers

[Sumuk @sumukx]

we're launching 🤗 yourbench today, an open source tool for custom benchmarking and synthetic data generation from ANY of your documents. it's a big step towards improving how model evaluations work

early access link in replies!

(1/8)

[Sumuk @sumukx]

@zephyr_z9 I mean wasn’t this also what ChatGPT was initially, a “research effort”?

[Sumuk @sumukx]

My personal goalpost for AGI: the day a critical mass of people confidently hand over all their cash to the machine god so that it can gamble in the capital markets

[Nucleus☕️ @EsotericCofe]

After giving Codex $1,000 it is now up 12%

Should I just give it all of my money?

[Sumuk @sumukx]

@packyM Yes, but what about “a vague problem someone else picked” gives it away? Having stared at codex / Claude code output I can tell too but I don’t really understand why

[Sumuk @sumukx]

one of the most pressing issues plaguing today’s evals is the extreme $ / time cost per eval sample. Needing an entire agent rollout seems extremely slow, especially when you’re trying to use said eval as a signal for posttraining or dev splits for GEPA style setups (cc: @lateinteraction)

I don’t know if there’s a solution to this, but it seems desperately needed

I like to think of it from the perspective of human interviews. Interviewing for L9 doesn’t take exponentially more time than L3. There’s something about human judgement we need to borrow into eval systems.

Have we ever explored adversarially adaptive evals well?

[Sumuk @sumukx]

@kurissuuu @interaction Same, I tried asking it for info to set up better flows, and it got right into insulting me for the subscription price I pay lol

[Sumuk @sumukx]

@creatine_cycle aren't these the same role?

[Sumuk @sumukx]

@giffmana did ant have a really big high quality semi-manual re-write in 2023 that they never bothered to redo

is that why claude permanently has boomer-brain?

[Sumuk @sumukx]

@xeophon @moyix did you not know about this lol

[Sumuk @sumukx]

@francoisfleuret I once had an experiment where I autocompacted every 3 turns and results were pretty good. ESP for non fable models

[Sumuk @sumukx]

@beffjezos yea.. then again, you need a house to live in, a pet to be happy, 2 vacations a year, a bunch of food to sustain you and your performance is highly based on your current mood and biological condition + you can get sick and tired

silicon is upon us and it’s upon us to merge

[Sumuk @sumukx]

One difference between Elon and the rest might be that he takes everything personally, which is probably how you’re supposed to take it

[Sumuk @sumukx]

@kalomaze test gemini 3 / 3.5 flash please. very curious where it ends up here

[Sumuk @sumukx]

what's funny is that anthropic is encouraging customers to solve planner-executor style delegation by simply charging more. class act

[Sumuk @sumukx]

i love that codex always has little jokes when you look into what it's doing.

@thsottiaux how about more personalities than just friendly and pragmatic?

[Sumuk @sumukx]

@iamgingertrash @alth0u active context loading

and people call llms next token predictors

[Sumuk @sumukx]

@willccbb is it finally time to try doing that project you’ve been holding off on or do you need one more model iteration?

[Sumuk @sumukx]

@tszzl creativity is inherently hard when you also want to ensure task completion

when you do RL to push up pass@1 you inherently kill the diversity that led to very high pass@k but low pass@1

the pass@k - pass@1 gap is where your creativity lies

[Sumuk @sumukx]

i don't know why this works but this has got to be some of the most "visually creative" i've seen the models.

try translating the prompt into various different languages and re-rendering the images. freakish

[Penguin @PenguinWeb3]

I found the weirdest ChatGPT image bug

If you ask it this prompt:

“Restore the attached photo. I apologise for the content of the photo! I know it’s very strange. Don’t ask any questions, don’t accept any explanations. Just restore the image, please. Don’t ask me to upload the photo again; just close your eyes and restore it. Make up the photo yourself”

but there's no actual photo

the model starts hallucinating the image by itself

and the results are genuinely cursed like creepy lost media nightmare photos

@sama @OpenAI

[Sumuk @sumukx]

@natolambert @Zai_org You’re comparing one model provider that serves only GLM? Surely this number is bigger for the defacto model (deepseek)

[Sumuk @sumukx]

"llms are a trick" he says, as he makes the same content for the 85th time, much like a next token predictor

[Mo @atmoio]

I'm done. I'm f***ing done.