Brett Thomas

minka, nyc, surge ai, wellness

[echen @echen]

LMArena is a cancer on AI.

I was hoping it would die out 6 months ago, after Maverick showed what it gets you.

But it keeps rearing its head. The WSJ talks about how important it is. A new VP asks what their team is doing to climb it. The cycle continues.

It’s fundamentally broken, optimized for the wrong incentives:

Users spend 2 seconds skimming responses before clicking their favorite. They're not reading carefully.

They're not fact-checking. They're just picking whichever model response catches their eye.

This means that the easiest way to win on LMArena is by...

Being verbose - longer responses look more authoritative!

Formatting aggressively - bold headers look like polished writing!

Vibing - wild, colorful emojis grab your attention!

It doesn't matter if a model completely hallucinates. If it looks impressive, LMSYS users will vote for it over a correct answer.

After all, remember this and all the sycophancy issues we've seen this year?

[Brett Thomas @BrettThomas]

LLM memory today feels similar to agents before Cursor Composer got really good.

[Brett Thomas @BrettThomas]

@hwchung27 Do you think this will change with age?

[Brett Thomas @BrettThomas]

@mdwelsh Seen any good successors to your "End of Programming" talk, with learnings from AI agent dev since then?

[Brett Thomas @BrettThomas]

@GregKamradt Nice. Do you know a good example of (5)?

[Brett Thomas @BrettThomas]

@jxnlco Big fan of your account and your work. I'll reach out this week!

[Brett Thomas @BrettThomas]

@dvargas92495 Just getting started, but seems to me that gpt4 function calling struggles with planning prompts, e.g. think() in autogpt. Using functions literally causes worse thinking.

Curious if you've seen that too?

[Brett Thomas @BrettThomas]

@ajratner Have you seen any good examples of fine tuning retrieval? I'm experimenting with this now.

[Brett Thomas @BrettThomas]

@__codak__ True in your 20s, not in your 50s

[Brett Thomas @BrettThomas]

@alex_barashkov @neondatabase @splinetool @runwayml Runway -> spline was manual, right?

[Brett Thomas @BrettThomas]

Hey @chesaboudin before I sign this thing, is there a simple list of arrests and if/why they were prosecuted?

[Brett Thomas @BrettThomas]

@NotionHQ Is there a way to expand sidebar hierarchy to current page? (Like ⌥ -F1 in IntelliJ)

[Brett Thomas @BrettThomas]

@razibkhan Even if they scaled enrollment and set it to perfect racial proportions...the university model would still make no sense. IMO that’s what causes the fear of change.

[Brett Thomas @BrettThomas]

@Mail_Gun 1255031 thanks!!

[Brett Thomas @BrettThomas]

@Mail_Gun anybody on call right now? we can't get a response from support

[Rachel Thomas @rachel_r_thomas]

The Thomas family is all in on @PeteButtigieg!!

[Rachel Thomas @rachel_r_thomas]

Sign up for #fwiw. All the cool kids are doing it https://t.co/shvEB4HK2N

[Brett Thomas @BrettThomas]

I think @Billbrowder should run for president as a Republican

[Brett Thomas @BrettThomas]

"Alexa what are my deals?" seems like a missing chapter from Homo Deus