zack

enterprise software @ spacex+starlink / rocket enjoyer / entropy herder

[gabriel @gabriel1]

after now having interviewed 50+ people, it's crazy how rare it is to talk about things you've done in concrete terms

always only talk with high precision about what you did and why no one else could

it doesn't mean anything that you "made X better" or "led thing Y"

examples:

[zack @zackR154]

of course they STILL used agents all the way down for judging this, sigh.

if you have too many tasks for a human to review then you surely have enough for a statistically significant random sample, instead of burning tokens and introducing potential bias for no reason!

[OpenAI @OpenAI]

To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent experienced software engineers.

That helped us examine tasks at scale while keeping expert judgment at the center.

[zack @zackR154]

in general almost every bench I’ve seen sucks, and it’s insane that OpenAI recommended one without reading enough test cases to realize ~30% of them were bullshit

there is no substitute for reading the stupid thing. otherwise you will spend ages benchmaxxing against nonsense

[zack @zackR154]

ah it’s back

[OpenAI @OpenAI]

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability.

We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval.
https://t.co/wDdSEjBe4F

[zack @zackR154]

@shakoistsLog relatedly, openai claims SWE-Bench Pro is basically useless, after recommending it

[zack @zackR154]

in general almost every bench I’ve seen sucks, and it’s insane that OpenAI recommended one without reading enough test cases to realize ~30% of them were bullshit

there is no substitute for reading the stupid thing. otherwise you will spend ages benchmaxxing against nonsense

[zack @zackR154]

this is the exact interface I’m complaining about

behold the state of the art in automotive design

[eternal classic @eternalclassic_]

enshittification is undefeated

[zack @zackR154]

the new Volvo XC60 I am renting this week has some of the worst UX I’ve ever seen in a car

>adjusting the climate control requires two taps on the touchscreen. there are physical buttons for volume and the defrosters, but nothing else

>the instrument panel has two display options: 5fps version of your map, or black screen

>the ignition is an inbred child of a knob and button which you turn 1/8th clockwise to turn the car either on or off, which doesn’t make sense even if you’re trying to imitate a key

>turning off auto-stop requires going into a settings page on the touchscreen on every drive

>because the shift lever has no secondary lock on it, it cannot be used to shift into neutral while driving except by pressing the brake pedal, which is probably a safety issue!

>no wireless CarPlay

>a bunch of center console real estate is taken up by a 12V socket, covered with a free-floating cap that will instantly get lost

and none of the online reviews mention any of this, they’re all praising the connected vehicle technology and the fact that it kind of sort of runs Android, I feel like I’m going insane

hard to see me buying another car that isn’t a Tesla anytime soon

[zack @zackR154]

@atlanticesque I constantly use “Spiders Georg” or “X Georg” at work

[zack @zackR154]

@ansonyuu then: “Enguerrand VII, Sire de Coucy, Count of Soissons, Earl of Bedford”

now: “Kevin, prev. Palantir, Jane Street AMP, YC X26”

[zack @zackR154]

@iv_hacks I almost forgot about the Toyota Corolla I had last month which would proactively brake the car if it thought you were following too close or if a turn was coming up, until I figured out how to turn that off

[zack @zackR154]

related

[Jeff Ermann @Jeff_Ermann]

Am I missing something, or does $20 mil seem way low to RENAME YOUR CONFERENCE AFTER AN ENERGY DRINK

[zack @zackR154]

some crazy error bars but also $150k/day for Madison Square Garden is crazy low, isn’t it?

at either price I’m a little surprised some AI startup hasn’t done something mildly tasteless there

[zack @zackR154]

this is, of course, Matt Levine

https://t.co/jE75aeXNgA

[zack @zackR154]

I don’t think I’ve seen this criticism of prediction markets articulated this clearly before

well aside from waving your hands and shouting “GOODHART’S LAW WINS AGAIN!”

[zack @zackR154]

this seems like it might not be a complete privacy disclosure

[zack @zackR154]

@xuelinda7 this is super cool - I see lots of startups try and solve perceived problems, find out the bottleneck isn’t what they thought, and die, but this is the first time someone’s actually written it up

can all the “let’s solve tech hiring” startups do this next

[zack @zackR154]

mildly infuriating hotel room

[zack @zackR154]

I think maneuvering your bag through crowds at the airport should be an Olympic sport

[holden m. accountable @noahpasaran]

have dined with orc before? orc do things differently here. orc actually recommend start with two or three small meat to share. meat get bigger as you go down menu. orc restaurant rated highly in new orc times

[zack @zackR154]

1) what