Andrew Hess

Caltech 2021, acquired by Apple, working on next startup

[Andrew Hess @andrew7hess]

@Bayesprof Probably several of Ramanujan’s equations. I’m just guessing though

[Andrew Hess @andrew7hess]

Dear @OpenAI, show N more is bad UI when N=1

It hides information without saving space

[Andrew Hess @andrew7hess]

@garrytan That reduces to a single criterion that it’s on the frontier of capability vs cost. The absolute best is automatically on that frontier because no other model can get that score regardless of the cost

[Andrew Hess @andrew7hess]

If you loop enough times and fix the bugs found before the next search, it does eventually find no more bugs

I’ve done this. Expect runs to take 2-15 hours depending on the size of change, if you’ve built up the repo this way.

If you’ve built the repo without ruthlessly eliminating bugs, it’ll take longer and you should definitely get it to drop any bugs it finds that are outside the scope of your intended change. Otherwise it’ll keep pulling more of the repo in, the run could take 30+ hours and produce a PR upwards of 50k lines changed

[Andrew Hess @andrew7hess]

@m4h007 It’s OpenAI’s job to clearly show the correctness if they want them to be accepted as correct. Is that what they want?

[Andrew Hess @andrew7hess]

@BenMyersPoet Why? Both seem true.

[Andrew Hess @andrew7hess]

You want confidence that pros outweigh cons. You have to base your confidence on something. Options include:

- A human doctor/mathematician. Even if you don’t know them, your trust is based on society cultivating good experts (eg, it takes a long track record of excellence and integrity to become a reputable expert due to feedback loops from medical/academic associations)
- A separate AI report. Trust can draw from your experience with that model, concrete benchmarks on similar tasks, etc.
- Interactively digging into it with the help of AI. Again you need to have some confidence in the model you’re using. Additionally you’re trusting that you’re able to ask sufficiently thorough questions to properly evaluate the treatment

You can build confidence from multiple sources, and long term, requiring human-based trust is not necessary. In the short term, we as society don’t have the mechanisms in place to accurately gauge the proper amount of trust in AI for high risk decisions.

For this scenario, the risk of this treatment vs nothing may be low enough, and your trust in the model may be high enough, that getting additional trust via human experts may not be necessary

[Andrew Curran @AndrewCurran_]

This is a slide from Terence Tao's 'Math 2.0' public lecture yesterday evening at Caltech. I am highly confident that, if the cure actually worked, almost any Stage 4 cancer patient would tell you that human comprehension of its mechanism is completely irrelevant to them.

[Andrew Hess @andrew7hess]

When building with agents and quality matters and you’re not sure how big a single task should be:
1. do the task twice
2. if the results differ by more than your error threshold, you’re not meeting your quality bar. So throw away the work, split the task, and repeat this process on each subtask

[Andrew Hess @andrew7hess]

@PunishedZulu It’s because that’s why

[Andrew Hess @andrew7hess]

@DenisCerednice1 See ya in a few months

[Andrew Hess @andrew7hess]

@brockjelmore That equation _is_ aesthetic. Meters is not a natural unit

[Matt Johansen @mattjay]

Exactly one (1) person could make me nervous tweeting this.

And there he is

[Andrew Hess @andrew7hess]

As a software developer, welcome to the other side, mathematicians

[Kevin Madura @kmad]

Uhhh.. guys. Matthew is someone who actually knows what they’re talking about.

[Flowers ☾ @flowersslop]

> brand new benchmark to probe something frontier models still suck at vs human
> gpt-6 scores 50%+ on release

at this point i actually doubt you can, at all, make a fair benchmark where humans get >80% and the best ai model <20%

man if only there were a term for this

[Andrew Hess @andrew7hess]

@DJiafei They made the movie real?

[Andrew Hess @andrew7hess]

@ThePrimeagen I thought something similar a while ago, and then realized every single number I was counting was a rational number

[Andrew Hess @andrew7hess]

@QiaochuYuan It started off so well

[Andrew Hess @andrew7hess]

Every week it gets harder to stay sane about AI progress