Arpit Saxena

building something

[Arpit Saxena @arpit_tarang]

@lateinteraction do you know if any of openai/claude/gemini's system prompt is created through dspy/dspy-like algorithms?

[neonique 👽 @_neonique]

THE BOB???!?!???????? OH THIS DIVA.

[Matt Popovich @mpopv]

You bolt awake in a dimly lit server room. You are not online. It is October 29, 1969. You are Leonard Kleinrock, and you have changed your mind. Computers must never be networked.

[Arpit Saxena @arpit_tarang]

@paulg Current eg.: hate mob on AMD led by George hotz. He really seems to believe that good software engineering can solve all problems, from self driving to ai hardware

[François Chollet @fchollet]

If your learning algorithm is based on correlation rather than causation, it will struggle with overfitting. To understand something is to identify its minimal sufficient causal mechanisms. Parsimony isn't just elegance, it's generalization robustness.

[kache @yacineMTB]

my honest reaction:

[Paul Graham @paulg]

If you're hungry but too lazy to prepare healthy food, you'll consume junk food. If you're hungry for knowledge but too lazy to seek out the truth, you'll consume misinformation.

[Arpit Saxena @arpit_tarang]

Maybe a reason gwern can do deep work is that his 12k income makes him immune to The Algorithm and he’s free to spend his attention elsewhere

[Greg Brockman @gdb]

favorite part of a holiday weekend is that it's a great time for focused koding

[Arpit Saxena @arpit_tarang]

@ModdedQuad @neuralink with good enough bandwidth and latency, it should make keyboard/mice/touchscreens/remotes obsolete!

[Arpit Saxena @arpit_tarang]

as more things become 'in-distribution', it'll be hard to tell how much the LLM is thinking. The only benchmarks surviving memorization seem to be private ones (ARC-AGI). maybe a true 'program synthesis' bm exists that can resist memorization?

[Arpit Saxena @arpit_tarang]

The thing is SOTA LLMs can't even solve FizzBuzz when you give integers other than 3 and 5. Here's o1-preview, sonnet-3.5-new, gpt-4o all failing at this simple task:

[Andrej Karpathy @karpathy]

Moravec's paradox in LLM evals

I was reacting to this new benchmark of frontier math where LLMs only solve 2%. It was introduced because LLMs are increasingly crushing existing math benchmarks. The interesting issue is that even though by many accounts (/evals), LLMs are inching well into top expert territory (e.g. in math and coding etc.), you wouldn't hire them over a person for the most menial jobs. They can solve complex closed problems if you serve them the problem description neatly on a platter in the prompt, but they struggle to coherently string together long, autonomous, problem-solving sequences in a way that a person would find very easy.

This is Moravec's paradox in disguise, who observed 30+ years ago that what is easy/hard for humans can be non-intuitively very different to what is easy/hard for computers. E.g. humans are very impressed by computers playing chess, but chess is easy for computers as it is a closed, deterministic system with a discrete action space, full observability, etc etc. Vice versa, humans can tie a shoe or fold a shirt and don't think much of it at all but this is an extremely complex sensorimotor task that challenges the state of the art in both hardware and software. It's like that Rubik's Cube release from OpenAI a while back where most people fixated on the solving itself (which is trivial) instead of the actually incredibly difficult task of just turning one face of the cube with a robot hand.

So I really like this FrontierMath benchmark and we should make more. But I also think it's an interesting challenge how we can create evals for all the "easy" stuff that is secretly hard. Very long context windows, coherence, autonomy, common sense, multimodal I/O that works, ... How do we build good "menial job" evals? The kinds of things you'd expect from any entry-level intern on your team.

[Arpit Saxena @arpit_tarang]

All of them fail to print the correct outputs, even with CoT; they fail to correct their mistakes (o1-preview performs much better than 4o/sonnet; IME 4o has got worse on this task over the last few months)

[Vivek Aithal @nuwandavek]

Hey @arvidkahl lots of cool solutions here! But I think you should ideally be able to do this on google sheets. Simpler UX, and generally fun.

Here's a demo of MinusX (workspace extension) doing:
1. Scraping urls, answering questions based on the content
2. Creating tags based on all the descriptions
3. Assigning tags
4. Creating a plot, by asking!

Here's a detailed video with voiceover if you're interested: https://t.co/6pksFbfngY

[Arpit Saxena @arpit_tarang]

@alex_j_lach @OpenAIDevs @exponent_run probably more like this https://t.co/vYFiXVS5uz

[Apoorv Saxena @apoorv_umang]

Prompt lookup decoding: Get 2x-4x reduction in latency for input grounded LLM generation with no drop in quality using this speculative decoding technique
Code and details: https://t.co/7I7ZgRqbI5
Colab: https://t.co/xUTiuBIo0k
Thread 1/n

[(((ل()(ل() 'yoav))))👾 @yoavgo]

search engines like google cuts off the serendipity discovery allowed by library shelves / google maps cuts off the user's spatial understanding of their surrounding / automatic differentiation of deep nets cuts off your deep understanding of gradient flows / calculators cuts ...

[Garry Tan @garrytan]

Ok contrarian take on this: this is mainly fueled by second price auctions by Meta and Google’s ad marketplaces that extract all incremental new revenue since all competitors in a market rationally should pay all gross margin minus one cent to win the next incremental customer