nickcdryan

nlp, deep learning, NYC

[nickcdryan @nickcdryan]

I'm happy to introduce TextEvolve, a system that iteratively generates and tests new programs over your dataset, evolving its approach with LLM evaluation.

TextEvolve changes program flow, tries new ideas, and outputs optimized programs in the form of Python scripts.

[David @DavidSHolz]

Most researchers agree that autoregression is best when memory bandwidth is cheap and diffusion is best when FLOPS are cheap. They also admit the future of compute is all FLOPS because memory scaling is hard and scaling FLOPS is easy. So why not go all in on diffusion????

[Zachary Nado @zacharynado]

the sloptimizer field is just getting started with shampoo and muon gen algorithms, the graveyard of adam variants got so bad you can't list them all on a page

[nickcdryan @nickcdryan]

@srush_nlp It's nice that everything has a clean referent but bad that it's all informationally flat.

They should be named as honestly as the wandb runs:

GRPO-tweak1
GRPO-tweak2
GRPO-tweak2+tweak1

[nickcdryan @nickcdryan]

Low exploration costs improve craftsmanship.

I can think of few trades where free, low-cost exploration doesn't lead to better knowledge and craftsmanship.

Software design has been largely, rightly, guided by risk-aversion. A changing explore/exploit balance is a good thing.

[Erik Meijer @headinthebox]

Slop generators like Ralph, Loom and Gastown treat developers as if they are free resources, replacing costly humans with swarms of agents churning out specs or cloned patterns.

My take is that the real disruption is not about free labor, but about the fact that changing your mind has become free.

Once the cost of change collapses, software development stops being craft and becomes pure brute force exploration.

That is why software engineering as we knew it, is effectively dead.

[nickcdryan @nickcdryan]

Even simpler: it's just a basic requirement because there isn't enough time or $ to gridsearch your idea.

And if gridsearching your idea is the only way to make it work it's probably not worth it anyway.

[yi @agihippo]

ablations are for the weak. just yolo your runs.

(ok, do some small amount of ablations, but don't over do it).

instinct is everything in ML and AI.

[nickcdryan @nickcdryan]

@jxmnop why do you think models won't close the gap on writing good kernels?

[nickcdryan @nickcdryan]

@LysandreJik Agree with the sentiment - those all seemed like big, distant step changes in the early decoder days. And very cool viz.

That said...the dates are broken for all those models.

[nickcdryan @nickcdryan]

Getting to the heart of the matter here, and fixing it.

Heard many times batching is the culprit, but this is the first in-depth explanation I've seen.

[Thinking Machines @thinkymachines]

Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”

We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly.

The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.

https://t.co/lrJioBmpbT

[Brian Huang @brianryhuang]

It's one thing to read about reward hacking in the literature and think it's not that bad, it's another thing to do some RL and see that reward hacking really got hands

[nickcdryan @nickcdryan]

@zmkzmkz haha ok that makes sense.

Either way, as the model size scales up the cost of that extra unembedding gets amortized away. Good to know for the smaller model comparisons though.

[nickcdryan @nickcdryan]

@zmkzmkz Nice, TOP uses fewer flops per step? I was thinking about the extra unembedding in TOP - do you cut out a layer somewhere to compensate?

[nickcdryan @nickcdryan]

@vikhyatk @omouamoua

[nickcdryan @nickcdryan]

@robinhanson An Atlantic article with some summaries and links out to the research.

Interesting article, I don't see this specific interpretation in there though.

https://t.co/W1S9TCSOmU

[nickcdryan @nickcdryan]

> score is calculated based on "how often words are used together"
> just pick obscure words that don't get used at all
> rejected

[KC @amphichrome_]

what’s the highest you guys got to on the divergent vocabulary test

[nickcdryan @nickcdryan]

@xeophon Nice eval! Interesting framing though.

Would you interpret the task as measuring a behavior, without opining on whether this behavior is good or bad? Or is it just good?

I'd say there are clear use cases where this behavior is nice and where it is not nice.

[nickcdryan @nickcdryan]

TIL some people are still using ROUGE

[elvis @omarsar0]

The Illusion of Progress

It's well known that there are caveats with benchmarks and metrics that measure LLM capabilities.

It's no different for hallucination detection.

"ROUGE fails to reliably capture true hallucination"

Here are my notes:

[nickcdryan @nickcdryan]

@immortal_0698 @natolambert Sure, but if you're not forced to use it then it's just a bad feature that you don't use.

[nickcdryan @nickcdryan]

@gatelice_ @ddkang Sure, and as you say there are scenarios where a datapoint is worth $100 or $1000 or $1 or $.01.

I'm trying to get the authors to provide the evidence they used for their estimates.