Daniel Quelali

[ali @waterloo_intern]

it took 1 intern 3 months of continuous work, but eventually, a quantization method that beat every other algo in the market, including @nvidia's official modelopt

to explain why this matters, i ask for exactly 69 seconds of your attention (275 words @ avg reading speed of 238 wpm):

frontier models (like glm52) are huge (~0.8T params). as released, each parameter takes 2 bytes (bf16), so overall size is about 1.6 tb

a b200 has 180gb of memory. a node of 8 gives you 1.44 tb, barely fits weights, much less activations / kv cache

must quantize the model (reduce the size of each individual parameters) to serve. fp8 quantization means each parameter takes 1 byte (fits in 0.8 tb), fp4 takes 1/2 a byte (fits in 0.4 tb)

cutting the model to a quarter its original size is necessary for it to run a) cheap b) fast, and every lab serving models does this.

but, quantization lobotomizes the model if not done correctly (this is why you see people complain about @AnthropicAI nerfing claude or @OpenAI nerfing codex)

there are currently several algorithms (like Nvidia's official model-opt) that attempt to figure how to quantize a model with the least amount of damage.

they find the redundant layers that can be slashed, and sensitive/important layers that need to stay in full-precision.

these algo's have two drawbacks:
1) they take a long time to run
2) they quite often result in a sub-optimal configuration

for the past 3 months, a research (and, as always, waterloo) intern on our model perf team (@the_joshua_hill) came up with a new quant algorithm.

it consistently finds the optimal configuration:
a) in less time than SOTA
b) with more aggressive quant than SOTA
c) scoring higher on benchmarks than SOTA

achieving just one of the above is a feat on its own.

all three...excited for the paper to come out this week

[th0rgal @Th0rgal_]

Just reverse engineered my Oura Ring 5 so I can control my computer like a wizzard. @ouraring please send my love to whoever buried a feature to stream live accelerometer data

[Alok Bishoyi @alokbishoyi97]

https://t.co/fb5CmHTw7n

[Daniel Quelali @DanielQuelali]

@levelsio Couldn't. I mentioned pandadnb the other day :)

[Daniel Quelali @DanielQuelali]

@chiefofautism @0xSero You don't need short feedback loops, you could massively parallelize, then iterate. We can definitely optimize probabilistic output, too. E.g., social media

[Daniel Quelali @DanielQuelali]

@karpathy Btw, I had never written Rust in my life. Still haven't

[Daniel Quelali @DanielQuelali]

Autoresearch is helping me build the Stockfish for Settlers of Catan, while I sleep.

Before tackling RL for the players, I have agents maximizing my simulation throughput. They are continuously mutating the Rust engine.

Got a +330% increase overnight.

Thanks @karpathy !

[Daniel Quelali @DanielQuelali]

@karpathy Throughput (Playouts/CPU/Second): Benchmarking the performance through alternating treatments design. First, a short warmup that gets discarded. Then, interleaved A/B/A/B test, to reduce exogenous noise, like thermal throttling

[Daniel Quelali @DanielQuelali]

@levelsio Next thing you tell me is Sub Focus is your brother or something

https://t.co/v9slUJcGrv

[Daniel Quelali @DanielQuelali]

@levelsio wtf?!? Man, I used to play CS 1.6 while listening dnb, including Pandadnb. I've been following you on this platform for years without knowing!

[Daniel Quelali @DanielQuelali]

@steipete omg, wake-up call to start posting. I risk being flagged as tiny/small/degen account otherwise :p

[w̸͕͂͂a̷͔̗͐t̴̙͗e̵̬̔̕r̴̰̓̊m̵͙͖̓̽a̵̢̗̓͒r̸̲̽ķ̷͔́͝ @anthrupad]

Sonnet 4.6 debating Eliezer Yudkowsky before giving up to just walk them through the loss landscape

they generated this ffmpeg video

[Peter Steinberger 🦞 @steipete]

@notsunsakis @bcherny don't call it vibe coding - that's associated with yolo i smash head on keyboard, not thinking, engineering, building, testing, debugging, iterating.

agentic engineering, or just...coding. We move faster, but it's still hard.

[Bojan Tunguz @tunguz]

BREAKING: the AI community discovers seed averaging and ensembling. Something we’ve been doing on Kaggle since … forever.

[Daniel Quelali @DanielQuelali]

@jxnlco An agentic skills framework to look at your data

[Daniel Quelali @DanielQuelali]

@tunguz I remember that. Two weeks later, AutoGPT was released.

[Daniel Quelali @DanielQuelali]

@tunguz Same here. When building a Data Viz agent, using the good-ol' expert persona in the prompt still works and alleviates the problem. Hand holding is still needed, though it feels like being part of building the harness