haybales

Developer

[haybales @hybls]

@cremieuxrecueil Anthropic has really killer models and aesthetic but the rest of their company kinda sucks

[haybales @hybls]

@theo Need CUDA or MLX native for the work I’m doing. Whenever they update the mac minis I will be there

[haybales @hybls]

@SemiAnalysis_ I think the transformer architecture (sans-attention) is the real critical piece! Attention can be swapped out at the head level. The transformer architecture just lets you build giant systems without gradients blowing up

[haybales @hybls]

Top 5 on BabyLM (non-challenge submission) with my custom constant-memory linear-scaling StateHead architecture.
Beat several strong baselines on the Strict-Small track (10M words). Trained efficiently on my macbook air.
Lots of performance still on the table!

[haybales @hybls]

@Halo Hate the fat halo. Even if it’s lore-inaccurate I liked the elegance of the big skinny ring arcing overhead

[haybales @hybls]

@ID_AA_Carmack Should we bury our data centers

[haybales @hybls]

All models ~1M params, trained on 1 epoch of 20M tokens of tinystories.

[haybales @hybls]

Working on an experiment where I take a transformer model and replace the attention heads with super-basic RNN's, each head has its own state.

Calling it StateHead, and I'm getting super unexpected results compared to vanilla transformers and even mamba2, same layer/head/params

[haybales @hybls]

@alex_whedon @deepmatmul Still no hints of how they did it!

[haybales @hybls]

@Laz4rz Learning rate too high, it keeps overshooting and correcting

[haybales @hybls]

@aidaniil AWS uses open source Firecracker to spin up VM’s super fast for lambda/serverless stuff, might be useful

[haybales @hybls]

@basedjensen I’m still suspicious of the sum k=4 and quantization might lose information. We need clear needle in a haystack benches

[haybales @hybls]

Anthropic needs to bury mythos. They’ll need to match or beat it to keep up, and therefore have to bury all the safetyism they pulled

[haybales @hybls]

Claude for sure gasses you up very subtly compared to GPT 5.5

[haybales @hybls]

GPT 5.5 thinking is corny

[haybales @hybls]

@hamptonism @HamptonLibrary Brother… learned hierarchical world models

[haybales @hybls]

https://t.co/CDloH0PzFY