Tensor-Slayer

llm psychosis 1.0 survivor circa sonnet 3.5

[Tensor-Slayer @TensorSlay]

Exploring Direct Tensor Manipulation in Language Models: A Case Study in Binary-Level Model Enhancement: https://t.co/lQOLDFVzf0

[Tensor-Slayer @TensorSlay]

https://t.co/clXiiDOZqV

[Tensor-Slayer @TensorSlay]

Pretraining without backprop, 1.58 seconds to 4.2 loss.

This one's a tuning win, not a trick every run can use like Drift Leap (shoutout to the LAWA paper).

@RobertTLange is right, Track 2 is a systems race now. What I learned that might save others time is in the PR below. But 4.2 is mostly n-gram stats (bigram counts alone get 5.1), so I think Track 2's target needs a relook.

[Tensor-Slayer @TensorSlay]

Pretraining without backprop, 14.6 seconds to loss 4.2

[Robert Lange @RobertTLange]

0th-order pretraining keeps getting faster. I just opened a new Noise track-2 PR: a 37.7M-parameter transformer trained to val loss 4.2 in 1.66 s on one 8×H100 node, with no backprop anywhere. The current record is 2.46s.

Two days ago the track-2 record was 47 s. Then came identity-path credit, moment kernels on tensor cores, structured starts, tail averaging, and a PyTorch port of our JAX trainer, built by a lot of people in the open.

This PR doesn't change the algorithm at all. It uses #39's estimator and #38's constants with a faster update: 6.27 → 4.44 ms per update. Val matches #39 within 0.0004 on every seed, so the gain is pure systems.

Most changes were worth 1–4%. Together, they cut 40% off #37's update. Where the time went:

- The credit kernels and matmuls cost exactly the same as before.
- 0.8 ms of eager PyTorch copies and casts were compiled away.
- The credit kernels build their features in-kernel, merge their products (14 → 6 per tile), and run every block in one launch: 553 → 349 kernels per update.
- Gradients and weight gathers move in bf16, and both all-to-alls run under compute: exposed NCCL is 0.25 → 0.10 ms.
- GPU idle time is 0.42 → 0.16 ms.

A critical note on the benchmark itself: At 1.66 s, track 2 has become a systems benchmark. The last 30% came from kernel launches, copies and collectives, not from better credit assignment. Each update now does the same math in fewer launches. That's fun engineering, but it says little about whether forward-only learning scales. And 4.2 is a low bar. It is only 0.33 nats past n-gram statistics.

Track 1 (lowest val loss within 1 epoch) now edges past the backprop + Adam reference on loss (3.722 vs 3.769), but it needs 82 minutes on 8×H100 to get there. Matching backprop's loss is possible today; matching its efficiency isn't really close (yet?).

[Tensor-Slayer @TensorSlay]

https://t.co/ZaMKwK5hya

[Tensor-Slayer @TensorSlay]

Pretraining without backprop, 14.6 seconds to loss 4.2

[Samip @industriaalist]

Replacing backprop with zeroth-order optimization is inevitable.

Introducing Noise: a speedrun for pretraining without backprop.

Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware.

We built Noise to iterate on this faster.

[Zephyr @zephyr_z9]

Inference Gross Margins - 87.76% to 96.94%
Assumptions:
97% cached input, 2% uncached input and 1% output tokens
Prefill: 80k uncached input tokens/s/GPU.
Decode: 16k output tokens/s/GPU (250 tokens/s/user)
Token mix: 970M cached input + 20M uncached input + 10M output per billion total tokens.

It takes around 875 GPU seconds (250 for prefill & 625 for decode) to process 20M uncached input + 10M output tokens.
Assuming GPU rental cost is around $3-$6/hr, the GPU system cost per billion tokens is around $0.73-$1.46
1B tokens generate $11.91 in revenue at off-peak prices and $23.82 at peak prices
At off-peak prices, inference margins are between 87.76%-93.88% depending on the GPU rental cost ($3-$6/hr)
At peak prices, inference margins are between 93.88%-96.94% depending on the GPU rental cost ($3-$6/hr)

[Tensor-Slayer @TensorSlay]

@willcb Me in 2024 btw

[Matt @Matthewagi]

LLM Writing is limited by semantic understanding. I wrote about it in my blog. This is the first essay in a series on LLMs and writing quality.

The link is in the next tweet.

[Tensor-Slayer @TensorSlay]

i love opus

[Loktar 🇺🇸 @loktar00]

DeepSeek V4.1 Flash 1017 everyone's talking about it for a reason.

It does great with 3d built this honeybee on a sunflower like it was nothing.

Used Deepseek Harness, 1 turn, looks so good!

[Tensor-Slayer @TensorSlay]

Fable 5.5 spotted

[pushkar 🪔 @thepushkarp]

i was blown away by the naturalness of bulbul v4 when i first saw the demo at epoch a couple months ago

we're launching v4 flash today and bulbul v4 pro to follow soon!

[Amit Prakash @amit05prakash]

Can AI agents do interpretability research?

At @PrimeIntellect Residency program, we built BioCircuit, an RL environment where agents trace attribution graphs on Qwen3-4B, find the features behind a concept and get graded by intervening on prompts they've never seen.

Link : https://t.co/0xVymHoCM3

[Tensor-Slayer @TensorSlay]

@andrew_n_carr @NaderLikeLadder Clients are out fabling consultants so its appearing bleak

[Tensor-Slayer @TensorSlay]

https://t.co/djD4gFS2K4

[Tensor-Slayer @TensorSlay]

Piphone is on Omarchy store >> https://t.co/djD4gFS2K4

[Tensor-Slayer @TensorSlay]

Omarchy plugin layer is amazing @dhh , truly one step closer to agentic OS.

Opus and i built Piphone: an @OmarchyLinux plugin that lets an agent drive my iPhone over USB.

[Tensor-Slayer @TensorSlay]

Thank you @hancore_linux

[Tensor-Slayer @TensorSlay]

@dhh @OmarchyLinux @tom_doerr @AndrewCurran_

[Tensor-Slayer @TensorSlay]

Omarchy plugin layer is amazing @dhh , truly one step closer to agentic OS.

Opus and i built Piphone: an @OmarchyLinux plugin that lets an agent drive my iPhone over USB.

[Tensor-Slayer @TensorSlay]

@dhh @OmarchyLinux tagging for visibility

[Tensor-Slayer @TensorSlay]

@TheXSami Yo thank you means alot