BendTorch is out: neural networks trained in Bend 2, with the backward pass in the types. MNIST, a small Llama and SmolLM2-135M fine-tuning, each checked against a PyTorch twin from the same weights.
Every gradient has the type of the value it is the gradient of, so a transposed gradient or two layers that do not fit do not compile. The backward formulas of the linear layers, the chain rule through Seq and Fork, and the causal mask are laws that Bend's kernel checks.
It computes what PyTorch computes: MNIST's loss and its 9,108 test hits are PyTorch's, with the same bits on 1 and 16 threads. After 400 fine-tuning steps on Shakespeare, SmolLM2-135M writes "And her love is in the bosom more than thou art in me", and greedy decoding gives PyTorch's tokens. Of 98 bugs I planted on purpose, 96 are caught by a law or a test; the other 2 still compute the right result.
It is slower: about 6x PyTorch eager on MNIST and 4.3x on a SmolLM2-135M training step on 16 threads. It is faster only on a tiny model, where PyTorch's per-operation overhead dominates. On the GPU it is 1.7x slower than 16 CPU threads. BENCHMARK.md compares eleven systems, losses included.
All of the math is Bend: no BLAS, no C or CUDA kernels. It builds on bend-ml and on what llama.bend taught me about the Llama layers, and it is on BendHub as four packages ([email protected], plus -fast, -gpu and -load).
Cc @VictorTaelin
https://t.co/oCk5zbdLtA