TensorTonic

Infrastructure to run ML & GPU algorithms in cloud-native sandboxes

[TensorTonic @TensorTonic]

https://t.co/Ks86tMrhGj

[TensorTonic @TensorTonic]

you can now implement E-means algorithm from scratch in numpy

https://t.co/rwSSNnQe4I

[Swann Bessa @swann_bessa]

Introducing e-means, an algorithm beating k-means in 10x fewer epochs.

With order of magnitude wall-clock speedups, e-means is released in FAISS and needs no hyperparameter tuning beyond k. It is robust to various scales and datasets. Still skeptical ? Just try e-means [1/10]🧵

[Manas @HUMANCODERMJ]

Starting back on @TensorTonic

[TensorTonic @TensorTonic]

we're launching a new study plan for JAX, built around ML systems.

compute per-example gradients with vmap, train with mixed precision and loss scaling, build a paged KV cache, spec decoding etc

https://t.co/rwSSNnQe4I

[TensorTonic @TensorTonic]

@VSK_HEIS3NBERG please contact us at [email protected]

[TensorTonic @TensorTonic]

you can now implement Google's EmbeddingGemma2 research paper from scratch in PyTorch

https://t.co/rwSSNnQe4I

[Omar Sanseviero (in SF) @osanseviero]

The EmbeddingGemma 2 paper is out! Enjoy!

https://t.co/z6zBOSYLPD

[TensorTonic @TensorTonic]

you can now implement a Diffusion model from scratch in Pytorch and Numpy.

start with gaussian basics, compute KL divergence, residual blocks, noise prediction loss, flow matching, small DiT block and more.

https://t.co/rwSSNnQe4I

[TensorTonic @TensorTonic]

the entire math needed to understand ML

https://t.co/8mDIcA1zFt

[TensorTonic @TensorTonic]

Inference engineering is becoming one of the most valuable skills in AI.

The field in 10 ideas:

> Prefill and decode are different workloads
Prefill processes the whole prompt in parallel and is usually compute-bound. Decode generates one token at a time, rereading the weights and the KV cache at every step, and is usually memory-bandwidth-bound. Most of inference engineering follows from this split.

> The KV cache is the real memory bill
Weights are a fixed cost. The KV cache grows with every token of every active request:
2 × layers × tokens × KV heads × head dim × bytes per value

> Fewer KV heads, smaller cache
MQA shares one key/value head across all query heads. GQA shares one per group. MLA compresses keys and values into a small latent vector. Same goal each time: less cache per token.

> FlashAttention cuts memory traffic, not math
It computes exact attention tile by tile in on-chip SRAM, so the N×N score matrix never gets written to HBM.

> PagedAttention fixes fragmentation
The KV cache lives in fixed-size blocks allocated on demand, so far more requests fit on one GPU.

> Continuous batching keeps the GPU busy
Requests join and leave the batch at every step instead of waiting for the slowest sequence to finish.

> Chunked prefill protects decode
Long prompts are split into chunks and scheduled alongside running decodes, so one huge prompt doesn’t stall everyone else.

> Prefix caching skips repeated work
Requests that share a prefix, like a system prompt, reuse its KV state. It speeds up prefill, not generation.

> Quantization and speculative decoding attack decode cost
Lower-precision weights mean fewer bytes moved per token. Speculative decoding lets a cheap draft propose tokens that the large model verifies in one pass. The gains from both depend on hardware, format, batch size and acceptance rate.

> Tokens per second is not the only metric
Watch TTFT (time to first token), ITL (time between tokens), P99 latency and goodput, the load you can serve while still meeting your latency targets.

Where to look when something breaks -

1. High TTFT → queueing or expensive prefill
2. Slow ITL → decode bandwidth, KV-cache traffic or prefill interference
3. OOM as concurrency rises → the KV cache outgrew memory
4. Poor multi-GPU scaling → communication

Every lever trades latency, throughput, memory or cost against another.

[TensorTonic @TensorTonic]

@meet_shoryak from here

https://t.co/rwSSNnQe4I

[TensorTonic @TensorTonic]

https://t.co/CnzlE18FsE

[TensorTonic @TensorTonic]

Why does dropout divide by (1 − p)?
(a popular ML interview question)

Take 4 activations: [2, 4, 1, 3]. Sum = 10.
Apply dropout with p = 0.5. On average, half the neurons get zeroed, so the next layer learns to expect a sum of around 5.

Then at inference, dropout turns off. Every neuron fires, and the next layer suddenly gets 10.

Double what it was trained on.

The fix: multiply every surviving neuron by 1/(1 − p).
Each neuron is kept with probability (1 − p) and boosted by exactly the inverse, so the two cancel. Its expected value stays the same.

Training and inference now see the same scale, and at inference dropout becomes a plain identity. No mask, no rescaling.

The visual below shows all three steps.

[TensorTonic @TensorTonic]

https://t.co/KqYvZGPiko

[TensorTonic @TensorTonic]

A 6 month plan for becoming a research engineer.

> Math and core ML
Linear algebra, probability, calculus and optimization, followed by loss functions, classification, metrics and evaluation. Evaluation needs more hours because a research engineer is usually the person who has to say whether a result is real.

> Deep learning from scratch
Neural networks, activations, optimizers, PyTorch and an autograd engine that you write yourself.

> Transformers and language models
Attention, embeddings and the full encoder block, then BERT and GPT pretraining.

> Post training
Fine tuning with LoRA and QLoRA, then SFT, reinforcement learning with verifiable rewards and how RL environments are built.

> GPUs and inference

GPU programming in CUDA and Triton, inference and KV caching, Quantization and speculative decoding. This stage lets you make your own experiments faster instead of waiting on someone else’s infrastructure.

Reproduce papers alongside all of it. Pick a recent one, implement it end to end, run the ablations and write up what matched and what did not. It is the closest thing to the actual job and it is what you can show in an interview.

Most of these stages are available on TensorTonic as problems where you implement the idea yourself and get graded against test cases, with GPUs provided for the heavy ones.

Start at https://t.co/jni5KcSizC

[TensorTonic @TensorTonic]

https://t.co/KqYvZGPiko

[TensorTonic @TensorTonic]

pretraining any LLM is a lot of small pieces that all have to be right:

> the tokenizer
> the shifted targets
> the causal mask
> the loss
> the optimizer
> the learning rate schedule

and any one of them can be wrong without throwing an error.

if you want to read more in detail, we wrote a full lesson on TensorTonic - https://t.co/iUMJc7e75S

[TensorTonic @TensorTonic]

https://t.co/rljvyRN9QB

[TensorTonic @TensorTonic]

Gradient checking is the most underrated debugging skill in deep learning.

1. Use a tiny input, a few parameters and float64. Not your full model.
2. Compute the gradient with your backward pass.
3. Compute it numerically too
4. Compare them with relative error, not the plain difference.
5. Check one parameter at a time to find which layer is wrong.

Build Micrograd from Scratch has 25 exercises, including gradient checks.

https://t.co/OWoGR6e5f0