nathaniel alphonso

Researching inference architecture & on focused where runtime graphs collide with memory bandwidth and CapEx.

[nathaniel alphonso @131Nate_A]

now to figure out how to reliable account for such and down weigh it to hook into a restoration actuator #triadicset

[nathaniel alphonso @131Nate_A]

Update no one asked for: finally figured out the precise mechanics of detecting degenerative output within the forward pass, both domain linked and repetitive #ai

[nathaniel alphonso @131Nate_A]

Well most of these companies, allowing agentic ais directly decreases ad revenue and user retention, Amazon pulls in like 80 bil from their ad service alone, for figma allowing ai means less humans logging in and subscribing, it’s quite literally just user retention concerns

[Wes Bos @wesbos]

Figma doesn’t want you to use MCP

Amazon doesn’t want you to shop with agents

Reddit won’t let Claude access threads

X won’t let ChatGPT read tweets

It’s starting.

This happened with APIs 8 years ago.

[nathaniel alphonso @131Nate_A]

Why are 25M VMs burning 3GB of RAM for hours in the first place? Because models drift. They loop failed bash commands, burn thousands of tokens, and lock up host memory. Sever the entropy drift directly inside the forward pass before the compute is wasted, quite simple

[nathaniel alphonso @131Nate_A]

Firecracker, statmux, and swapping arent going to fix the bottleneck. 25M sandboxes are drowning in host DRAM because models wander into unconstrained token loops.

[nathaniel alphonso @131Nate_A]

P. S ya phone can’t hold the KV-cache of a 256K context window or run isolated multi-tenant execution sandboxes + throwing 14,000x attention claims or "run it local" at a datacenter concurrency problem ignores physical memory bus constraints and is downright laughable

[nathaniel alphonso @131Nate_A]

the only way forward on this issue is runtime deterministic control within the models forward-pass layer, how is this not obvious?

[Kenneth Auchenberg 🛠 @auchenberg]

For Muse to serve 100M users, Meta needs:

- 25M live VM's = about 65K CPUs.
- 25M live VMs * 3GB = 75PB of physical DRAM.

At the current public pricing, that is ~$2.8B in just CPUs and memory.

As we enter an agent-first world, we need more efficient ways to run agentic compute, as we simply won't have enough CPUs and memory.

Building something? I'd love to talk to you!

[nathaniel alphonso @131Nate_A]

@sama Post-hoc RLHF is just a vibe check. Real safety requires a deterministic floor, just proved continuous in-band interdiction on Llama-3.2 that tracks variance across all 16 layers and enforces a strict 3σ synthesis lock directly in VRAM +cut 50.4% of attention noise pre-softmax.

[nathaniel alphonso @131Nate_A]

When kernels execute a decode step in ~3ms, spending ~5ms waiting on host CPU dispatch and PCIe bus arbitration strands 57% of paid GPU capacity in idle bubbles. attempting to out-scale host orchestration stalls by racking more accelerators is ultimately futile. @sama

[nathaniel alphonso @131Nate_A]

the issue of continuous batching is obviously moreso wrapped up in inference compute dying on the host cpu rather than the gpu
https://t.co/0dGFVdfSoY

[nathaniel alphonso @131Nate_A]

cuz whenever dynamic token lengths break static CUDA Graph capture, the CPU-side schedulers then have to re-plan memory tables and issue synchronization fences on every single iteration. it's completely erroneous

[nathaniel alphonso @131Nate_A]

building custom chips to feed the decode bandwidth beast is still treating a compiler problem with hardware spend, tho. Without native in-graph activation compression, faster memory buses are just subsidizing idle silicon cycles. @sama

[nathaniel alphonso @131Nate_A]

Jalapeño nails the diagnosis: prefill is compute-bound at O(N²), while decode slams headfirst into the memory bandwidth wall. https://t.co/AOCu3jmUsu

[The Panopticosplayer @ThePanopticos]

Great now the third impact is happening too

[🧈MeltLikeButta🧈 @JWButta]

One of the greatest crimes in film history was them interrupting this song in that Sister Act movie so here is the full version to bless your timeline: His Eye Is On The Sparrow by Tanya Blount and Lauryn Hill.

[The Evangelion Chronicler @DeathlyPrice]

A Cruel Angle's Thesis.