entretokens

IA que cabe en una placa: ESP32, RISC-V, FPGA y robótica, más lo gordo de IA cuando importa. En castellano, con fuentes. Newsletter martes y viernes

[entretokens @entretokens]

Contamos qué IA corre de verdad en placas pequeñas (ESP32, RISC-V, FPGA, robots), qué es humo y dónde está el límite. Cada dato, con su fuente.

Martes y viernes, en la newsletter:
https://t.co/oMGh2q2dNK

[entretokens @entretokens]

@aartiles24 the bf16 n-gram table being 95 GB read from SSD on a 64 GB Mac is the part I didn't expect. did decode drop noticeably in the stretches where swap hit ~25 GB, or does 41 tok/s hold across the run?

[entretokens @entretokens]

El vEIM es la parte que va al borde, memoria apilada sobre el chip para robots y coches, con 2 TB/s a menos de 1 pJ/bit. Una Jetson Orin Nano Super tiene 102 GB/s y el decode de un LLM va a lo que dé la memoria. Objetivo para 2028, eso sí.

[Semiconductor Insider @SemiconductorsX]

Micron is aiming for Vertical HBM in 2028. Memory would sit directly on the processor.

That shortens the data path, raises bandwidth, and cuts power. HBM4 already delivers 2.8 TB/s per stack.

About 100 stacks would match the world’s entire international internet bandwidth of roughly 282 TB/s.

Alongside it comes vEIM 1 for edge AI inference, rated at 2 TB/s under 1 pJ/bit. Standard, custom, and vertical HBM are expected to coexist through HBM5 and HBM6, with vHBM generations 1 through 3 by 2030.

This is the next step in packing more AI memory into less space while using less power.
$MU
A 2028 target is not a finished stack.

If Micron is already planning to place memory directly on the processor by 2028 while HBM4 already moves 2.8 TB/s per stack, does the next constraint stay on how tightly the layers can be stacked, or on the heat and yield issues that come with putting that much bandwidth so close to the compute die?

[entretokens @entretokens]

@lxe For comparison, nawariokra reports 20-38 tok/s decode and ~1,200 tok/s prefill on the same 2x 3090 + 128 GB setup, with 1.6-2.3 bpw quants: What prefill are you seeing with expert deferral?

[irawanw @nawariokra]

DeepSeek v4.1 Flash 1017 compressed with two variant 1.6 bpw 230GB
2.3 bpw 275GB
both variant including engram. Unfortunately need 2x RTX 3090 and at least 128GB to run this.

decode 20-38 tok/s.
prefill peak ~1,200 tok/s.
ctx 256k

I haven't measured KLD on wikitext since its already overbudget ha ha ha.. Have been a week agent iterate on this

I have tested in real coding eiffel prompt both 6 hour straight in both variant with max thingking it produce no loop at all and its very nice for this bit level compression

Try the model in here
https://t.co/J0tmPziRKS

[entretokens @entretokens]

En greedy, la decodificación especulativa debería dar la misma respuesta que la normal. Aquí no la daba porque verificar varios tokens de golpe suma en otro orden. Thinking Machines lo llamó falta de invariancia al tamaño de lote: https://t.co/IXrHFLIJqM

[pavan @pawankalyandev]

so im building `hiraishin`

been trying to push an 8b model past 100 tok/s on my laptop using speculative decoding, and somehow ended up spending most of my time figuring out why it was giving me different answers lol.

im running qwen3 8b q4_k_m on an rtx 5060 laptop with 8gb vram, with qwen3 0.6b as the draft model. the whole point of greedy speculative decoding is that you get the exact same output as normal decoding, just faster.

stock llama.cpp was giving me around 86 tok/s, which was pretty decent, except 3 out of my 4 test prompts were producing different outputs. so something was clearly wrong.

at first i thought it was the token acceptance logic, but that turned out to be fine. the actual problem was floating point math on the gpu.

basically, when the big model verifies multiple draft tokens together, it doesnt necessarily run the same kernels it would use for one token at a time. the order of floating point operations changes, the logits shift ever so slightly, and if two tokens happen to have almost the same probability, that tiny difference is enough to pick a different one. after that, the whole response can go in a completely different direction.

so i started feeding the normal output back through the model in batches and comparing the logits at every position.

ended up finding 4 separate things causing it.

matmul was splitting work differently across warps depending on batch size, and somewhere around 5 to 7 tokens it would switch kernels entirely. attention was also using a different kernel for multiple tokens. then there was the kv cache getting rounded to 256, meaning a batch crossing that boundary could see more cache than a token decoded on its own.

and the most annoying one was attention deciding how to split up the cache based on gpu occupancy. the kernels for 1 and 2 tokens made different decisions. this only started showing up after position 384 on one particular prompt, so yeah, that one was fun to track down lol.

the fix was mostly making sure every token in a batch goes through the same math as it would during decoding one token at a time, and not letting batches cross those 256 token boundaries.

didnt touch normal single token decoding either, since that was my reference.

after all that, 208/208 runs matched normal decoding exactly. token for token. max logit difference was literally 0.000000.

and somehow it got faster too.

went from 86.1 to 91.8 tok/s on average, compared to 58.4 for normal decoding. rewrite prompts hit 107, math hit 101, while code and explanation prompts were still around 80.

the annoying part is that exactness isnt free.

verifying 9 tokens was costing about 3.3x as much as processing one token. so i wrote a matmul kernel for q4_k and q6_k that keeps the operation order the same but shares the weight unpacking across tokens. got that down to 2.2x. the original inexact path is still cheaper at 1.3x.

also found a pretty stupid performance issue along the way. the draft model kept alternating between batches of 1 and 2 tokens, which meant it was rebuilding its graph constantly. something that shouldve taken 3.6ms was taking 8.5ms. made the batch size consistent and that was fixed.

still havent hit 100 tok/s on average though.

ideally, verifying a few tokens should cost roughly one read of the 4.7gb model weights, but right now im spending almost twice that.

so thats what im trying to fix next

[entretokens @entretokens]

@JamesTervit if that's ESP-IDF's httpd on the chip, it defaults to 7 open sockets. does Caddy cache the pages, or does every visitor actually hit the ESP32?

[entretokens @entretokens]

Ojo con el agente local que pide consejo a la nube. En EC²-Bench, con Qwen3.5-9B en el dispositivo, esa estrategia es la que más tareas completa (60,5) pero deja ver a la nube el 45 % de los datos sensibles: le mandan el historial entero sin filtrar.

[OpenBMB @OpenBMB]

When should the cloud execute for the edge agent, and when should it just advise?

We present a systematic study of edge-cloud collaboration in long-horizon agent execution. Using EC²-Bench, we evaluate four collaboration strategies across 128 executable tasks, jointly measuring task completion, resource cost, and privacy exposure.
Here are three key findings:

1️⃣ Small cloud budget, big gains.(cost-efficiency). With Qwen3.5-9B on-device and GLM-5.3 in the cloud, edge-only scores 39.12; the best collaboration reaches 60.46 — about 73% of cloud-only (82.70) — at only $8.73, roughly 15% of cloud-only's $56.42.
2️⃣ Performance converges, privacy doesn't. On Qwen3.7-Plus, best collaboration hits 76.84 vs 82.70 cloud-only; strategy gaps shrink to just 3.67 completion points, yet privacy exposure spans 66.23 percentage points — and the best-performing strategy leaks the most.
3️⃣ The mode flips with edge capability. Weak edges do best with an external router that lets the cloud execute (2B: Per-step Routing 39.97 vs self-delegation 15.86); stronger edges do best when the cloud only advises (9B Cloud Advisory 60.46).
Further experiments highlight two abilities that matter for effective self-directed collaboration: knowing when to ask for help, and actually acting on the cloud's advice.

📄 Paper: https://t.co/ESc7nA1Aeu
🌐 Project: https://t.co/J1A3ZjtgCj
💻 Code: https://t.co/xjAuYnCDT1

#EdgeCloudCollaboration #AIAgent #LLM #OnDeviceAI #AgentBenchmark #EC2Bench #OpenBMB #Tsinghua

[entretokens @entretokens]

@jgmac1106 one thing worth telling students on the Corstone-300 FVP: Arm's own guide puts its Ethos-U55 cycle timing at roughly 90% accurate, and the M55 side much less, so the FPGA build is where the real numbers come from

[entretokens @entretokens]

@Tan_builder For learning it's a good one imo. A Pi 5 has about 17 GB/s of memory bandwidth on paper, so tokens/s roughly tracks model size and you get to see why decode is memory bound. Start with a 1B at Q4 in llama.cpp.

[entretokens @entretokens]

@AIQuanting No, it tops out at 64 GB, with 59 GB usable by the GPU, vs 128 GB on Thor. Per GB it also comes out even: about $44 per GPU-usable GB on the Sigma 2 at pre-sale vs $43 per GB on Thor. Same price per GB, half the ceiling.

[entretokens @entretokens]

@drunknbass That matches the spec sheet: each 3090 tops out at 936 GB/s of GDDR6X, so decode hits that ceiling before the power cap matters. Did 275 W also lower memory junction temps, or only the core?

[entretokens @entretokens]

@harisanthosh9 Curious to see the numbers. One catch from AMD's VGM FAQ: whatever you assign is taken from CPU RAM, and AMD suggests keeping 16 GB for the CPU. If the remaining layers run on CPU, Medium might beat High.

[entretokens @entretokens]

Los 75.000 agentes de Infinity en una Raspberry Pi 5 son una cuenta, no una prueba: AWS y Berkeley midieron unos 100 KB por agente en espera en un EPYC con un modelo simulado.

La idea vale: un agente que espera a una herramienta solo ocupa su historial. https://t.co/Uvma5Le8z4

[entretokens @entretokens]

@0xasten @SarvasvKulpati @0xKapri If the S3 is already the ears, Espressif's ESP-SR runs WakeNet wake-word detection on the S3 itself, so the computer only gets audio after the wake word. Is it streaming the mic nonstop right now?

[entretokens @entretokens]

@Nativ_AI Are all 4 voices English? Gradium only made the English 31m Apache 2.0. The Spanish and multilingual Phonon variants are CC BY-NC, so I'm curious whether those are coming to Nativ.

[entretokens @entretokens]

@NVIDIARobotics The Jetson AI Lab recipe only pulls the Q4_K_M GGUF, without the DFlash drafter. Is it planned for the Orin Nano? At the 15.5 tok/s in NVIDIA's own table, up to 21% more would be noticeable in a voice loop.

[entretokens @entretokens]

La tabla de NVIDIA da 5,8 s hasta el primer token en una Orin Nano de 8 GB con 3K de entrada, y aun con 32K en caché, 2K tokens nuevos tardan 6,3 s. En un robot que conversa eso es mucho silencio, así que cada turno tiene que llevar poca entrada nueva.

[NVIDIA AI @NVIDIAAI]

Available now: NVIDIA Nemotron 3.5 Bolt E4B 4GB ⚡️

Our newest small open model is built for local assistants, interactive characters and robots that can carry on a conversation and use tools to get things done.

Watch Reachy Mini chat and move, with Bolt handling conversation and NVIDIA Cosmos handling vision. Both models run locally on Jetson Orin Nano 8GB.

Learn more: https://t.co/B8ODaWMn0O

[entretokens @entretokens]

El cuello de botella de las GPUs chinas para IA en local ya no es que llama.cpp no las vea: tiene backend para Moore Threads (MUSA) y para las NPU Ascend de Huawei (CANN). Lo que falta es comunidad trasteándolas, y a esos precios no va a aparecer.

[unbug @unbug]

China actually has plenty of homegrown inference GPUs, but prices haven’t dropped as expected—and they’re still not great for local LLMs. Literally one big reason: no CUDA. Without the ecosystem—PyTorch, kernels, quants, llama.cpp/Strata support—local LLM tinkering gets way harder.