Juan Mugica

I build, therefore I am. Previously at Google and MIT, now figuring out creating a company. You’ll find me running, coding, training, selling, cooking.

[Juan Mugica @loonstep]

Great read. I’m amazed @karpathy was already talking about speculative decoders in 2023. Now they become real

[alphaXiv @askalphaxiv]

DeepSeek just published DSpark, a speculative decoding system that boosts live DeepSeek V4 serving throughput by 51% to 406% under stricter latency targets.

With how most speculative decoding methods draft more tokens, but waste verification compute when those tokens get rejected, DSpark fixes this with a semi autoregressive drafter for more coherent long drafts, plus a confidence scheduler that only verifies prefixes likely to survive.

It also gives 60% to 85% faster per user generation at matched throughput.

[Juan Mugica @loonstep]

amazing this is from 2023

[Andrej Karpathy @karpathy]

Speculative execution for LLMs is an excellent inference-time optimization.

It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might think). This unintuitive fact is because sampling is heavily memory bound: most of the "work" is not doing compute, it is reading in the weights of the transformer from VRAM into on-chip cache for processing. So if you're going to do all that work of reading in all those weights, you might as well apply them to a whole batch of input vectors. I went into more detail in an earlier thread:
https://t.co/Lbtpq4VDeY

The reason we can't naively use this fact to sample in chunks of K tokens at a time is that every N-th token depends on what token we sample at time at step N-1. There is a serial dependency, so the baseline implementation just goes one by one left to right.

Now the clever idea is to use a small and cheap draft model to first generate a candidate sequence of K tokens - a "draft". Then we feed all of these together through the big model in a batch. This is almost as fast as feeding in just one token, per the above. Then we go from left to right over the logits predicted by the model and sample tokens. Any sample that agrees with the draft allows us to immediately skip forward to the next token. If there is a disagreement then we throw the draft away and eat the cost of doing some throwaway work (sampling the draft and the forward passing for all the later tokens).

The reason this works in practice is that most of the time the draft tokens get accepted, because they are easy, so even a much smaller draft model gets them. As these easy tokens get accepted, we skip through those parts in leaps. The hard tokens where the big model disagrees "fall back" to original speed, but actually a bit slower because of all the extra work.

So TLDR: this one weird trick works because LLMs are memory bound at inference time, in the "batch size 1" setting of sampling a single sequence of interest, that a large fraction of "local LLM" use cases fall into. And because most tokens are "easy".

References
https://t.co/sIBCSmsyKN
https://t.co/uSpmTzfWhR
https://t.co/7t7orHBybo

[Juan Mugica @loonstep]

está guapo el Antrópico

[Juan Mugica @loonstep]

@livekit This is gold

[LiveKit @livekit]

Voice AI has a benchmarking problem.

Everyone claims their end-of-turn model is the best, but you couldn't actually compare them. Datasets are proprietary, methods are opaque, and there is no shared ground truth.

That changes today.

We hit this while developing Turn Detector v1, so we open-sourced eot-bench. 5,000+ real user conversation turns across 14 languages, an evaluation harness that measures the real production tradeoff between latency and false cutoffs, and a live public leaderboard.

This should become the default way we evaluate turn detection models.

[Juan Mugica @loonstep]

@TomasStambulsky @levelsio @fdotinc @benln @JuanCereigido So cool! 😍

[Juan Mugica @loonstep]

@davidzh @livekit Thanks David 👀

[Juan Mugica @loonstep]

This is crazy cool

[Alex Volkov @ AI Engineer @altryne]

MidJourney just announced... a full body ultrasound! Yup... read on because it's as crazy as it sounds.

"As powerful as MRI and as casual as a trip to the spa"

They are calling it "the @midjourney scanner"

Insane details:

- First, the scale. The device uses 8,960 individual transducers arranged in a ring around your body

- The precision is the most jaw-dropping part: it resolves motion at the picometer range. It can image internal tissues finer than the width of an atom. We are talking sub-atomic level diagnostic capability

- The compute requirement is massive. The system processes 17 gigabytes of data per second.

It takes 40GB of raw data to reconstruct just one cross-sectional slice. And they are planning to scan 100 slices?

- Midjourney claims that fewer than 12 of these machines could perform more full-body scans than every MRI machine on Earth combined.

Welcome to the future of healthcare!

Not only these scanners are announced, they will exist in a "Midjourney SPA" - with hot tubs, saunas, cold plunges, and 9-10 whole body scanners.

[Juan Mugica @loonstep]

@victorgxn @appfounders_ @adapty shit the 35 degrees in Madrid hit harder seeing this haha

[Juan Mugica @loonstep]

I'm picking cool videos and doing some reverse engineering with CC so I can craft other ones with Hyperframes.

[Juan Mugica @loonstep]

@MarkVillacampa Brutal

[Juan Mugica @loonstep]

@metasurfero 😂

[Juan Mugica @loonstep]

what model is gemini code review running on?

[jason @jxnlco]

jason from the codex team here,
heres a draft on codex maxxing and the primatives i use on a daily basis

https://t.co/DR4N6xtAwe

would love any feedback

[Thariq @trq212]

https://t.co/MXt5XS4xBX

[Álvaro @metasurfero]

Cómo se construye una de las herramientas open source más usadas del mundo? La semana que viene lo hablamos con @sameer_alsakran, CEO de @metabase, en Mad Builders.
Link en comentarios 👇

[Álvaro @metasurfero]

This is why we created Mad Builders in Madrid

[Juan Mugica @loonstep]

@GuliMoreno @matiii @exp8fellowship let's do it!! 🚀

[Álvaro @metasurfero]

The best of Madrid at Mad Builders today, sharing their 10x dev stacks. What a night