Alex Quach

ML @reductoai (prev @mit @liquidai). Here to understand network effects

[Yifei Hu @hu_yifei]

https://t.co/Q90usoIblj

[Alex Quach @theAlexQuach]

I saw some criticism on the writing style, but I feel like writing as to manifest and encourage the possibility of dramatically ambitious outcomes very nice to open debate on how the future of AI should look like

[Daniel Kokotajlo @DKokotajlo]

In AI 2027, we predicted that AI would take over the world or irreversibly concentrate power.

In AI 2040: Plan A, we've laid out our positive vision for what should happen instead.

[Alex Quach @theAlexQuach]

singing solfège for Sol, call me Sloprano

[Jakeup @myhandle]

writing fables with Fable, call me Aeslop

[Alex Quach @theAlexQuach]

there’s so much room to go

[Big Boss @0xBADB01E]

First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things.

As an example, let’s look at Llama 3.3. It has 70B of weights and at FP8 that’s 70 GB of memory which is enough to fit on a single H100. Now that H100 has 3.35 TB/s of HBM, so the fastest it can ever decode for one user is 70/3.35 ≈ 21 ms/token or ~48 tok/s while using under 1% of its FLOPs. Now if we pipeline it across 8 chips: each chip holds ~8.75 GB, which means it only needs 1/8th the bandwidth and 1/8th the FLOPs to sustain the same aggregate throughput. Now crucially the token/sec a user gets is limited by the amount of data that crosses the link. In current LLMs all that is a small amount of activations for LLama 3.3 it’s ~8 KB per token….

Yes, you read that right it’s 8 KILOBYTES we are sending over a <900 GB/s link. That’s only 9 ns of serialization time but the overhead of 224G PAM4 SerDes adds ~100 ns per link traversal with RS-FEC which is 11x longer than the payload itself. And then you have the NVSwitch adding ~300 ns per hop and you need to pay twice. That’s ~600 ns of just hardware latency wrapped around 9 ns of data making a 98% tax before software even shows up. Then NCCL’s collective stack turns 600 ns into 10-20+ us… all to move 8 kilobytes lol. For comparison 8 KB serializes over 10 Gigabit Ethernet NRZ, in just 6.6 us. Pipeline parallelism however doesn’t make a single user faster as the token still needs to visit every layer in the sequence, so per-user speed is still weights / per-chip bandwidth.

To get more speed per user token you need to use tensor parallelism and have all the chips work on the same layer simultaneously. TP costs you 2 all reduce OPs per layer, 160 per token on llama 3, that’s still kilobytes of traffic but with NVLink overhead it’s a massive tax and why pipeline parallelism on most models still gives more interactivity per user. However, this gives you a huge latency lever to pull that scales tokens per second with interconnect speed instead of memory BW.

The clever amongst you might have also realized that sharding doesn’t just cut memory bandwidth per chip it also cuts FLOPs per chip and is why we have such bad MFU on decode. So once you’ve sized the link for the memory, you need to size the compute for it too. This is called “balancing the pipeline”, and currently no shipping chip does it because they were all designed as standalone monsters. Remember Tokens/sec = ~aggregate memory BW / bytes touched per token. At batch 64 in FP4 you need ~250 FLOPs per byte, and Blackwell ships 1,250. Provisioned 5x more than the narrow pipe of HBM. Nobody saturates shit cause they are all building around HBM.

So now it all comes full circle. Parallelism reduces memory bw pressure and thus FLOPs but increases interconnect latency pressure. Despite having HBM and GigaSERDES we aren’t actually doing more work lol. But if you really wanted to balance the pipeline you need to match the memory bandwidth, the flops, and most importantly the interconnect.

So what does that look like ? Well if you build around LPDDR’s lower bandwidth, lower your interconnect latency, you actually can beat Nvidia on decode with a fraction of the silicon.

[Alex Quach @theAlexQuach]

@MainzOnX might be triton inference server (naming sucks)

[Alex Quach @theAlexQuach]

@AnthropicAI Is Fable still API-only after June 23⁉️

[Alex Quach @theAlexQuach]

pushing the pareto slop-usefulness curve

[NotebookLM @NotebookLM]

Doom scrolling but make it educational 🤓

Introducing Short Video Overviews in NotebookLM! Turn your most complex sources into 60-second, vertical videos that deep dive into any concept.

Rolling out now to Google AI Ultra and Pro subscribers on mobile &amp; web (free users soon!)

[Alex Quach @theAlexQuach]

The opposite of needle-in-a-haystack: tens of thousands of needles across hundreds of barns

[micro1 @micro1_ai]

Today we're publishing LongExtractBench, a benchmark commissioned by @reductoai and independently validated by micro1.

We evaluated seven production document extraction systems across the same 225 complex enterprise documents. The benchmark was intentionally difficult: documents averaged 358 pages and contained roughly 88,700 ground-truth fields each. Every system was evaluated using the configuration documented in the benchmark methodology.

Key findings:
• Reducto Deep Extract was the only system to successfully complete all 225 documents.
• Direct frontier LLM baselines achieved substantially lower completion rates on long, complex documents.
• In this benchmark, dedicated extraction platforms achieved higher completion rates than the direct frontier LLM baselines.
• Recall was the clearest differentiator. Precision remained high across systems, but recall ranged from 33.8% to 99.6%, highlighting which systems consistently captured the information contained in long, complex documents.

The full report includes the benchmark methodology, limitations, and reproducibility resources. Check out the report and results in the comments below.

[Alex Quach @theAlexQuach]

A perfect example of how Reducto is working on the long-tail hard problems -- extracting ~88,700 ground truth fields on each of 225 super long documents

[Adit @aditabrm]

Many companies are #1 in a benchmark they crafted.

We worked with @micro1 to create an independently audited benchmark to measure document extraction performance with long documents.

The results of LongExtractBench show the nuances companies are likely to find in the real world. micro1 tested frontier models with max reasoning and document processing platforms with their strongest configurations, and found notable precision/recall and completion tradeoffs across most.

Reducto’s Deep Extract leads the industry by a wide margin. 🧵

[Alex Quach @theAlexQuach]

@TheOtherGandhi_ some are saying opus with gigathink is solid

[Alex Quach @theAlexQuach]

@aditabrm @hu_yifei yeah 😂

[Alex Quach @theAlexQuach]

GLP, GLM, why did they skip over GLN?

[Alex Quach @theAlexQuach]

@_vatsadev 😂

[Alex Quach @theAlexQuach]

the hoarder in me can't take it

[Alex Quach @theAlexQuach]

2 resets available, things are getting ridiculous

[Alex Quach @theAlexQuach]

the future is single-process, multithreaded

[Evis Drenova @evisdrenova]

my ideal AI interface is a single never-ending chat thread.

i don't want to think about the concepts of sessions, context windows, worktrees, mcp servers or really anything else.

The harness should automate everything transparently.

[Alex Quach @theAlexQuach]

entropy increases

[Alex Quach @theAlexQuach]

the lion does not concern itself with boats 🇸🇳🇸🇳🇸🇳

[Maximiliano Bretos @MaxBretosSports]

Imagine you are heading to Times Square NYC to see “Les Miserables” and you stumble into this …
#Norway play #Senegal tomorrow at MetLife Stadium

[Alex Quach @theAlexQuach]

@palak_agarwal6 down to start my substack career