EXO Labs

Frontier AI on local hardware.
EXO 1.0 is now open-source (Apache 2.0):

[EXO Labs @exolabs]

The future of AI is open source and decentralized

[AI at Meta @AIatMeta]

"Exo's use of Llama 405B and consumer-grade devices to run inference at scale on the edge shows that the future of AI is open source and decentralized." - @mo_baioumy https://t.co/MFejDKgHIU

[0xSero @0xSero]

GLM-5.3-Flash in a backpack

Okay last one for today.

running on 2x M5 Mac Ultras (256gb each)

- single stream 150 tok/s
- 8 concurrent streams 380 tok/s
- 1.4 million token kV cache

This has been powered by 2x e-bike batteries for over 90 minutes generating nonstop

[This Week in AI @ThisWeeknAI]

Sam Altman’s AI predictions rely on a vision for “recursive self improvement” that does not yet exist, at least not publicly.

On TWiAI 36, the panel considers whether or not one frontier lab really could run away with the entire AI game, or if Altman’s “3 pathways forward for AI” prediction rests on a faulty premise?

Are the major labs talking about regulation so much because they’ve seen what kinds of models are coming next? Or just because regulation is good for whoever gets out to an early lead?

Expert panelists Jaleh Rezaei (Mutiny) and Alex Cheema (EXO Labs) join Jason and Lon for a brand-new This Week in AI.

@jason, @jalehr, @alexocheema, @lons, @mutinycorp, @exolabs

0:00 Cold Open / Show Intro
0:58 PayPal - Pay zero processing fees on your first $100K in eligible PayPal payment volume. Learn more at https://t.co/rxqKyhAWuO
2:56 How much hardware it takes to run local AI models
4:32 DHS orders DeepSeek out of federal systems
5:53 Why enterprise clients are uneasy about Chinese models
9:47 Could an open-source model "phone home" to Beijing?
14:10 Why enterprises are much more price sensitive about AI tools
16:39 Are Sam Altman's AI endgame theories "deranged"?
25:45 Why the major labs want more regulations
26:28 The ElevenLabs Archer controversy continues
34:30 Who actually owns the rights to your voice?
45:17 "Substack for AI Models"
46:20 China needs more chips than Nvidia can make
52:40 Are enterprises running more local models?

🎥 Watch the full episode here 👇

[Alex Cheema @alexocheema]

The memory unit economics of Apple Silicon are insane.

When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference.

M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s
DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s
DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s

M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station.

I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...)

I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory.

With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each.

The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.

[Julien Chaumond @julien_c]

Really great little DGX Spark guide from @0xSero

read it on HF https://t.co/8iKPdmtrep

[EXO Labs @exolabs]

https://t.co/c6i4mctkrK

[EXO Labs @exolabs]

💛

[Arto Bendiken @bendiken]

Just @alexocheema and @exolabs rocking it, and with yet more coming down the pike

https://t.co/E5dwM0l2WH

[0xSero @0xSero]

I interviewed DHH, the creator of Omarchy - The Agent Operating system + Ruby on Rails, one of the most important frameworks of our time.

He's a father, builder, race-car driver, and such an inspiring person.

One of the most positive people I've had the honor of meeting.

[0xSero @0xSero]

In the Lab rn!

Exo is cooking!

[Alex Cheema @alexocheema]

My GTM lead asked me to share this. For serious business people pls.

https://t.co/Tu6Sq1paiv

[@jason @Jason]

@Austen @johnternus has the clear path to owning sovereign AI on the desktop and phone thanks to Jobs making uber own silicon AND ramping up support for @exolabs AND putting 128gigs+ in machines

Predict they buy @perplexity_ai

[Alex Cheema @alexocheema]

Apple's AI strategy is Apple Silicon.

The iPhone, iPad, MacBook, Mac Mini and Mac Studio all use the same hardware architecture.

Apple Silicon is energy efficient, quiet, and the memory unit economics are incredible.

Apple has leaned hard into Local AI, now serving every segment of Local AI, from SLMs running on an iPhone/iPad to 2T models running on a cluster of 4 x 512GB M5 Ultra with RDMA using @exolabs (the cluster has 2TB memory @ 4.8TB/s).

Most inference will run locally, and Apple wants it to run on their silicon.

The main issue I see right now is the software. We need a stable, canonical inference engine rather than 10 unstable, incomplete ones. Who is solving this?

[EXO Labs @exolabs]

@BabushkaRobot Yes! We would appreciate you testing. Can you dm @alexocheema

[EXO Labs @exolabs]

exo featured on Apple's new M5 Ultra Mac Studio and M6 / M5 Pro Mac Mini pages.

Over the past year, we have worked closely with Apple on low-latency RDMA networking over Thunderbolt 5, enabling clusters of Macs to run massive models like Kimi K3 and GLM-5.3 at API speeds.

With RDMA, aggregate memory bandwidth across Macs scales ~linearly.

A cluster of 4 x M5 Ultra Mac Studios scale to an aggregate memory bandwidth of ~4.8TB/s. Previously, these were speeds only achievable with data center GPUs.

Our vision is a data center on every desk. Apple Silicon's superior memory unit economics, power efficiency, and out of the box experience for Local AI make that possible.

Thank you to @angeloskath, @awnihannun, @twid and countless others at Apple who tirelessly to bring this technology to the world.

[Alex Cheema @alexocheema]

hiring a founding designer for @exolabs
in-person in SF
$20k referral fee

https://t.co/V5wkdAq6Hm

[EXO Labs @exolabs]

@gingofthesouth 💛

[EXO Labs @exolabs]

@eiselems Yes.

[EXO Labs @exolabs]

@GedHughes71 🫡

[EXO Labs @exolabs]

Our patch enabling direct USB-C networking between Apple Silicon Macs and DGX Spark has landed in the mainline Linux Kernel.

This simplifies heterogeneous Local AI setups like P/D disaggregation over an ordinary USB-C cable.

https://t.co/XdMVGF8okb

[Alex Cheema @alexocheema]

Interesting article in @theinformation.

Macs are Apple's fastest growing business right now, driven by massive AI demand.

Apple's AI strategy IS the Mac.

Trillion dollar opportunity imho.

https://t.co/lcRRJ5cmIZ

[EXO Labs @exolabs]

@mweinbach 💛