Cascadia

Run powerful AI on the hardware you already own.

[Cascadia @withcascadia]

We’re working on upstreaming this into Cascadia, so that anyone with the available hardware can try it out.

Follow the progress (and leave a star) here:

https://t.co/OeMlTMCcDv

[Cascadia @withcascadia]

Cascadia is now serving Inkling (975B) from @thinkymachines across eleven Intel Core Ultra PCs, in a deployment with @IntelBusiness.

Quantized, Inkling sits at ~475GB in size, which we split into shards and distributed across the eleven machines. Each one runs a custom MoE engine built on top of OpenVINO.

Peak aggregate throughput reaches 60.29 tok/s at 88 concurrent streams.

[Jack Smith @jackwsmth]

Your AI wouldn't be down if you were using @withcascadia rn 🤠

[Tate @TateBerenbaum]

Weekends are for running @withcascadia on chromebooks…

Solid 4gb of RAM 🤠

[Tate @TateBerenbaum]

Why choose between local and hosted if you want to use both together?

We built yeschef so Claude Code and Codex can spawn subagents that run locally.

Works out of the box with @withcascadia, @Ollama, @exolabs, etc.

Save on your Claude usage limits and put your GPU to use.

[Intel Business @IntelBusiness]

Your business already owns the AI compute, but it’s just sitting idle.

@TateBerenbaum and https://t.co/KYmaHnTPDB work with Intel to help teams take advantage of this power potential by turning their existing Intel hardware into a private, on-prem, distributed inference network—one that you own and that can run a range of open-weight and local models without additional hardware spend or per-token cloud costs.

Learn more: https://t.co/PrsuN4j7SP

[Tate @TateBerenbaum]

The best place to find machines with 1.5TB of memory for <$10k?

2019 Mac Pro.

Using to test @withcascadia

[Cascadia @withcascadia]

@SergiiioBS Thanks for testing! Curious if you'd be willing to run the same throughput tests with OpenVINO instead of Cascadia for single device to see if this is an opportunity for improvement on our part or OV's.

[Tate @TateBerenbaum]

Excited to share the first research paper behind @withcascadia

We discovered pipeline micro-batching and speculative decoding can make up for much of the added latency of sharding models over TCP. This paper breaks down how, along with some novel work on NPU continuous batching.

[四花味醂 @shikamirin]

I did some follow-up testing with Cascadia!

Qwen3.8-27B didn’t work out of the box, so I had to export it to OpenVINO FP16, re-quantize it to INT4 with NNCF, and simplify the chat template for Minja before I could finally get it running.

On the Arc 140V, INT4_ASYM / g128 reached about 6.3–6.4 tok/s, while INT4_SYM / g128 reached about 6.6 tok/s. That’s noticeably faster than the llama.cpp baseline of about 4.4 tok/s even without speculative decoding.

Prompt Lookup caused the output to break down, and draft-model speculative decoding isn’t supported with this VLM-layout export.

Thanks for the interesting suggestion — this was a lot of fun to experiment with!

[Tate @TateBerenbaum]

I’ve been trying to figure out the best way to apply NPUs to inference.

Because NPUs are better at parallelism (and therefore digesting prefill tokens), and CPUs are generally better at token-by-token decoding, you can take a hybrid approach to inference if you have a CPU + NPU chipset.

TTFT speedup was 35x on a 1B model, and 40x on a 3B model (compared to solely CPU inference). Merged into @withcascadia a few weeks ago :)

[Cascadia @withcascadia]

Check it out here:

https://t.co/Mgbs3oMagk

Announcement:

https://t.co/mTok2OExIM

[Cascadia @withcascadia]

Run the most powerful AI models across Intel silicon.

Co-developed with @IntelBusiness.

[Tate @TateBerenbaum]

People shouldn’t have to build data centers to access frontier AI.

We just need it to run better on commodity hardware.

So we teamed up with @IntelBusiness to build a solution. This is Cascadia, and now you can use it too.