Hamster Labs

Engineering, research and open source releases from the @usehamster team

[Eyal Toledano @EyalToledano]

@tobi @jeffreyguenther if you want to try doing this yourself, take a lookie here https://t.co/NSGvO78lJc

[Hamster Labs @HamsterResearch]

@EyalToledano @tobi @jeffreyguenther Holy alpha leak batman!

[Eyal Toledano @EyalToledano]

More Qwen3.8-Flash-Next breakthroughs, releasing soon in a new tiered model and supported by a custom MLX engine built by the @HamsterResearch team

- you only ever download the bf16 tiered model (upcoming)
- pMLX can quantize it up/down bf16/q8/q4/q3 on the fly
- 130-150 tok/s aggregate decode (crucial for sub agents)
- adjustable n-gram streaming from nvme
- adjustable experts residency ratio
- prune/REAP experts on the fly
- powered by pMLX

hundreds of config possibilities self-tuned from your hardware

examples for code-editing (which this decodes 1.5x faster):
- q4 @ 32k context, 95% experts in ram, n-gram in memory, 87 tok/s burst, 58 tok/s sustained (72gb memory), 149 tok/s aggregate
- q4 @ 256k context, 95% experts in ram, n-gram in memory, 87 tok/s burst, 50 tok/s sustained (74gb memory), 130 tok/s aggregate
- q3 @ 32k context, 95% experts in ram, n-gram in memory, 111 tok/s burst, 74 tok/s sustained (54gb memory), 190 tok/s aggregate

lowest memory footprint config: 12gb 🤯
- q3 @ 8k context, 30% experts in ram, n-gram on disk, 16 tok/s burst, 10 tok/s sustained (12gb memory), 29 tok/s aggregate

launch targets achieved, now packaging it up for you!

[Eyal Toledano @EyalToledano]

"which quant should I download?" is a question you may never have to answer again

the team @HamsterResearch has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it

tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need

I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs.

short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram

long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes.

there are many possibilities since I have the ram to spar

- if i need speed, Q4 100% resident (74gb ram @ 62 tok/s)
- if i need balance, Q8 95% resident (76gb @ 38 tok/s)
- if i need accuracy, bf16 70% resident (96gb @ 16 tok/s)

given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed

last thing to optimize is speed. there are so many things we want to power with open models at @usehamster and these 180b-300b class models have the potential to play a big role in that

[Hamster Labs @HamsterResearch]

@jayuhdinger @EyalToledano @samfajreldines Vision tower unaffected! Still has vision and multimodal

[Eyal Toledano @EyalToledano]

https://t.co/pmMcZzS2Jt

[Hamster Labs @HamsterResearch]

@iansltx @EyalToledano Enjoy! https://t.co/pYuEmQVTfi

[Eyal Toledano @EyalToledano]

Huggingface Links

Model
https://t.co/kxeAbtS6Mh

MTP drafter
https://t.co/lKJEKcujWr

[Hamster Labs @HamsterResearch]

@Siddhartha__me @EyalToledano

[Hamster Labs @HamsterResearch]

Huge fans of mlx-vm and mlx-vlm

Many optimizations we want to try are constrained by a lot of overhead in the inference pass

It’s part of why MTP doesn’t deliver much gain outside M5 chips for example (time to choose tokens is slower than just generating the tokens)

Our new pmlx engine collapses much of that overhead which makes the MTP math work and deliver 2x+ decode acceleration, even on non M5 chips

We look forward to releasing the engine and kicking off community contributions!

[Eyal Toledano @EyalToledano]

Another Qwen3.8-Flash-Next breakthrough

I stored 60% of experts on disk which stream to memory on-demand. Similar technique as n-gram streaming

Just eval'd with FULL experts (no prune) running on just 37gb of memory and decoding at 40 tok/s on my M4 Max

Hold on to your socks!

[Hamster Labs @HamsterResearch]

@onthexitter69 @EyalToledano @huggingface 🤝

[Eyal Toledano @EyalToledano]

From the @HamsterResearch lab:

Introducing Qwen3.8-Flash-Next-REAP-288-MLX-4bit, a 180B-class model running on just 39gb of memory

- MLX-native 4-bit 60% smaller than stock q4
- Pruned 512→288 experts via REAP
- 91.5% HumanEval (vs 93.9% stock)

@huggingface links below ↓

[Eyal Toledano @EyalToledano]

To answer questions about a document, a model normally keeps a whole document in its context.

But context is not free: every token in the prompt leaves a slice of key/value cache in memory for as long as the model reasons over it. That cache grows straight with document length.

An adapter gives a model knowledge that lives inside its weights, not in the context window. That means increasing context doesn't cost additional KV cache. The adapter size is fixed.

Adapters are good for:
Stable knowledge you want the model to just know like your domain, product docs, house style/voice, accumulated methods,

It's always-on, costs zero context budget, with approximate recall meaning it will "speak from" the knowledge rather than read it verbatim

Adapters are not good for:
Anything that changes per request (today's data, recently created/modified docs), or anything you need quoted verbatim. That stays in context/RAG.

If the same thing appears in every prompt, it should be baked it into the adapter. If it changes between prompts, it shouldn't.