Anders Lie

building fast local inference @usemagnitude

[Anders Lie @anderslie]

We've been working hard to make it easy to run fully-local agents that are actually solid. If you've been meaning to try local models but want an optimized experience that works out of the box, give magnitude a shot!

[Tom Greenwald @tomgreenwald]

Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.

Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.

Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.

It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.

Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.

Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides

npm i -g @magnitudedev/cli

GitHub: https://t.co/5SjPQWkmrF

[Anders Lie @anderslie]

@AsherDale13 @omnaraai smh agents never heard of platform detection

[Anders Lie @anderslie]

@ishaansehgal the people will soon own their agents end to end

[Anders Lie @anderslie]

@nickbaumann_ Plenty of incentives to optimize datacenter-class inference, but lots to be improved still for local inference!

[Anders Lie @anderslie]

Many long nights went into making these kernels as fast as possible across different hardware.

A lot of cool techniques under the hood that make this work:

- Autotuning system that fits each kernel to specific hardware -> fast on any hardware
- Using hardware acceleration to the fullest extent where available (e.g. M5 tensor cores) -> faster everything
- Decode kernels that unpack quantized weights just-in-time to reduce bandwidth needs -> faster decode
- Quantized KV cache with fast decode kernels that compute with quantized values in-place -> less memory, faster decode
- Operation packing that fits two 8-bit multiplies inside one 32-bit multiply on Macs that don't have tensor cores -> faster prefill
- Filter unwinnable token candidates during unembedding by using top 4 bits as a bound -> faster decode
...and many more.

This all adds up to making Magnitude the fastest engine for local AI across all models and hardware we support.

And yet even more ideas for performance are in the pipeline, as well as ways to run bigger models on less hardware. Stay tuned!

[Tom Greenwald @tomgreenwald]

We made open models run over 2x faster than Ollama, without any new hardware

Introducing Magnitude, an open source inference engine that runs open models as fast as your hardware allows. It tunes its kernels on your machine to find the fastest way to run each model

Works on Apple Silicon, NVIDIA, AMD, or just a CPU

On average across 11 popular local models:
Mac M4 Pro vs Ollama (MLX) 1.32x faster prefill, 2.19x faster decode
DGX Spark vs Ollama (llama.cpp) 1.34x faster prefill, 2.17x faster decode

Ollama, LM Studio, and llama.cpp ship one-size-fits-all kernels, because tuning each model for each chip by hand would take forever

Magnitude tunes them automatically. When you download a model, it spends up to a minute on your actual chip trying different ways to split up the math (how many GPU threads team up, how big a chunk each one takes, how far ahead to load data) and keeps the fastest combination that doesn't change the model's output

Each model also comes paired with the best speculative decoding drafter, for even more speed

One click connects Magnitude to the agent you already use (Pi, OpenCode, Hermes, Codex, and more). It's also built specifically for agents:
• Each agent uses about half the memory, thanks to a quantized KV cache
• Memory grows and shrinks with your sessions, so your computer stays usable
• Agents share prefix caches, so several can run at once without slowing down

Next, we're working on getting even more out of your hardware:
• Expert streaming from RAM or disk, to run models bigger than your GPU allows
• Splitting one model across all your GPUs and CPU so nothing sits idle
• Attention pruning that skips low-impact math, for much faster prefill

Ships as a desktop app on Mac, Windows, and Linux
100% free and open source

Download → https://t.co/GgA3qLow9t
GitHub → https://t.co/lR8GMis9pF

[Anders Lie @anderslie]

@tomgreenwald local models go brr

[Anders Lie @anderslie]

@charles_irl O(1) but the 1 is actually infinity my b

[Anders Lie @anderslie]

@kitlangton Please make an interactive effect tutorial in the form of a limbo puzzle

[Anders Lie @anderslie]

@nicoleegong Gotta hit the lower haight cafes, du soleil and duboce park cafe are good vibes

[Anders Lie @anderslie]

@MichaelArnaldi We must hope that before everyone forgets how to write anything but slop that the models figure out how to deslop for us

[Anders Lie @anderslie]

@nickbaumann_ Dude don’t read my logs

[Anders Lie @anderslie]

@nicoleegong @Techweek_ the event value paradox. though if you get enough cool people at one event you attract other cool people

[Anders Lie @anderslie]

@tomgreenwald Gonna make the AI operating system also be able to run the fastest AI locally @dhh

[Anders Lie @anderslie]

@0xblacklight @samgoodwin89 yo what

[Anders Lie @anderslie]

@MistralAI @ClementDelangue I can finally get back to work

https://t.co/f9acAXIzK9

[Anders Lie @anderslie]

Assuming Mistral is able to release Le Chaton Fat in the next few days, there's literally zero point doing any meaningful work until it is released.

What can be done in 100 hours with Le Chaton can be done in 1 with Le Chaton Fat.

Hopefully this is figured out quickly.

[Anders Lie @anderslie]

Cool trick for faster decode: read half the bytes during unembedding.

How? Store unembedding weights in "planes" so you can read a 4-bit approximation first.

Use this approximation to filter for the highest logits.

Store the 5th bit in another plane, use it to eliminate more candidates.

Use the last 3 bits for the final 8-bit precision logits.

Even though a 4-bit representation was used to filter the logits, it's lossless because it only drops tokens that provably can't win.

Fewer bytes read here generally corresponds to faster decode (tested on an M4 Pro with Qwen 3.6 35B A3B, +6.5% tok/s).

[Anders Lie @anderslie]

@vikhyatk Come hang in lower haight

[Anders Lie @anderslie]

@RohitPaulK @usemagnitude Hey could you share your hardware and OS? Also if you wouldn’t mind cutting a GitHub issue with the details that’d be helpful!

[Tom Greenwald @tomgreenwald]

Magnitude has hit #1 on HN! Lots of great discussion from the community so far

TLDR is that Magnitude runs open models as fast as possible on your hardware. Up to 2x faster than llama.cpp in our benchmarking

It works on Mac, Linux, and Windows, on any hardware. Macbooks, NVIDIA and AMD GPUs, DGX Sparks, Strix Halos, or just a laptop CPU

Our kernels are tuned on your actual device for maximum performance. And it's built for agents, where sessions are long, several run at once, and you still want to use your computer for other things

Feel free to read more and join the discussion on HN :)