galo

[email protected] | @HP | all things distributed | 🎿

[Alex Cheema @alexocheema]

Guess what?

local ai for everyone (: we did it

https://t.co/b6s9nCUDko

[Unsloth AI @UnslothAI]

Gemma 4 now runs 2x faster with MTP GGUFs! Run locally on just 6GB RAM. ⚡️

MTP enables Google Gemma 4 run ~1.4–2.2× faster with no accuracy loss.

Gemma 4 12B MTP can run at 162 t/s vs. 52 t/s without MTP. 31B reaches 101 t/s.

GGUFs + Guide: https://t.co/c4gAUlb6YE

[Shubham Saboo @Saboo_Shubham_]

This is the FUTURE.

Local models running on your phone with borrowed compute from your MacBook.

100% free and works with all Open Source models.

https://t.co/S6KeJulpYT

[Georgi Gerganov @ggerganov]

Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp

Over the last few months llama.cpp maintainers and engineers from NVIDIA collaborated to improve the multi-GPU performance in ggml. This resulted in significant performance gains on RTX systems and laid the groundwork for hardware-agnostic tensor parallelism in ggml.

For more information on this and other advancements in the low-level inference engine of llama.cpp, check the technical blog by @NVIDIARTXSpark below

[NVIDIA RTX Spark @NVIDIARTXSpark]

Build on-device personal AI agents on Windows PCs with new tools from NVIDIA and Microsoft, including secure sandboxing, faster local inference, multi-GPU support, and RTX acceleration for Windows AI APIs.

Read the technical blog: https://t.co/vNIsEded46

[Georgi Gerganov @ggerganov]

Highlighting the new WebGPU backend in llama.cpp/ggml

The work to bring full-fledged WebGPU support in llama.cpp started about an year and a half ago. It has been lead by @reeselevine and team at USCS.

For more information, checkout the interactive blog and paper in the quoted post. Here are 2 excerpts from the paper, summarizing the implemented software architecture.

[0xSero @0xSero]

You ever go on Huggingface and see:

- GGUF
- Unsloth
- Llama.cpp
- Dynamic GGUF
- Q_4_M / IQ_4XL etc.

Here's what's going on under the hood.

https://t.co/ebtE28ikvI

[Unsloth AI @UnslothAI]

We benchmarked Gemma 4 26B-A4B GGUFs to identify the best performing quants.

Unsloth ranks first in ALL 22 of 22 model sizes on mean KL divergence, making them SOTA.

GGUFs: https://t.co/uOj5RAckXc

[Georgi Gerganov @ggerganov]

I think the consensus is that Qwen3.5 is a step change so atm I would recommend explore that, given that it covers a range of sizes suitable for all devices.

Note that the main issues that people currently unknowingly face with local models mostly revolve around the harness and some intricacies around model chat templates and prompt construction. Sometimes there are even pure inference bugs. From typing the task in the client to the actual result, there is a long chain of components that atm are not only fragile - are also developed by different parties. So it's difficult to consolidate the entire stack and you have to keep in mind that what you are currently observing is with very high probability still broken in some subtle way along that chain.

But things are improving on all levels and everything will become better across the board soon.

Best way to evaluate things IMO:

- Start with full quality models that you fit on your hardware
- Make sure you know what your harness actually does. F.ex. don't expect to hook Claude Code or Codex to some local model and the magic to happen. The developers of CC don't care (yet) if it is compatible with Qwen3.5. Best is to write your own harness so you know what happens every step of the way. Or use llama-server's webui (we now have MCP support out of the box)
- When things start to click, look for optimizations to make it faster. Here is where you can start quantizing for speed or looks for some advice in the community for optimal parameters

So I can just say that on the low-level inference side, we will ship the right solution for sure. We still need to make the user-facing stack work better with local models - I'm hoping this will happen, though I feel less capable to control that.

And to answer your question more straightforward, I've experimented with the following models and have found useful applications (mostly around chat, MCP and coding) with all of them:

- gpt-oss-120b
- Qwen3-Coder-30B
- GLM-4.7-Flash
- MiniMax-M2.5
- Qwen3.5-35B-A3B

With the exception of gpt-oss-120b and MiniMax-M2.5, I've used Q8_0 variants to keep most of the original quality.

Unfortunately, I am not familiar with tool calling benchmarks specifically, so I cannot recommend. From my PoV, as long as we make sure the fundamental inference computation is correct, tool calling efficiency will depend just on:

- Model intelligence (something we do not control)
- Chat template parsing (something we are still actively improving on our end in llama.cpp)

[Georgi Gerganov @ggerganov]

llama.cpp at 100k stars

now that 90% of the code worldwide is being written by AI agents, I predict that within 3-6 months, 90% of all AI agents will be running locally with llama.cpp 😄

Jokes aside, I am going to use this small milestone as an opportunity to reflect a bit on the project and the state of AI from the perspective of local applications. There is a lot to say and discuss and yet it feels less and less important to try to make a point. Opinions about viability of local LLMs are strongly polarized, details are overlooked, the scientific approach is lacking. Arguments are predominantly based on vibes and hype waves.

One thing is clear though - local LLMs are used more and more. I expect this trend to continue and likely 2026 will end up being one of the most important years for the local AI movement.

I admit that I didn't expect the agentic era to come so quickly to the local LLM space. One year ago, the available models were too computationally expensive for doing long-context tasks. There wasn't an obvious path towards meaningful agentic applications. The memory and compute requirements were huge. Last summer, with the release of gpt-oss, things started to change. It was the first time we saw a glimpse of tool calling that actually works well within the resource constraints of our daily devices. Later in the year, even better models were released and by now, useful local agentic workflows are a reality.

Comparing local vs hosted capabilities at a given moment of time is pointless. To try put things into perspective:

- We don't need frontier intelligence to automate searches and sending emails
- We don't need trillion parameter models to be able to summarize articles or technical documents
- We don't need massive GPU data centers to control our home appliances or turn the lights off in the garage

I believe that there is a certain level of intelligence we as humans can comprehend and meaningfully utilize to improve our working process. Beyond that level, access to more intelligence becomes unnecessary at best and counterproductive at worst. I also believe that that level of useful artificial intelligence is completely within reach locally and it has always been just a matter of implementing the right software stack to bring it to the end user.

With llama.cpp, I am confident that we continue to be on the right track of building that software stack!

The llama.cpp project is going stronger than ever. With more than 1500 contributors, the project keeps growing steadily.

From technical point of view, I think that llama.cpp + ggml is the only solution that actually makes sense. That is, the software stack must run efficiently on every possible device, hardware and operating system. The technology is too important to be vendor-locked. It has to be developed in the open, by the community, together with the independent hardware vendors. This is the only right way to build something that will truly make a difference in the long run.

I won't try to convince you about what is currently and will be possible with local AI. We will just continue to build as usual. I am confident that after the smoke clears and we look objectively at what we have built together, the benefits will be obvious to everyone.

Big shoutout to all llama.cpp maintainers. I feel extremely lucky to be able to work together with so many talented contributors. Every day I learn something new and I feel there is so much more cool stuff that we are going to build. Also, I am really thankful that the project continues to have reliable partners to support it!

Cheers!

[Nick Frosst @nickfrosst]

@cohere transcribe

Sota open source transcription model running in the browser :)

Weights on @huggingface link below

[waterloo intern @waterloo_intern]

https://t.co/El9KziCw96

[Claude @claudeai]

You can now enable Claude to use your computer to complete tasks.

It opens your apps, navigates your browser, fills in spreadsheets—anything you'd do sitting at your desk.

Research preview in Claude Cowork and Claude Code, macOS only.

[Unsloth AI @UnslothAI]

Introducing Unsloth Studio ✨
A new open-source web UI to train and run LLMs.

• Run models locally on Mac, Windows, Linux
• Train 500+ models 2x faster with 70% less VRAM
• Supports GGUF, vision, audio, embedding models
• Auto-create datasets from PDF, CSV, DOCX
• Self-healing tool calling and code execution
• Compare models side by side + export to GGUF
GitHub: https://t.co/2kXqhhvLsb
Blog and Guide: https://t.co/ENuTWal5AA

Available now on Hugging Face, NVIDIA, Docker and Colab.

[Stripe @stripe]

https://t.co/m1LvWf3PBz

[Just a Dude Who Invests @DudeWhoInvests]

This lecture by Jensen Huang Nvidia $NVDA CEO was an hour long, and it teaches you more about business than any college could…

[Connor Waslo @ConnorWaslo]

we cut half of our product and it's so much better now. huge glow up and the makeover is just getting started.

[The Register @TheRegister]

Anthropic's Claude Opus 4.6 spends $20K trying to write a C compiler https://t.co/FeGV5nhP2t

[antirez @antirez]

Yesterday @MistralAI released an open weights transcription model able to work in real time, Voxtral Mini 4B. Today, following the Whisper.cpp lesson, here is a C inference pipeline ready to use as a library, I hope you'll enjoy it:

https://t.co/O7pS228DME

[Lukas Ziegler @lukas_m_ziegler]

End-to-end neural networks racing drones in Abu Dhabi! 🚁

Check out the drone racing team from Delft University of Technology!

A completely end-to-end neural network solution, from pixels to direct motor commands.

No Kalman filters. No computer vision feature detectors. Just neurons flying the drone.

The challenge is extreme. These drones fly at high speeds and need split-second decisions with minimal onboard resources: a single rolling-shutter camera and an IMU.

Their approach is called SkyDreamer, based on the Dreamer-v3 reinforcement learning algorithm.

First, a world model is trained in simulation. Then, the neural network learns how to fly in its dreams through reinforcement learning. The network's internal state can be read out to see where it thinks it is on the track or how fast it's going.

Even better, the drone estimates some of its own body characteristics during flight, like the camera angle relative to the body, eliminating time-consuming manual calibration.

The system uses only a single camera and the gyros from the IMU, ignoring the accelerometers, just like human FPV pilots do.

~~

♻️ Join the weekly robotics newsletter, and never miss any news → https://t.co/GoA3ZuwoPB