Richard Wei

mts on on-device AI | previously engineering leader @ apple

[Pratik Gandhi @pratikg]

FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders

Sharing highlights from this week's session hosted by @shshnkp (Apple AI/ML Evangelism Team) with @mgorbach and engineers from Foundation Models, Core AI, Siri and App Intents teams at Apple.

1) Apple’s app framework can use your open-weight model as its LLM. Apple’s new app framework can use Apple’s on-device model, Apple’s cloud model or a Core AI converted model, which is interchangeable with a one liner! And your model gets access to the same tool calling, structured output, profiling with Instruments, etc. as Apple’s models.

2) Don’t use Apple’s on-device model for coding, even according to Apple engineers! Use Core AI to bring in a model for coding. One engineer uses a setup where he can use MLX and Core AI models in different “coding harnesses” and reports good experiences.

3) Free local Chat Completions endpoint on any Apple silicon Mac with macOS 27! fm serve provides an OpenAI style Chat Completions endpoint backed by Apple’s on-device model on any Mac. Nothing to install! (shared by Shashank the day after the lab)

4) Apple’s new on-device model is 20B, but only has 8K context. AFM Core Advanced (20B) will be available on the latest iPhones, iPads and M3+ Macs. Can’t opt out of Apple Intelligence in 27, and the model will always be present. Tip: use typed units for fields and let constrained decoding handle it (instead of wasting tokens explaining in prompt).

5) Agentic Siri only works when online. Tool calls from Siri to apps are done using Private Cloud Compute. When offline, Siri will use the on-device model but won’t be able to call into apps.

6) Lessons Apple engineers follow when building agents apply to any agent:

- write down instructions to the model about how/when to use your tools in a markdown file (“skills” are the same idea as agent skills everywhere else!)
- skills are just context to the model, meaning no ordering guarantees. If something should always happen in an order, it should probably be a tool
- make tools accept lists of items to operate on to avoid agent having to call it in a loop
- unit test your tools (deterministic) before testing the whole agent (non-deterministic)

Curated Q&A relevant to Local AI builders in 🧵

[Richard Wei @rxwei]

@migueldeicaza Thank you Miguel!

[Richard Wei @rxwei]

A few weeks ago, I wrapped up 7 great years at Apple. I've since joined a startup!

Looking back, I’m really proud of the core developer frameworks and SDKs we built, and of helping bring free, fast on-device AI models, guided generation, and Private Cloud Compute to app developers. To me, great API design is an art form. It takes taste, empathy for developers, and a relentless pursuit of simplicity.

I’m also grateful for the opportunity to help shape lots of WWDC videos, speak twice at the WWDC Platforms State of the Union, and build multiple teams from scratch. Above all, these experiences reinforced my belief in what small, talented teams can accomplish. That feels especially true today.

Thank you to everyone who made this chapter so special. I’m staying in SF and can’t wait to share what’s next.

[Richard Wei @rxwei]

@bunchjesse Thank you Jesse!

[Shashank Prasanna @shshnkp]

At yesterday's Apple Intelligence Group Lab livestream, many of you had questions about Apple Foundation Models context size. We recently published a doc Article with more details.
https://t.co/bziCxR2Xh7

You can also see it on the terminal using the fm cli.

Also, fun fact: You can start a Chat Completions API server using the on-device model using fm serve

[Richard Wei @rxwei]

@LouisDhauwe ex engineering channeling ex product marketing @joshelman

[Richard Wei @rxwei]

@LouisDhauwe CoreAI -> Core AI
FoundationModels -> Foundation Models
😉

[Josh Elman @joshelman]

This is huge news for Siri AI (and one of the last things I was working on when I stepped away)

Any app can now offer dynamic skills from their app to Siri so users can take broader actions across their apps

(App Intents instead of MCP)

[Sam Lazarus @slazarus]

We're laser-focused on the Apple platforms development experience for really serious builders. Some examples:

- Providing world class simulator use tools for your agent
- A system that allows agents to work with Xcodebuild in a way that gets out of your way (priority queueing builds based on whether you're looking at them / the agent is doing them in the background, etc. More to come here soon as well, I have something really special cooking 🧑‍🍳)
- Support for sims even apple doesn't provide (CarPlay sim!)
- A great out-of-the-box suite of skills + tools that help with Apple platforms development
- Best-in-class support for letting your agent use App Store Connect so you never need to worry about things like taking screenshots, uploading builds, etc.

There are many great orchestrators out there, but there are a lot of unique things about development for Apple's platforms that a standard Agent + Orchestrator setup doesn't equip your AI well to handle. That's what Bitrig is here for!

[Jesse Bunch @bunchjesse]

Huge day for Creator Studio today.

Our new in-app assistants can help you get things done in record time. MCP allows you to bring your own agent if you want. And an open source format for Keynote (and Python library!) so you can create Keynote documents programmatically on any platform.

Super proud of the team. Hope you love it!

[Alex Choi @AlexChhk]

wtf i just discovered that a macbook comes with a free local LLM in your terminal

[Shashank Prasanna @shshnkp]

Today we also published benchmarks and a deep dive into LLM inference on M5 Ultra Mac Studio, covering MLX, GPU Neural Accelerators, tensor and pipeline parallelism, and Thunderbolt 5 RDMA.

https://t.co/IisNwOc6Ml

[Peter Friese @peterfriese]

You can now use @OpenAIDevs ‘s Decision model via Apple’s Foundation Model framework!

One API to rule them all 💍

[Peter Friese @peterfriese]

Decision models like Jev are all the rage - they are super fast, dirt cheap, and - best of all - deterministic.

What if... you could use Jev (and other System One models) using Apple Foundation Models framework?

Meet System One models in Swift:
https://t.co/g68Ew91gTs

[Richard Wei @rxwei]

The LanguageModel implementation would throw a runtime error when a request was made with an unsupported schema.

This is already the case for some LanguageModel instances. For example, some models do not support generating top-level enums or arrays and would throw an error on those. The benefit of sharing common APIs seems bigger than strict type safety!

[Richard Wei @rxwei]

Jev is a fantastic example of co-designing SDKs and models to unlock a new paradigm!

When we were designing the Swift API for Apple’s Foundation Models framework two years ago, we wanted to make type-safe structured responses the standard for apps on our platform. That is why we created the `@​Generable` API, making constrained decoding approachable to app developers. This effectively eliminated structural hallucinations in both tool call arguments and responses, which were especially important for a small on-device model. https://t.co/TnDSoCImNw

It would be fascinating to see decision models conform to the `LanguageModel` protocol and work with `@​Generable` natively! The best part is that all the APIs are already there: Bools are "noul", enums represent choices, and `@​Guide` descriptions are instructions. A Swift model implementation simply needs to handle a schema conversation, and output probabilities can be stored directly in the response metadata. https://t.co/q5PATIZ0bs

[Richard Wei @rxwei]

@rudrank @peterfriese The best new API is the existing API!

[Richard Wei @rxwei]

@kyle_fowler2 @peterfriese @typesafeai @SwiftLang Not if the service provider adopted App Attest! For example: https://t.co/o31QInu5B8

Both Google and Anthropic's integrations with Foundation Models support App Attest. We definitely hope to see more!

[Richard Wei @rxwei]

@fayzez_com https://t.co/JAG84xvQAU

[Richard Wei @rxwei]

It supports both the on-device model and Private Cloud Compute! We built it to enable effortless automations for Mac users. It's free, zero setup, no privacy compromises.

[Taha Tesser @TahaTesser]

TIL macOS 27 now comes with new fm cli 🤯

We can now use Apple's Foundation Models (on device) from CLI!

[Richard Wei @rxwei]

@sebuzdugan There is no routing! Users pick the model

[Richard Wei @rxwei]

💯

[Peter Friese @peterfriese]

Jev is typesafe, so it's only natural to map Swift types directly to Jev decision primitives: