bryanwhl

i obsess over cool problems

building production serving @ microsoft ai

[Tech with Mak @techNmak]

Inference Engineering is becoming one of the most important skill sets in AI, and I think a lot of engineers are still underestimating it.

Training gets most of the attention because that is where the model is created. But once a model has to serve real traffic, a completely different class of problems appears. You are suddenly dealing with queueing, batching, KV-cache pressure, GPU memory, scheduling, routing, kernel efficiency, tail latency, and the uncomfortable fact that two requests hitting the same model can have completely different cost profiles.

The first thing worth understanding is that inference itself is not one homogeneous workload. Prefill and decode behave very differently. Prefill processes the prompt and builds the KV cache, with work across prompt positions that can be parallelized efficiently. Decode is autoregressive, each sequence advances token by token while repeatedly reading model weights and accumulated KV state. That is why prefill is often compute-heavy, while decode at practical serving batch sizes can become strongly constrained by memory bandwidth.

Once you understand that split, a lot of modern inference work stops looking like a random collection of tricks. FlashAttention reduces memory traffic inside attention. PagedAttention improves how KV-cache memory is allocated and managed. GQA and MQA reduce the amount of KV state stored per token. Continuous batching lets the scheduler keep mixing active sequences instead of waiting for a fixed batch to finish. Chunked prefill prevents a very long prompt from monopolizing execution. Prefix caching avoids recomputing context the system has already processed.

Scale makes the problem harder. Tensor parallelism reduces the per-device model memory footprint at the cost of communication. Pipeline parallelism introduces scheduling complexity and pipeline bubbles. MoE serving adds expert routing, communication, and load imbalance. Some systems even separate prefill and decode onto different workers because they have sufficiently different resource profiles that optimizing them together can become difficult.

This is also why tokens per second is a dangerously incomplete metric. A production system has to care about TTFT, inter-token latency, throughput, queueing delay, KV-cache utilization, batch composition, GPU utilization, p95/p99 latency, and goodput, meaning how much useful request throughput actually meets the latency objective. A server can show excellent aggregate throughput while individual users are still waiting far too long for their first token.

The interesting part is that inference engineering sits at the intersection of ML systems, distributed systems, compilers, GPU architecture, networking, scheduling, and performance engineering. As models get larger and serving volumes grow, improving the model is only half the problem. Making every GPU deliver more useful work becomes its own engineering discipline.

The model decides what token should come next. Inference engineering determines how much serving that token costs, how quickly it reaches the user, and how many users your hardware can support at once.

That is why I think inference engineering is going to matter far more over the next few years than most people currently realize.

[John Ternus @johnternus]

hello

[bryanwhl @bryanwhl1]

new paradigm of engineering is the constant decision paralysis of whether i should:
1. prompt the agent directly
2. do workflows or loop engineering
3. write simple skills for future use
4. build local app with UI for ease of digestion when i need to be in the loop

[Tim Cook @tim_cook]

Today’s agenda at Apple Park: Introduce the @Pokemon team to John, talk gaming, and politely ask Pikachu to stay out of the pond. Two out of three were accomplished.

[Andrew Ambrosino @ajambrosino]

the next unlock is inference-native context liquidity. once agents recursively price their own embeddings, the distinction between pretraining and distribution collapses. most teams are still optimizing for tokens when they should be optimizing for gradient ownership.

[bryanwhl @bryanwhl1]

Excited to be part of this launch with an incredible team!

[Mustafa Suleyman @mustafasuleyman]

Our first reasoning model, MAI-Thinking-1, is built from scratch. Now available in Microsoft Foundry. Kudos to the team! More below.

[Brandon Ong @bdono_]

Today we're launching Hebbian Robotics (YC S26) @hbr_pbc .

We build APIs for evaluating data quality, without training a robotics model.

Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure.

High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale.

We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality:

- Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds.

- Changed your collection SOPs? Monitor for quality drift and regression.

- Suspect duplicate samples? Verify with a single API call.

Before Hebbian, I trained AI models for industrial cleaning robots, and @kstonekuan spent years building large-scale infrastructure at Jane Street.

[Khai Loong Aw @khai_loong_aw]

Our Stanford NeuroAI Lab is presenting at the Cognitive Computational Neuroscience conference #CCN2026 🥳 Come say hi to us!!

@cogphilosopher @Norcal_Neuro @dyamins @KJedoui Yash @seojin_lee @ClionaODoherty @khai_loong_aw

[Andrej Karpathy @karpathy]

We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for three js render of it. Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story. It's kind of janky but fun. But it's a bit mindboggling that the LLM has to place and orchestrate various polygon assets in (x,y,z) coordinates and write code that animates it all, and that it even does anything at all.

I also like this kind of examples because no one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world, so it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free". There might be a lot more. But I'm excited about creating hyper custom worlds that you can imagine dropping players into, e.g. here to participate in the LoTR story as a spectator NPC, or one of the characters, or etc. Something like an ephemeral GTA of X on demand.

Last thought is that the domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them. Here, Opus 5 had to very slowly and painstakingly take screenshots at different points, and it messed up a few times and created a bunch of jank. An example of raw capability (multimodal, gameplay) that I think is still quite lacking.

[Khai Loong Aw @khai_loong_aw]

Today's best AI needs orders of magnitude more data than a human child to achieve visual competence.

We introduce the Zero-shot World Model (ZWM), an approach that substantially narrows this gap. Even when trained on the first-person experience of a single child, BabyZWM matches state-of-the-art models on diverse visual-cognitive tasks – with no task-specific training, i.e., zero-shot. 🧵

[Brandon Ong @bdono_]

Five major supply chain attacks in two weeks, including LiteLLM and axios. We install most of these packages without thinking twice.

@kstonekuan and I built YOINK, an AI agent that removes complex dependencies you only use for a handful of functions, by reimplementing only what you need, so you don't need to worry about supply chain attacks anymore.

@karpathy recently called for re-evaluating the belief that "dependencies are good". OpenAI's @_lopopolo echoed this in his harness engineering article: agents reason better from reimplemented functionality they have full visibility into, over opaque third-party libraries.

YOINK makes this capability accessible to anyone.

[🇨🇳XuZhenqing徐祯卿 @XueJia24682]

✨🇨🇳A Chinese company, Unipath, has launched a household robot that is now in real-home use. It can wake users up on time, operate home appliances, organize storage spaces, and even cook meals automatically.

[Chu You Chia @churroschia]

SF, we are so back

[ruban @rubanlah]

Got 2nd place in the @GoogleDeepMind SG hackathon!

I built a webapp that converts 2d topo maps into 3d colourful renders + lets you fly through and ask gemini live api what you're looking at.

[bryanwhl @bryanwhl1]

get it using @vercel https://t.co/vAfcoG7hQR. one command to install:

npx skills add bryanwhl/ffmpeg-video-editor

[bryanwhl @bryanwhl1]

built an ffmpeg skill for claude code with @kstonekuan and used it to make this switzerland reel from my travel backlog with just 1 prompt:

- blur letterbox
- 2x timelapse
- xfade transitions
- color grade

it then auto-generated a side-by-side comparison showing raw and final

[bryanwhl @bryanwhl1]

https://t.co/o1LalujluX

[Jacob Andreou @jacobandreou]

I’ve never written more code than I have over the last 2 months alongside our insanely talented team to deliver the first ever frontier experience from MAI.

Today we’re announcing Copilot Tasks, a to-do list that does itself. Tasks gets better the more you use it. It has its own computer, browser, and a ton of tools/MCPs. As you connect with more services and Tasks learns your preferences, it takes the bs off your plate.

Just last week, I asked it to “book me a car from the office to catch my flight.” That was it. Behind the scenes, it checked my calendar, looked at my flight details, noted when my last meeting ended, factored in travel time to the airport, and made sure I had plenty of buffer to get through security and to my gate.
Within minutes, a car was booked at the right time, no calendar math, no stress.

All the thinking I normally do around meetings, flights, and timing has disappeared. It's not just about getting tasks done it’s about getting your time back.

This week, Tasks enters Research Preview for some users to test it out. If you want to check it out early, join the waitlist for our pre-launch:

[Andrej Karpathy @karpathy]

Very interested in what the coming era of highly bespoke software might look like.

Example from this morning - I've become a bit loosy goosy with my cardio recently so I decided to do a more srs, regimented experiment to try to lower my Resting Heart Rate from 50 -> 45, over experiment duration of 8 weeks. The primary way to do this is to aspire to a certain sum total minute goals in Zone 2 cardio and 1 HIIT/week.

1 hour later I vibe coded this super custom dashboard for this very specific experiment that shows me how I'm tracking. Claude had to reverse engineer the Woodway treadmill cloud API to pull raw data, process, filter, debug it and create a web UI frontend to track the experiment. It wasn't a fully smooth experience and I had to notice and ask to fix bugs e.g. it screwed up metric vs. imperial system units and it screwed up on the calendar matching up days to dates etc.

But I still feel like the overall direction is clear:
1) There will never be (and shouldn't be) a specific app on the app store for this kind of thing. I shouldn't have to look for, download and use some kind of a "Cardio experiment tracker", when this thing is ~300 lines of code that an LLM agent will give you in seconds. The idea of an "app store" of a long tail of discrete set of apps you choose from feels somehow wrong and outdated when LLM agents can improvise the app on the spot and just for you.
2) Second, the industry has to reconfigure into a set of services of sensors and actuators with agent native ergonomics. My Woodway treadmill is a sensor - it turns physical state into digital knowledge. It shouldn't maintain some human-readable frontend and my LLM agent shouldn't have to reverse engineer it, it should be an API/CLI easily usable by my agent. I'm a little bit disappointed (and my timelines are correspondingly slower) with how slowly this progression is happening in the industry overall. 99% of products/services still don't have an AI-native CLI yet. 99% of products/services maintain .html/.css docs like I won't immediately look for how to copy paste the whole thing to my agent to get something done. They give you a list of instructions on a webpage to open this or that url and click here or there to do a thing. In 2026. What am I a computer? You do it. Or have my agent do it.

So anyway today I am impressed that this random thing took 1 hour (it would have been ~10 hours 2 years ago). But what excites me more is thinking through how this really should have been 1 minute tops. What has to be in place so that it would be 1 minute? So that I could simply say "Hi can you help me track my cardio over the next 8 weeks", and after a very brief Q&A the app would be up. The AI would already have a lot personal context, it would gather the extra needed data, it would reference and search related skill libraries, and maintain all my little apps/automations.

TLDR the "app store" of a set of discrete apps that you choose from is an increasingly outdated concept all by itself. The future are services of AI-native sensors & actuators orchestrated via LLM glue into highly custom, ephemeral apps. It's just not here yet.

[Google Gemini @GeminiApp]

Introducing Lyria 3, our new music generation model in Gemini that lets you turn any idea, photo, or video into a high-fidelity track with custom lyrics.

From funny jingles to lo-fi beats, you can create custom 30-second soundtracks for any moment.

See how it works. 🧵