Asmit

Per aspera, ad astra
LLM Inference for Siri @Apple | CS @GeorgiaTech

[Asmit @asmitks]

AGI will be the ultimate monument to human pride, and simultaneously its ultimate undoing

[Deedy @deedydas]

99.99% of people cannot comprehend how insane FrontierMath is.

The problems are crafted by Math profs and not in any training data.

Math legend Terry Tao said "These are extremely challenging. I think they will resist AIs for several years at least."

OpenAI o3 did 25% on THIS.

[Asmit @asmitks]

@Aartiutwani

[Asmit @asmitks]

@Hunter_Weiss Manish on the beat, shabang

[Asmit @asmitks]

Presented our work from  Apple at EuroMLSys 2026 in Edinburgh:

“Asynchronous Verified Semantic Caching for Tiered LLM Architectures”

The core idea:
semantic caches are usually conservative because false positives are catastrophic. Instead of pushing more verification into the serving path, we moved verification off the critical path entirely. Near miss cache interactions asynchronously trigger an LLM judge verifier. Verified pairs are promoted into the dynamic tier over time, increasing effective cache coverage while preserving the latency behavior of the original system.
Interesting systems tension:
you want higher cache hit quality without paying synchronous verification costs on user requests.

Paper:
https://t.co/zZhwftl5pR

#MLSys #LLM #Inference #Caching #EuroSys

[Amrit Khera @Amritkhera10]

Incredibly excited to release GPT-5.1 Pro to the world! It’s great at tackling the hardest, messiest problems with clearer, more comprehensive responses.
I’m eager to see you throw your toughest problems at it. Please try it and share what you think :)

[Asmit @asmitks]

@sporadica still got a grazer in me

[Asmit @asmitks]

@rcx86 could have dropped the "unit", to make it more dense.

[Asmit @asmitks]

@rohanpaul_ai It gets dicey when the sources cited by the likes of perplexity/AI mode/searchGPT are AI generated themselves. As long as it's only used for paraphrasing/ readability it should be fine.

[Asmit @asmitks]

@raphaelschaad You can ask Siri iPhone questions.

[Asmit @asmitks]

@yacineMTB Is ur username really कछे 🩲

[Asmit @asmitks]

@tszzl This is why humans have conquered the seas, crossed the lands. Couldn't just sit and enjoy some bread.

[Asmit @asmitks]

@hovinthenorth

[Asmit @asmitks]

@khoomeik @tszzl It would check out tho. When there are no adventures, no wishes, nothing to conquer for the sapiens. A perfectly created AI slop machine will quench the brain's thirst to be unsettled. Stimulations to move our brains around, but not move us anywhere.

[Asmit @asmitks]

@sporadica Even after having this prior, there is an uncontrollable primal urge to yeet patience and have a fleeting shot at reaching the exit cave first. Definitely some evolutionary slag.

[Asmit @asmitks]

@tszzl Our fractal minds have a tendency to see self similar orders. The similarity in the abstraction of a man, and a state is an exhibit.

[Asmit @asmitks]

@_nonfigurativ_ @sanity_io Its giving Pantheon

[Amey Agrawal @agrawalamey12]

The bitter lesson of AI infra: The hardest part about building faster LLM inference systems is not designing the systems, but rather it is evaluating if the system is actually faster! 🤔

This graph from a recent top systems venue paper about long-context serving shows average normalized input token latency for a trace with both short and 100K+ token requests. System X looks like a clear win: lower normalized latency and higher request rates. But normalized metrics can obscure the actual user experience: at those rates, long inputs see >2hr delays to the first token!

Let’s do the math!🧮

[Amey Agrawal @agrawalamey12]

Super long-context models with context window spanning millions of tokens are becoming commonplace (@GoogleDeepMind Gemini, @xai Grok 3, @Alibaba_Qwen Qwen2.5). But efficiently serving these models is tough, especially alongside short requests. Head-of-Line (HOL) blocking becomes a major issue, hurting latency for everyone.

We present Medha, a system designed to handle this mix efficiently. Achieving 30x lower latency, and 5x higher throughput compared to the state-of-the-art. Full paper: https://t.co/PQlwLtlnD5. 🧵

[Asmit @asmitks]

This Apple Intelligence feature hits the spot for inquisitive but impatient types like me. Information snacking.

[Asmit @asmitks]

@emilymbender @AravSrinivas And as far as, incorrect summarisation is concerned, most blogs and pages are second/third order pieces of information which themselves are an attempted summarisation of the facts. LLMs are superior in playing with words as of now.

[Asmit @asmitks]

@emilymbender Most of the “LLM-driven” searchbots are RAG based, aligned to only emit lines that can be cited from external sources. I think @AravSrinivas mentioned that you can imagine this search response to be like writing an academic paper, where each line refers to a citation or itself.