Yassin

Senior SWE @ LiteLLM

[Yassin @YassinKortam]

@LiteLLM now supports cache-aware routing for self-hosted models on Kubernetes.

It keeps each agent session on the same GPU pod, so vLLM can reuse its prompt cache.

Read the blog: https://t.co/OWeUm3uSES

[Yassin @YassinKortam]

Microsoft 365 Copilot is now available through LiteLLM AI Gateway.

With LiteLLM and Copilot, it's now possible for your apps to answer questions like "summarize my emails and my last call”

https://t.co/Kl7A1MCJ3d

[Yassin @YassinKortam]

We cut time to first byte through @LiteLLM by 94% for long-prompts.

For a 440,000-token conversation, median latency went from 553 ms to 35 ms.

Read the full post: https://t.co/KUPYlmHk1s