Need to customize vLLM? Don't fork it. 🔌
vLLM's plugin system lets you inject surgical modifications without maintaining a fork or monkey-patching entire modules. Blog by Dhruvil Bhatt from AWS SageMaker 👇
Why plugins > forks:
• vLLM releases every 2 weeks with 100s of PRs merged
• Forks require constant rebasing & conflict resolution
• Monkey patches break on every vLLM upgrade
How it works:
• Use VLLMPatch[TargetClass] for precise, class-level mods
• Register via vllm.general_plugins entry point
• Control patches with env vars (VLLM_CUSTOM_PATCHES)
• Version-guard with min_vllm_version decorator
Example: Add priority scheduling to vLLM's scheduler in ~20 lines. One Docker image serves multiple models with different patches enabled via environment variables.
The plugin loads in ALL vLLM processes (main, workers, GPU/CPU) before any inference starts—ensuring consistent behavior across distributed setups.
Read the full implementation guide with code examples:
https://t.co/StIh3oTNeE
RIP fine-tuning ☠️
This new Stanford paper just killed it.
It’s called 'Agentic Context Engineering (ACE)' and it proves you can make models smarter without touching a single weight.
Instead of retraining, ACE evolves the context itself.
The model writes, reflects, and edits its own prompt over and over until it becomes a self-improving system.
Think of it like the model keeping a growing notebook of what works.
Each failure becomes a strategy. Each success becomes a rule.
The results are absurd:
+10.6% better than GPT-4–powered agents on AppWorld.
+8.6% on finance reasoning.
86.9% lower cost and latency.
No labels. Just feedback.
Everyone’s been obsessed with “short, clean” prompts.
ACE flips that. It builds long, detailed evolving playbooks that never forget. And it works because LLMs don’t want simplicity, they want *context density.
If this scales, the next generation of AI won’t be “fine-tuned.”
It’ll be self-tuned.
We’re entering the era of living prompts.