Agents should run in virtual machines, not containers. I've been saying this for a while (see our 2017 research paper: https://t.co/BEIlwCTHne), and the recent wave of CVEs is only making the case stronger.
That's the thesis I went on the @OssStartup Podcast to talk about with @tnachen and @robby_mtf. We got into why the premise of “containers are fast and secure, VMs are secure but slow” is false and just the result of poor engineering. If you build the stack right, you get both: in cloud deployments, you don't have to give up speed to get security.
We also talked about why cold start latency matters: responsiveness is important to ensure agents reply right away, but also if you’re chaining agents or tools they use the delays compound. That’s bad for:
1. User experience
2. Cost, as any CPU cycles you’re burning just starting things up is money down the drain)
And “solving” the problem with warm pools, by leaving groups of idling agents/VMs around so you can be responsive is horribly wasteful, especially at scale…which, if you’re running an agentic service, you either have, or will soon have. Millisecond, statefully-restored scale to zero is a big friend here.
🎙️ Full episode here: https://t.co/5yXPhxR0Jv , hopefully you find interesting, let me know what you think in the comments!
Everyone goes on talking about open source agents but, it increasingly looks like you'd need a mini NVIDIA GB200 rack, DataCenter to run full model locally.
@Marcia_Ong 💯 and to all those wanting a local AI in their machine and realtime, one needs minimum 512GB unified memory and with tcgen support not these DGX sparksssss
my take on this situation currently is that they'll unban it in a few days and the net effect will be increased demand for Fable
however this kind of thing is extremely disruptive and distracting for people inside of the company. the only comparable scenario i can remember is Sam Altman's firing which was resolved relatively quickly. even though things went back to the way they were, i do think that disrupted their momentum for a while
hoping for a good outcome here!