People who don't read the code have no idea how much complexity agents dump into their codebase.
I still keep agents on a short leash, but sometimes I give them a broader task and more freedom - eg, with personal dev tooling. That's when they go completely nuts with defensive code and remind me just how far into the stratosphere a complexity can get. They are literally heroin addicts without supervision.
I've iterated on my "don't overcomplicate shit" rule a lot. Maybe it's me, but I still can't make them reliably stop doing this.
My guess is that labs would rather bias toward defensive code than risk missing an edge case. Balancing that against complexity is still a human job.
Especially compared by per-task pricing, which is the metric that should matter, I don't think there is anything competitive anywhere in the market.
We want people to be able to use tons of AI; it is important to being able to explore this new renaissance in front of us.
@nstlopez@ScriptedAlchemy Like multiple versions of material ui or react within one app. And needing to be on the same version across all MFEs. Or internal corporate layers on top of material ui needing to be on the same version
@yacineMTB I usually ask AI to help me understand. I haven’t tried not understanding in a while. I always wonder if it’s actually still necessary. At some point I wonder if it won’t be needed to understand
JEFF BEZOS: “I know there's a lot of concern that many smart people have, that AI is going to make humans redundant. I totally disagree with this point of view. In fact, AI is going to create a labor shortage … We have an endless set of things to invent.”
Today we rolled out Astra to every engineer at Databricks (N=~3500). Some notes that may be helpful to others:
1. Astra unambiguously out performs our previous highest-end models (Opus 5, Sol 5.6) on highly complex tasks, especially those related to high level system design or long range horizontal tasks.
2. Engineers given Astra increased overall coding spend by around 60% compared to baseline.
3. It is not clear Astra meaningfully improves on medium/low complexity coding tasks compared to earlier models. We suspect those tasks are mostly saturated (i.e. perfectly executed) by existing models.
4. We learned above by piloting Astra with around 200 users to gain signal on both quality and cost. We use Unity Gateway to do cohort-based experiments for all new models.
5. We give engineers a sub-budget specific to Astra to encourage them to use Astra selectively on complex tasks while preferring lower cost models for everyday tasks. Our engineers are able to mix-and-match tools and models within their overall budget envelope (we also allow for increased budgets through various mechanisms). These budgets are defined in Unity Gateway and regularly revisited.
Note: We do not have robust comparisons of Astra-vs-Fable because we have net yet rolled out Fable widely due to data retention policies.