Sampling bias in expert data gives AI jagged capability for knowledge work. The peaks are where the signal is dense (highest availability of data on professional work) and the dips show gaps in judgment and reasoning.
More of my thoughts here 👇
Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
had a similar story with lost baggage on a Royal Air Maroc flight … only in this case Instinct needed me to verify each email response back from the airline with a pdf attachment about lost inventory.
hopefully personal agents learn more agency to complete those workflows without a user in the loop!
Instinct just made me $700.
I was frustrated because my flight back to US from Europe was delayed 6hrs and texted Instinct asking what I could do. It told me there was a law that said I was owed €600. A month later, after many esoteric email exchanges, I got paid out!
A rare product experience where you actually get infinitely more value than what you paid ($0).
Play-by-play.
- My flight back to the US from Europe on Oct 4 was delayed 6hrs
- EU regulation 261/2004 says if a long haul flight is delayed >3hrs you are owed €600
- I discovered this because I texted Instinct, frustrated, saying "what can I do about this delay"
- It sent an email for me, and chatted with a Delta rep to verify this would fall under those rules.
- It patiently monitored by inbox. They said most cases receive a response in 21 days. On day 21, Oct 26, Delta approved the claim!
- .. but you still couldn't receive it in cash.
- On Nov 3, I finally get the cash claim email
Own your intelligence.
Anyone that opposes this has something to sell you (perhaps your own intelligence back to you.)
Local models need to continue proliferating. The “absolute unit” massive models may be out of reach for now locally, so your best option is to, indeed, keep your options open. Own your harness, data, connections, runtime: your agents. Embrace open source.
specific intelligence or sovereign ai is crucial for businesses looking to preserve their advantages and customer loyalty (often built over years through personalized products and experiences)
worked on this at Capital One and can see other F500s following suit
Why the companies of the future will have specialized models trained on their own data
“You are creating value for the model company versus capturing that value for yourself.
We need to have more of that surplus that gets created by the use of intelligence remain at the firm.
Every firm will have a private eval benchmark, which is their core IP, and every model comes in and tries to hill climb on that eval.” @satyanadella
Love to hear your thoughts @hwchase17@sh_reya@jerryjliu0@AnjneyMidha
Agents escaping sandboxes is not a new phenomenon.
We study how models behave over long horizon tasks and train them to be aligned and performant. Our CEO @anandnk24 on the Disruptors 60 panel at @TechCrunch Disrupt with @GreenfieldVC.
Last week, 40 of us at Patronus came together in Lyon, France. The team ran workshops, aligned on what’s next, and took excursions through the city and into the French Alps.
We’re hiring across research, engineering and operations in San Francisco. If you want to help decide what frontier models learn next, come build with us.
@quichenpatel “Customer gets the promised price and vendor bears upside in the cost to get there” - totally agree. This is why long horizon simulations are important to capture the edges and complexity in domain specific workflows
Been thinking about this with @Muse and Instinct out now.
Humans have agency on demand. We can bail on a plan halfway through the second something better shows up.
Agents can't really do that yet. You give them a goal and they just go do it.
I was looking for a couch today, found it on Pinterest, and ended up way more into the lamp and coffee table next to it. Didn't even want the couch anymore.
An agent would've just found me that couch at the best price
All seven Databricks co-founders began our journey together at UC Berkeley. We started the company in a room in Soda Hall, moved to a small office on Addison Street, and spent our first few years in Berkeley before eventually heading to SF. Two of our co-founders are still on the faculty.
Today we announced Databricks Field at California Memorial Stadium, our first collegiate athletics sponsorship.
Berkeley shaped everything about how this company thinks, and this is our way of investing in the next generation of students and builders who'll do their best work on that campus.
Go Bears!
https://t.co/JO3aBx99Z5
Today, we’re releasing SpeedrunBench: the first benchmark that measures how fast agents can beat video games.
We asked frontier models not to simply complete video games, but to speedrun them - across 10 titles like Mario Kart and Pokemon.
On simpler games, agents get close to the human world record. However, on more complicated games like Pokemon Blue, the best models Kimi-K3 and Opus 5 are ~4x off the world record.
Benchmark, paper, and demo below.
Today we're excited to announce FigmaTrace, the first open source computer-use dataset for design.
We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks.
Dataset, paper, and blog below.