Not long ago, I was still spending enormous effort building scaffolds and retrofitting environments for models. Since then, models have become steadily better at using the same environments people do. Terminals were an early sign; now frontier models are closing the gap between GUI and API. As interaction becomes less of a handicap, the challenge shifts to making agents reliable and efficient enough to diffuse across more domains; and enabling them to learn from experience in the environments where people already work.
Most benchmarks ask whether a model can solve an assignment in an exam-like setting. With ApprenticeBench, we asked something closer to how people evaluate a new employee: can an agent enter a workplace, learn from history and feedback, and grow expertise in the role over time? 🧵
Introducing ApprenticeBench: computer use + continual learning on a real job.
We show Fable 5.1 and GPT-6 Astra can now continually learn on a job and surpass human professionals. A decisive step change in AI's job readiness.
No FDEs. Agents deploy themselves into the job. 🧵
Model context has grown beyond any human can keep in mind. But learning is more than accumulating or recalling memory. Humans grow expertise by connecting and compressing experience into rules and beliefs for later use. We are just seeing signs of this in two frontier models, and at substantial cost.
Meet a cohort of AI apprentices joining an accounting department. Use the same software and receive the same materials and feedback as people onboarding to the job. After seven months of learning and working, do you think they are ready to graduate from their apprenticeship?
Life update: I moved to silicon valley to tackle agents' biggest challenges: plasticity and reliability.
Today's agents are smart but brittle. They lack plasticity (continual learning and adaptation) and reliability (stable, predictable behavior with bounded failures). These two traits define whether agents become critical infrastructure or remain clever demos.
Plastic systems like to change. Reliable systems resist change. Is it even possible to have both of these seemingly conflicting traits? Fortunately, humans are a living example of that. We are constantly learning and adapting while staying remarkably dependable (for the most part, at least). The real question is, how can we achieve the same harmony within a different cognitive substrate?
We've brought together some of the world's best agent experts whose work (Mind2Web, MMMU, LLM-Planner, SeeAct, UGround) helped shape the modern agent field. Now we are taking on the new mission: unlocking plasticity and reliability for every agent.
We are looking for cracked researchers and engineers to join us in person in the bay area! If you strongly resonate with the mission, send your CV and thoughts to: [email protected]
I will be at #neurips2025. Happy to chat over coffee!
📢 New preprint alert! Check out our latest research on 🌟automatic evaluation of attribution by #LLMs, i.e., verifying whether the generated statement is supported by the cited reference. https://t.co/CeuVHia4ij [1/N] #NLP#NLProc
Did you know language models like GPT can also be used to encrypt and protect your data?
I'm excited to finally present SELM, a Symmetric Encryption algorithm with autoregressive Language Models, with @ysu_nlp and @osunlp!
Website: https://t.co/3UBydZb40C