Eric Lin

self-improving agents and evals
agent hillclimbing @ decagon
cs @ harvard

[Decagon @DecagonAI]

We put nine leading models through 1,373 battles to find the best balance of quality, speed, and cost on complex enterprise tasks.

In our latest results, GLM-5.3 broke the Pareto frontier by achieving frontier-level quality with incredible efficiency, finishing tasks in less than a third of the time than Opus 5.

More in the thread. ↓

[Eric Lin @EricLin1618]

Excited to share what I've been working on the past few months!

[Ashwin Sreenivas @AshwinSreenivas]

One of the hardest parts of building self-improving agents is proving they are actually improving.

That’s why, alongside Duet Autopilot, we built DuetBench: the first benchmark designed specifically for CX agents that learn and improve over time.

To evaluate Duet Autopilot, we compared its performance against certified human agent builders and graded both on outcome and methodology across 90 diagnostic investigations from simple metric lookups to root-causing CSAT drops.

We also evaluated Autopilot on enterprise agent-building tasks. Starting from messy design documents, it had to build AOPs and tools from scratch, generate simulations, and pass every associated test before a task was considered complete.

Autopilot demonstrated an iterative approach to agent building. Rather than solving problems in a single pass, it ran simulations, identified broken branches, repaired the AOP or underlying tool, and repeated the process until the workflow passed.

Another notable result was that Autopilot improved the quality of its own test set through self critique, increasing simulation accuracy from 58% to 88% across 520 benchmark runs.

As self-improving systems become more common, verified evaluation will matter just as much as model capability.

Excited to share the research behind it. Full writeup below. ↓

[Daniel Litt @littmath]

Basically any resource on a difficult subject—a colleague, Google, a published paper—will be wrong or incomplete in various ways. Usefulness isn’t only a matter of correctness.

[SkalskiP @skalskip92]

tip 4: few random ones

my favorite: adapt to tasks by formatting data in a proper way not by designing custom architecture.

[Boston Symphony @BostonSymphony]

(1/3) With great sorrow, we announce the death of our beloved Music Director Laureate Seiji Ozawa, who passed away on February 6 at age 88 in Tokyo.

[Eric Lin @EricLin1618]

Excite to finally announce what we've been working on...

[Dynamo AI @dynamo_ai]

DynamoFL is excited to announce Dynamo 8B, our Foundation Model, with leading performance across multilingual benchmarks vs. Llama2-13B, Mistral 7B, Bloom-7B!

Read our #benchmarks https://t.co/Uj0Ns7eb8v

Download Dynamo 8B on #HuggingFace https://t.co/nNWPc6PAEP (1/2)

[Eric Lin @EricLin1618]

@JagersbergKnut stay tuned for updates...

[Eric Lin @EricLin1618]

@MatternJustus 👏

[Eric Lin @EricLin1618]

@SeedLabsHQ ♢

[Edge Impulse @EdgeImpulse]

TinyFedTL is the first implementation of federated transfer learning on a resource-constrained microcontroller: https://t.co/GnT7KQwsNM

cc: @KavyaKopparapu @EricLin1618

[Magnus Carlsen @MagnusCarlsen]

🖐

[Catherine Yeo @catherinehyeo]

Just helped a friend in New Jersey book an appt for her COVID vaccine using a resource found on our site!

She's been waiting for *weeks* for an open slot at her local Walgreens. Using https://t.co/s2yYQ5cfdd took 2 mins to find a slot. Book an appt today & get your COVID shot ⬇️

[Catherine Yeo @catherinehyeo]

Today is April 19, which means all individuals 16+ are eligible to get the COVID vaccine in the US!

Make a vaccine appointment today by finding one in your state at https://t.co/s2yYQ5cfdd

[Catherine Yeo @catherinehyeo]

Thank you @VickyNguyenTV @MichelleTak_ for including https://t.co/s2yYQ5cfdd as a useful vaccination appointment resource on today's @TODAYshow segment!

As more resources emerge, it's key to be aware of your data privacy & which sites are helpful vs malicious. See more here:

[Harvard SEAS @hseas]

Students @catherinehyeo and @EricLin1618 launched a website to provide a one-stop-shop for the most up-to-date COVID-19 vaccination information from each U.S. state: https://t.co/blH157grDT

[Catherine Yeo @catherinehyeo]

Thank you @hseas for sharing our story about https://t.co/s2yYQ5cfdd! @EricLin1618

https://t.co/fmNNL5e3db

[Eric Lin @EricLin1618]

@catherinehyeo @Go_PixelHacks @erinjerri @jritten inspiring to see the journey of PixelHacks and how you all empower so many others!

[Catherine Yeo @catherinehyeo]

My grandma's COVID vaccine appt got cancelled today. My family panicked to find another.

I used https://t.co/s2yYQ5cfdd to find a resource to book her a new appt. @EricLin1618 & I built this site to share resources per state to help such situations.

Please check it out & share!

[Eric Lin @EricLin1618]

Ever wondered when you’ll be eligible for a COVID vaccine? 🤔@catherinehyeo & I created the COVID Vaccines Info Guide, your one-stop source for trusted info on getting COVID vaccines. Check it out, share it with your friends, & let us know what you think: https://t.co/C759qDAfpJ

[Eric Lin @EricLin1618]

@catherinehyeo Excited to launch this together and demystify such an important step in helping communities stay safe!