Joshua Qin

researching @conceptdotdev | @harvard, prev jane street, sig

[Joshua Qin @joshzqin]

Excited to share some recent work!

Being good at individual tasks doesn’t automatically make an agent a good company agent. It needs deep understanding of its company: people’s priorities, judgment, and working relationships. We built Company Agent Bench to start measuring this.

[Phil Chen @philhchen]

Today we’re releasing Company Agent Bench, a benchmark for company assistant products.

We compare Dots, Grok Bot, Claude Tag, and Devin on how well an agent understands its company.

We also found that API models with a simple harness perform better than these products (1/6)🧵

[Joshua Qin @joshzqin]

@siddsax @philhchen Totally agree, new eval idea :)

[Joshua Qin @joshzqin]

New result: @AnthropicAI now leads among both agent products and API models on Company Agent Bench.

We now include @claudeai Tag with Fable 5.2. It scores 77.9%, the highest among agent products we’ve tested, ahead of @OpenAI Dots (75.9%) and @Grok Bot (72.7%).

Interestingly, Tag with Fable 5.2 still scores below using the same Fable 5.2 model with a basic Slack harness across low, medium, and high reasoning effort.

Read more below, and further results to come soon!

[Phil Chen @philhchen]

Today we’re releasing Company Agent Bench, a benchmark for company assistant products.

We compare Dots, Grok Bot, Claude Tag, and Devin on how well an agent understands its company.

We also found that API models with a simple harness perform better than these products (1/6)🧵

[Joshua Qin @joshzqin]

@DanielLenton1 @tkkong @OpenAI @AnthropicAI We took a small sample and scored ~60% :)

[Joshua Qin @joshzqin]

@mattlam_ @philhchen appreciate it, we're excited too!

[Joshua Qin @joshzqin]

We save progress after each step, including which Slack messages have been replayed, the pending question, and any answers received. We also run periodic audits to verify that the replayed history is faithful and agents only have access to context available before each question’s cutoff!

[Joshua Qin @joshzqin]

@AIQuanting @philhchen Each product lives in their Slack sandbox and has access to the entire Slack history up to the question's cutoff, so up to the full month! The products themselves choose what relevant Slack context to retrieve and use in their answers.

[Joshua Qin @joshzqin]

@philhchen 🙌🙌 grateful to work on this together!

[TK Kong @tkkong]

Today, we’re launching https://t.co/WuxyRB4yn0

@OpenAI Dots placed #1 across agent products while @AnthropicAI Fable 5.2 placed #1 overall via API.

Our goal is to measure how well agents understand a specific company.

Check out the results and findings below!

Shoutout @joshzqin for spearheading this project

[Phil Chen @philhchen]

Today we’re releasing Company Agent Bench, a benchmark for company assistant products.

We compare Dots, Grok Bot, Claude Tag, and Devin on how well an agent understands its company.

We also found that API models with a simple harness perform better than these products (1/6)🧵

[Joshua Qin @joshzqin]

@OpenAI @grok See the full writeup here, curious to hear everyone’s thoughts!

https://t.co/05uVsiP2O8 (5/5)

[Joshua Qin @joshzqin]

Excited to share some recent work!

Being good at individual tasks doesn’t automatically make an agent a good company agent. It needs deep understanding of its company: people’s priorities, judgment, and working relationships. We built Company Agent Bench to start measuring this.

[Phil Chen @philhchen]

Today we’re releasing Company Agent Bench, a benchmark for company assistant products.

We compare Dots, Grok Bot, Claude Tag, and Devin on how well an agent understands its company.

We also found that API models with a simple harness perform better than these products (1/6)🧵

[Joshua Qin @joshzqin]

@OpenAI Dots scored highest among the four products at 75.9%, leading on authorship and mentions. @Grok Bot led on reply matching. (3/5)

[Joshua Qin @joshzqin]

We also find that dedicated agent products built for Slack don’t necessarily outperform an API model with a simple Slack harness.

Six of nine GPT configurations beat Dots, the highest-scoring product. There’s still significant room for products to turn access to company context into stronger company understanding. (4/5)

[Joshua Qin @joshzqin]

@clairexu198 @sailresearchco @khanacademy sharky xu

[Joshua Qin @joshzqin]

@jinwong01 @conceptdotdev 🐐

[Joshua Qin @joshzqin]

@philhchen Very insightful read, which of these problems do you think will be solved first?

[Joshua Qin @joshzqin]

@philhchen @MeckaAI awesome work!

[Joshua Qin @joshzqin]

@tkkong @MeckaAI 🔥🚀