Simon FL

Husband to @pearlsesq, Software engineer @databricks, French Canadian (i.e. likes poutine and hockey), previously @stripe, @SlackHQ, @Foursquare, @Google

[Michael Bendersky @bemikelive]

We just published OfficeQA Pro - a set of 133 challenging questions from the original OfficeQA benchmark. Even the best frontier agents still struggle on OfficeQA Pro with common issues stemming from errors in parsing, retrieval, and visual reasoning.

[Krista Opsahl-Ong @kristahopsalong]

Most AI benchmarks test reasoning in isolation.

Real enterprise tasks require grounded reasoning:
1️⃣ Find the right documents
2️⃣ Extract the right values
3️⃣ Perform analyses

OfficeQA Pro evaluates this end-to-end. Frontier agents still score <50%.

🧵Paper & details below!

[Krista Opsahl-Ong @kristahopsalong]

Today we’re releasing OfficeQA — a new benchmark for end-to-end grounded reasoning that reflects the real work enterprises need AI agents to do.

More details below 👇

[Simon FL @simonfl]

Why is it "two buck chuck" and not "two bucks chuck"? Is this like the "maple leafs" vs "maple leaves"? English plurals are insane

[Michael Bendersky @bemikelive]

Since joining @databricks, our research team has been hard at work on Agent Bricks, a new product that helps enterprises develop state-of-the-art domain-specific agents. We are now releasing a research blog about Agent Learning from Human Feedback (ALHF) https://t.co/2RDs3H6mkY

[Jonathan Frankle @jefrankle]

RLVR isn't just for math and coding! At @databricks, it's impacting products and users across domains. One example: SQL Q&A. We hit the top of the BIRD single-model single-generation leaderboard with our standard TAO+RLVR recipe - the one rolling out in our Agent Bricks product.

[Simon FL @simonfl]

Hey @minimax_ai, I'm trying to serve M1-80k on vLLM. Your docs say "a server with 8 H800s can process inputs up to 2 million tokens" but then recommend --max_model_len 4096. What settings did you use for 2M tokens? I'm trying this on 8 H100s.

[Simon FL @simonfl]

@harryh I sadly have no experience to offer then.

[Simon FL @simonfl]

@clementmiao @ffx You had me at "split keyboard on your pants"

[Erebus @IdemErebus]

DONDA is the new FAANG

Deepmind
Open AI
Nvidia
Databricks
Anthropic

[Simon FL @simonfl]

@what_mara_said same

[Simon FL @simonfl]

@harryh My bad, I must have misconstrued something I heard!

[Simon FL @simonfl]

@zackdotcomputer @etrade Be ready for that 1 cent to be transferred to Morgan Stanley soon!

[Simon FL @simonfl]

First day on the job at @databricks and we're already making some big moves. Exciting times ahead!

[Ali Ghodsi @alighodsi]

Big news: we've agreed to acquire @MosaicML, a leading generative AI platform. I couldn’t be more excited to join forces once the deal closes. https://t.co/L4TyrruUEU

[Simon FL @simonfl]

I feel like today's war on cars is so much tamer than it was 80 years ago

[Simon FL @simonfl]

@harryh Sidney is much better at this, except it gets confused easily, I think because of the other people who will show up on your LinkedIn profile, it thought I currently had @leok 's job . I assume it's the same problem with ChatGPT

[Simon FL @simonfl]

Now this is AI thought leadership at its finest. A+. No notes

[Yann LeCun @ylecun]

I don't mean to make a bad taste joke, but pronouncing GPT-4 in quasi-French ("gé pé té for") sounds *very* awkward.

[Simon FL @simonfl]

@TheKhandyman is you taking notes on a criminal fucking conspiracy?