Ryan Marten

Building @harborframework and @terminalbench

[Ryan Marten @ryan_marten]

https://t.co/ZVmVZVJpCX

[Ryan Marten @ryan_marten]

It doesn't take a superforecaster to know that Ven and team are going to do big things!

[Veniamin Veselovsky @VminVsky]

Over the past year I've been building out Preseen. We’re focused on building the best AI forecasting agent for decision makers.

Every decision relies on a forecast. Too often decisions are poorly made because the underlying forecast is bad. Humans are constrained. But if you can have a system that can forecast anything as well as the best humans, then human decision making will be improved.

AIs are reaching the point where they are better than some of the best human forecasters. This is going to change decision making.

Preseen will help drive that change.

[Ryan Marten @ryan_marten]

@modal_labs @AnthropicAI @OpenAI @Google @scale_AI @SnorkelAI @turingcom @gNucleusAI @joinHandshake @ellamindAI @giansegato @boolean_ai Thank you to @modal for providing the sandboxes for developing Frontier-Bench! This was particularly useful for unlocking GPU tasks (e.g. agent is asked to write kernels, run evals, train models, etc.).

[Ryan Marten @ryan_marten]

We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.

Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.

Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%

[Ryan Marten @ryan_marten]

@modal_labs @AnthropicAI @OpenAI @Google @scale_AI @SnorkelAI @turingcom @gNucleusAI @joinHandshake @ellamindAI @giansegato Thanks as well to @boolean_ai (missed the tag above). It was a pleasure working together!

[Ryan Marten @ryan_marten]

@Curline1222 working on it! we will update the leaderboard with new results soon! (other models are also in progress / on our run roadmap)

[Ryan Marten @ryan_marten]

@ajratner @SnorkelAI Thank you so much for your support! Open Benchmarks Grants is so great for supporting the evaluation ecosystem!

[Ryan Marten @ryan_marten]

@dwahdany Thank you Dariush for all your amazing work! It was really fun building this together!

[Ryan Marten @ryan_marten]

@ivanovm_ Thank you for contributing!

[Ryan Marten @ryan_marten]

@xliu_me 🫡

[Ryan Marten @ryan_marten]

@marcusmotill Would love to hear how it goes!

[Ryan Marten @ryan_marten]

@realjustinbauer Thank you for all your great work in making this happen! Pleasure collaborating with you Justin!