Stefan Krsteski

[Stefan Krsteski @stefkrsteski]

@Flowrian135 From what we've seen, enterprise tasks are harder to verify and require more checks. That may partly reflect the lack of real enterprise data. For language coverage, there are few non-English benchmarks to build on. We're currently assessing MAPS, but open to suggestions!

[Stefan Krsteski @stefkrsteski]

Excited to share Messier, an agent evaluation corpus accepted at #EMNLP2026.

Messier unifies 31 benchmarks, 12k tasks, and 118k agent trajectories. It standardizes environments, tasks, models, scaffolds, and verifiers (explicit types) in a single place.

https://t.co/1pUf9pYDsx

[Stefan Krsteski @stefkrsteski]

I’ll be presenting Messier at EMNLP 2026 in Budapest!

Before then, I’ll also be at #COLM2026 in San Francisco presenting our follow-up work on task-difficulty prediction at WAB.

If you’ll be at either event, I’d love to connect! 👋
https://t.co/siOIUb8a5I

[Stefan Krsteski @stefkrsteski]

One example: in contemporary task generation a key question revolves verifier design. Recent efforts, including System 1 models such as CLMs/Jev/Laya, train on agent trajectories and show promise as verifiers. We believe Messier is a great resource to test/train such models.

[Bob West @cervisiarius]

1/ The state-of-the-art way of monitoring state-of-the-art models is to read tea leaves.
Left: a model reasoning to itself.
Right: the same model talking to other agents.
Every frontier lab depends on this, @sama @DarioAmodei @demishassabis

[Stefan Krsteski @stefkrsteski]

Alignment doesn't have to start at post-training! Persona binding from token zero beats midtraining alignment at 3B params, with the advantage growing as pretraining budget increases. Excited about where this direction could go as it scales further.

[Raghav Singhal @ragghhavvv]

New paper!
Synthetic Persona Pretraining (SPP): Alignment from Token Zero

Rather than aligning models after pretraining, we do so from the very start.

SPP models are more faithful to our constitution, less misaligned, & more robust to jailbreaks. Advantage grows with scale! 🧵

[Raghav Singhal @ragghhavvv]

New paper!
Synthetic Persona Pretraining (SPP): Alignment from Token Zero

Rather than aligning models after pretraining, we do so from the very start.

SPP models are more faithful to our constitution, less misaligned, & more robust to jailbreaks. Advantage grows with scale! 🧵

[Julian Minder @jkminder]

New blog!
Synthetic Persona Pretraining (SPP): Alignment from Token Zero

Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. 🧵

[Stefan Krsteski @stefkrsteski]

Our main finding: under a fixed budget, it is often better to reserve most human responses for rectification than spend them all on fine-tuning. Useful for running surveys at scale, especially early on, where you can cheaply expand your effective sample size (up to 14%).
(4/4)

[Stefan Krsteski @stefkrsteski]

New paper at #ACL2026 (main): Valid Survey Simulations with Limited Human Data

Can LLMs replace human respondents? Not really; naive use can mislead. But can they still help? Yes!
📷 https://t.co/70R3eR7iEu
w/ @russogiusep @serinachang5 @cervisiarius @krisgligoric
(1/4)

[Stefan Krsteski @stefkrsteski]

The question then becomes: what to do once a survey is running and some real responses start coming in.
Should you spend them on better prompting, on fine-tuning, or on post-hoc correction? We need better evaluation to know how well these approaches work and interact!
(3/4)