Qokedas

The RL layer for (physical) Science

[Qokedas @Qokedas]

Can today’s frontier models reliably decode data from a real scientific instrument?

We gave four of them a raw recording from a 1,024-microphone acoustic camera and asked for heatmaps showing where the sound comes from.

[Qokedas @Qokedas]

@davidmanheim The models were told the instrument in the first sentence of the brief (Sorama CAM1K, 1,024 mics) and received the device’s own metadata: every microphone position, sample rate, channel table, plus a ground-truth output for the same recording. https://t.co/3NciPrLjGG

[Qokedas @Qokedas]

Can today’s frontier models reliably decode data from a real scientific instrument?

We gave four of them a raw recording from a 1,024-microphone acoustic camera and asked for heatmaps showing where the sound comes from.

[Qokedas @Qokedas]

https://t.co/GZSpliAp4o

[Qokedas @Qokedas]

https://t.co/eTlCXFWYBU

[Qokedas @Qokedas]

3/3 Our concern is simple. Science uses instruments from many vendors, with different formats and conventions. Models need to verify assumptions, respond to contradictory evidence and recover. This is a warning about that failure mode, not a claim that models cannot do science.

[Qokedas @Qokedas]

2/3 The models could compare their predictions with that reference throughout. Fable did so around 15 times, saw the mismatch, kept the same assumption, blamed vendor filters, timing offsets and band alignment, then submitted. The initial wrong guess was not the main result.