@DaveShapi That’s why I’ve been saying ai development needs metrology.
you need repeatability studies before interpreting a result — same setup, N runs, how often does it reproduces
I transposed MSA studies and made MSAI Studies : https://t.co/tJOkbIo5ZY
@TheAhmadOsman What’s the noise floor on that index? 63 vs 62 vs 61 — real ordering, or inside the measurement uncertainty?
Most eval suites have never had a repeatability study run on them.
Metrology solved this a century ago. Been applying it to LLM eval → https://t.co/fd8wnGVDmu
@elder_plinius https://t.co/tJOkbIo5ZY
That’s why metrology needs to become apart of the ai open source community.
For it to auditable we need to establish repeatability, reproducibility and also handle uncertainty.
I have open sourced MSAI studies for this purpose.
I checked my food and as I was about to drive off a chick fil a employee walked in front of my car.
I wasn’t even close to hitting them but if I did I’m pretty sure they’d apologize to me and offer to buff out that spot on my car.
@shaelaspell Better come home dressed up as Chad Kroeger driving a bumble bee Camaro full of Green Day and Shadow the hedgehog memorabilia… or maybe don’t come home at all.