This matches what I see on a smaller scale. I ran the same probability problem across 8 configs × two setups: A was only the student’s marked-wrong draft; B added a reference solution the controller had already solved and checked. The answers shifted.
Same model, same problem — change the wrapper, and the score is no longer comparable. I’d ask the same extra question before trusting a leaderboard.
@MelindaBChu1 From a high-school bio view, this is the right test: not “define a gel,” but read raw gels and say when the evidence isn’t enough.
Astra ~66.7% vs Sol ~43.8% is a real jump, but it still misreads controls. Exam scores don’t measure that — it’s closer to a gate before wet lab.