Most of the discussion about AI safety centers around coding and software stack exploits which are highly verifiable outcomes. Frontier models excel at verifiable tasks, because this is how they were post-trained.
But in domains where verifiability is not so obvious, like healthcare for example, most of the discussion boils down to whether the model is capable of citing the right research when prescribing a treatment or creating a diagnosis, making it look good on a benchmark.
But this is quite misleading and loses a lot of nuance that goes into this kind of reasoning and it's hard to verify how one treatment is better or worse than the other, what exact dosage should be prescribed and what order of treatment should be followed, especially in non trivial cases.
We started working on a benchmark which we first apply in a functional medicine space and tested a few frontier models vs our own Diadia harness that we built together with our doctors.
Below are just a few snapshots from a larger benchmark, and the key issues that we are seeing comparing to our own harness:
- Missing the order of treatment (this is almost a flip of a coin in terms of what part of the treatment should start first)
- Over-prescribing treatments patients might not need
- Missing key treatments that doctors would've prescribed
- Messing with the dosing
Getting these outcomes wrong can harm a patient even when every citation is correct and the model does well on a benchmark.
There is no worse experience than having an AI bot instantly respond to your question that you already understood from reading the troubleshooting guides
Or a response from a human where they havenโt understood the question and the context of your question.
Why not solve for satisfied customer the same day instead of optimising for a 300ms response time.
@thsottiaux Using mcp.json so I donโt have to manage one file for codex (toml) and another for Claude / Cursor (mcp.json)
Making reconnecting and managing mcps (main one) and skills easy. The latter https://t.co/GYTHHWUxvc helps with but a gui would be nice.