@TheRealArieh I don't think the goal of the benchmark is to find the best model to use in practice, it's to find places where the models can be challenged and find indicators of more general capabilities/intelligences
I'm confused why people are talking about the "top" results here. Bigger models being better at tasks resting only on parametric knowledge seems like a trivial result. The interesting result is among models of (known & ) similar size.
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet.
The best scored 80 out of 100. Many scored under 11. 🧵
@DavidRegev Am I understanding correctly that this is meant for increasing accessibility for the interested public? Also feel like I'm missing a basic piece of the idea here -- the inclusion of teaching reading traditions seems rather out of place among the list of features.
@yuwen_lu_ This looks fascinating! Do you think this could be used to learn/differentiate different manuscript hands based on decomposing the letter strokes? Is this building on existing work?