The Ruby Gem "llm-evals" ranks models by a weighted score across real intake cases. DeepSeek won overall on https://t.co/gsIAPrqUnP, but every dataset crowned a different winner. There is no best model, only the best model for your task. https://t.co/0rKto8VOb9
Debatable.
This was built for https://t.co/pUqVuvbbXg โ a custom form builder and lead qualifier. You design a form on the above app and drop generated JS on your static service website.
If you start your form with free text field like all the demos (and evals), I can imagine a product feature where YOU provide a set of intro texts for YOUR web form / questionnaire, and product runs evals behind the scenes to tell you the best model/effort/temperature for your text examples and your form.
You could also let the user define the weights but I am afraid it's getting too technical for a SaaS forms product.
In fact the 5-10 intros you write pretending to be your own client are likely to impact the end result more than anything.
That said, higher end models have gotten pretty damn good at writing their own test cases. So here you could just hit a button that would generate 10 text snippets based on various descriptors (like: 1. Precise, long text answers every question ahead, 2. vague and poor grammar, 3. Long and emotional, 4. Angry and demanding, and so on). AI generated these texts, plugs them into the form de structuring step, and picks the model based on weights applied to missed answers, invented answers, correct and so on.
Maybe I'll add this feature to the users who set themselves as "AI advanced" in the settings?
Debatable.
This was built for https://t.co/pUqVuvbbXg โ a custom form builder and lead qualifier. You design a form on the above app and drop generated JS on your static service website.
If you start your form with free text field like all the demos (and evals), I can imagine a product feature where YOU provide a set of intro texts for YOUR web form / questionnaire, and product runs evals behind the scenes to tell you the best model/effort/temperature for your text examples and your form.
You could also let the user define the weights but I am afraid it's getting too technical for a SaaS forms product.
In fact the 5-10 intros you write pretending to be your own client are likely to impact the end result more than anything.
That said, higher end models have gotten pretty damn good at writing their own test cases. So here you could just hit a button that would generate 10 text snippets based on various descriptors (like: 1. Precise, long text answers every question ahead, 2. vague and poor grammar, 3. Long and emotional, 4. Angry and demanding, and so on). AI generated these texts, plugs them into the form de structuring step, and picks the model based on weights applied to missed answers, invented answers, correct and so on.
Maybe I'll add this feature to the users who set themselves as "AI advanced" in the settings?
The Ruby Gem "llm-evals" ranks models by a weighted score across real intake cases. DeepSeek won overall on https://t.co/gsIAPrqUnP, but every dataset crowned a different winner. There is no best model, only the best model for your task. https://t.co/0rKto8VOb9
building architecture should be designed by AI and transmitted for execution to building robots in some AI invented machine language humans dont understand. So you get to enjoy a slop-full of a skyscraper while its falling over when its built to about 60%.
You want to live in the physical world built by robots controlled by AI, where they stuff a tube down your throught to force feed you so your obese body generates enough heat to power one tiny power cell.
Not sure how I ended up in the Matrix, but thats what building complex distributed software without a common visual language feels like.
@Parsa25519@Grady_Booch@AnthropicAI UML Distilled was one of the most influential technical books Ive read in my 30-year software career. To say that Claude code is somehow comparable to UML is like saying that
Sopsy 2.0 lets every member keep their key where it suits them, within the repository's bounds. It adds commands to change a single secret or feed secrets directly to a process, and it is honest about what rotating a key does. Don't YubiKeys work on Linux? https://t.co/VTGMgutp6l
Published a gem agent-evals that requires you to write a couple of adapters for your datasets, and it runs them and dumps the results however you want. No paid framework required but if you have a @braintrust account and you set the env vars jt will show up there as an experiment. Any number of eval measuring company adapters can be added if needed. The screenshot is of my config for a SaaS app I am running.
@Dakshagarwal76 They are part of the qualified at Reaper, which is a SaaS paid app and therefore private. They are all generated by myself and partially by AI.
@gaurav0bhowmick Weights don't pick the winner, they balance different wrongs against each other and versus being right. It affects the winner, but it's not a deterministic thing. That's the whole point.
I only have 4 cases per data set, and I could definitely have more. I think that a better strategy is to collect real users' entries and see how they perform, and then retune the weights or the model.
I don't know how ethical it is to use your users' actual data in the vowels. And I don't know if I feel good about doing that without consent. But from an actual accuracy standpoint, I don't really see a better way.
It's less of a failure to say nothing (or I don't know) than to hallucinate something and be confidently wrong. Thats why the weights penalize it more. In the above table "error" or "invalid_rate" or "hallucinated_rate" are top contributors that drag the overall score down. Collectively over 44% of the weighted score compared to 5% of score.