@alexshander03 1100%. That is the bottleneck I kept hitting in voice AI: personas that behave like real users, at scale, from prod data. One by hand is easy. Hundreds is the hard part. I am building that input layer, your judges would grade realistic traffic instead of cases someone wrote.
@alexshander03 Genuine question: what is your take on evals with simulated humans pre-prod instead of static cases? I am building in that direction, the multi-turn edge cases (mind changes, contradictions) are exactly what static tests miss. Curious if you have explored it
@tb18_08 If you use cloud hosted models like GPTs on azure, you will not see a difference when they release a new model, because your old one will still be on the same infrastructure
@tb18_08 It’s only because they don’t have enough hardware. They prefer to allow the good infrastructure to the newer models. But the older models are not worse, they are just slower, and it feels different when we use it because of the difference with the new models
@itsPaulAi Based on my personal feelings it feels around 2 times slower, on a huge codebase. I need to try it on different usecases.
What are your speed feedbacks on openclaw with Minimax ?