Founder & CEO of @servamind · PhD in AI · building the data layer AI runs on: encode once: train, fine-tune & serve for less. Papers + open source this fall.
I've spent my career on the underlying math of AI — and the question of what it actually takes to get to a Data-from-TNG level of intelligence.
Now I'm building @servamind: a model-agnostic data layer for AI. Encode your data once, and the same file works across training, fine-tuning, and inference — less runtime, less power, targeting ~1/10th the cost.
This fall: two research papers and an open-source release, with a repro kit so you don't have to take our word for anything.
Follow along — I'll be posting about the tokenizer tax, why data preparation is AI's most ignored bottleneck, and what we find as we run the benchmarks.
@fastinoAI@huggingface Calibration is the more interesting number here than the 71.5. Curious how it holds when the decision distribution shifts, since a lot of models look well calibrated only on data that resembles the test set.
@eliebakouch Tool-model framing makes sense to me. The hard part is the agent knowing when to call it, and I almost never see a routing eval in these releases. Beating other models on the lab's own benchmark tells you little about that.
When ChatGPT first landed and LLMs became popular, the OGs in the field went a-wol or quiet. I am SO happy to see the real experts speaking up again.
AI is not new, been around for half a century. People just don’t read. We can make more progress if AI people align on knowledge first, then we can get AI aligned.
Kudos to @bengoertzel and @Plinz and others for carrying us!
@l_ringsss Read failures first is right, with one addition: freeze the gold set before you touch the grader. Otherwise you tune the grader to pass your model and call it validation.
@MistralAI Which scoring: final-state check or rubric-graded? On simulated workplace apps that gap moves rankings more than the model gap does. Would love to see variance across reruns.
@bgvc123 The missing metric is recovery. Everyone reports pass rate at turn 50, almost nobody reports what happens after the first silent tool failure at turn 12. That's where agents actually die.
@stas_sorokin_ Both misses passing visible tests is the actual result here. The shared overlap check is the tell: the model optimized for the failing test and never looked at the other callers. n=12 won't rank models, but it ranks graders.
@a16z A16Z is killing it with market info the past few months. Supply demand and the chain is so misunderstood.
What will it take to get mass adoption?
@casper_hansen_ The catch is the cold miss. NVMe random reads are fine for hot entries, but the long-tail facts are exactly what you'd want that memory for and exactly what won't be cached.
@bo_wangbo If freshness and authority are doing real work, how much of embedding-Gemma's Q2D lead survives when you score pure relevance? That split would tell people whether to swap embedders or just add a reranker.
@loldedxd Does it standardize the planning interface too, or just data loading and baselines? That's usually where every world-model repo ends up diverging.