Vineeth Reddy Guda

building and doing @distil_labs

[Jacek Golebiowski @j_golebiowski]

Where does Jev end and a fine-tuned small model begin? We built an invoice pipeline with both; sorting the inbox: Jev 200/200, nothing to train. Deciding whether to pay: Jev lands at 0.79 but a 4B fine-tuned to reason first gets 0.98. Details from @distil_labs below.

[Jacek Golebiowski @j_golebiowski]

If you are looking for a model that does not exist (large too slow, generic small too dumb) just build one with distil labs!

[distil labs @distil_labs]

Introducing our new inference endpoint.

Just route your traffic through it, and we capture requests and responses while your LLM keeps answering.

Then, turn it into a synthetic dataset and a custom SLM. No data prep required.

Try it now: https://t.co/HqZyRtEAqu

[distil labs @distil_labs]

Before you add a model router to your stack: there is a version that does not need one. Train the small model to emit defer_to_larger_model when a request is beyond it. It handled 96% of support turns in our cascade, design write-up by @j_golebiowski.

[Vineeth Reddy Guda @VineethGuda]

actually we distilled a model to this job way cheaper than Jev and more accurate

Mode: https://t.co/kCUQ7FOR2r

[nikhil mudholkar @nikhilmudholkar]

Jev lost to Gemini on our email classification benchmark.

I’m still interested in putting it into production.

We tested 1,565 German and English business emails across 10 categories from Industrial suppliers.

The interesting result wasn’t accuracy. It was where the mistakes happened. 🧵

[Selim Nowicki @selimskii]

We’ve just launched mutators at @distil_labs. You tell the system which situations matter for your small model, even the rare ones, and it generates examples of them for training.

https://t.co/ODmCof6cxa

[Jacek Golebiowski @j_golebiowski]

Mutators are live at @distil_labs. You describe something you know about your product that is missing from your traces, and the teacher model turns it into training examples for your small model.

https://t.co/teiH4s6iU2

[distil labs @distil_labs]

Mutators are now available in distil labs. You describe the situations missing in your traces that you want in your training data and the teacher model generates examples of them for your small model to learn.

https://t.co/UdcmdqTJRr

[distil labs @distil_labs]

One of our customers builds an AI tool for reviewing documents. Their traffic came in bursts that hit the API’s rate limit, so we trained a small model and put it on dedicated capacity to handle those bursts. Their bill also fell by about 20%.

[distil labs @distil_labs]

You can now sign up and train a task-specific small model yourself, in about 30 minutes. Point it at your production logs; six stages run mostly unattended; out comes an OpenAI-compatible endpoint. From the CLI or your coding agent.

https://t.co/cmQerKW37H

[Jacek Golebiowski @j_golebiowski]

We ran our first Builder's Night at the @distil_labs office in Berlin yesterday: bring a task you send to a frontier model, train a small model on it, leave with it deployed.

20 engineers built, the last ones still running evals at sunset. We'll do this again.

[Vineeth Reddy Guda @VineethGuda]

@hthieblot Knowledge distillation

[distil labs @distil_labs]

If an LLM runs inside your product, you are already generating the training data for its cheaper replacement. One education platform's essay grader: fine-tuned from ~4,000 production logs, zero labeling, 89.9% vs 84.8% for the API it replaced. Details via @j_golebiowski below.

[Jacek Golebiowski @j_golebiowski]

The open models debate is running on ideology this week. From a production seat it's an empirical question, and for the narrow, high-volume tasks that dominate real AI traffic it looks settled: open weights already won that tier.

[Jacek Golebiowski @j_golebiowski]

https://t.co/Clr4uv8bWz

[Maciej @maciejgryka]

Nice internal refactor by the team resulting in much easier way to get started with distil.

[Jacek Golebiowski @j_golebiowski]

We just shipped something I'm quite happy about: distil labs now takes training data as complete conversations. One messages list per example, the same OpenAI-style format your logging stack already produces, instead of question/answer pairs.

[Vineeth Reddy Guda @VineethGuda]

@DhravyaShah Cool man :)

This is stg I’m actually looking at rn, happy to chat

[Jacek Golebiowski @j_golebiowski]

We keep building the same thing for education platforms: a classifier that replaces a frontier API on one narrow, high-volume decision. Scam detection, request routing, essay grading. Different tasks, and the same pattern keeps showing up.

[Jacek Golebiowski @j_golebiowski]

We pick small models by their out-of-the-box scores. But that measures what a model already knows, not how much it can learn. So with @NVIDIA we measured tunability instead: 4 mid-size MoE models, 7 tasks, before and after.