Jσsh

AI engineer | Building enterprise world models

[Jσsh @JoshKuechly]

@xadisingh Agreed! However, I will be interested to see the business case for @typesafeai evolve as more competition arrives.

[Jσsh @JoshKuechly]

@hmartenjoyer @nvidia @Meta @OpenAI True! :)

[Jσsh @JoshKuechly]

I don’t think Jev will survive.

@nvidia and @meta will release open-source Jev competitors within the next 30 days.

@OpenAI will add a Jev-like endpoint to their model list in the next 60 days.

And the rest of us will fine-tune custom classifiers that crush Jev and are 10x faster.

In two years we’ll look back at this flash-in-the-pan and say

“remember Jev?”

[Jσsh @JoshKuechly]

@Appyg99 @nvidia @Meta @OpenAI Fair! They’ve got some strong momentum. But do you think they have a moat that lasts?

[Jσsh @JoshKuechly]

Also, to be clear - I think generalist classifiers like Jev are awesome and will only get better. I think they’ll dominate the long tail ~80%, while fine-tunes will become popular for the Pareto ~20%.

I also think open-source generalist classifiers will beat out Jev. Meta / NVIDIA have a ton of resources to tackle this. Do you think they will?

[Jσsh @JoshKuechly]

@chadboyda @nvidia @Meta @OpenAI You make some good points, Chad! I will say that fine-tuning (with the help of coding agents) has gotten dramatically easier. It took me 51 minutes to tune a custom classifier that crushes Jev and is 8.8x faster:

https://t.co/WuNwLmD8Ay

[Jσsh @JoshKuechly]

Jev from @typesafeai is great, but way overhyped.

I fine-tuned an open-weight model on my Mac that beats Jev by almost 10pp, and is 8x faster.

It took less than an hour and runs on my local CPU. Yes, it's task-specific. Meaning it doesn't beat Jev on every task. But that's ok:

Jev is pretty good at zero-shot general classification, but a quick fine-tuning run allows GLiNER 2.5 (open weights, built by @fastinoAI) to beat it on a specific task as a specialized classifier.

In this experiment, I tested default GLiNER 2.5 vs Jev vs Fine-tuned GLiNER 2.5 on the Banking77 benchmark from @huggingface.

Default GLiNER 2.5 trailed Jev by ~9pp, despite being 10x faster.

After fine-tuning GLiNER 2.5 on ~10k training samples (this took about 51 minutes on my M4 Max), I retested.

Fine-tuned GLiNER 2.5 gained ~18pp of classifying accuracy on the task, beating Jev by ~9pp.

I'm betting that while Jev thrives as an easy plug-and-play endpoint for non-recurring classification tasks, most teams shipping to production will just take a little bit of time of front to fine-tune open-weight classifiers for tasks.

If you can spend an hour fine-tuning to get:
- 8-10x speed boost
- 10pt accuracy lift
- AND it can run locally on your hardware?

That seems like a no-brainer.

So yes, @rileybrown, it seems like you can fine-tune your own "jev" on your data :)

[Jσsh @JoshKuechly]

Jev didn’t invent cheap classifiers. But they executed well, with great story and marketing. Also, the application of classifiers to agentic workflows is really smart. Non-ML folks are blown away and think it’s brand new…Remember that most people think that LLMs didn’t exist before Nov 2022.

I’ve fine-tuned quite a few classifiers over the years. GLiNER is one of my favorites.

[Jσsh @JoshKuechly]

@RiqwanMThamir @nvidia @Meta @OpenAI Oh I am :)

[Jσsh @JoshKuechly]

@IgalPines @trycua Agreed. But even for the generalist classifier, I think open source will dominate. NVIDIA and Meta will ship their own open source classifiers that crush Jev. As will a bunch of smaller players.

[Jσsh @JoshKuechly]

Jev is great at zero-shot classification, but specialist classifiers will dominate commercial use cases.

@trycua tuned a tiny model that scored 99.7% on their form-filling eval. Hosted Jev scored 83.6%.

I tuned GLiNER 2.5 on a task in 51 minutes yesterday and it crushes Jev. And it's local. And 8.8x faster:

You too can do this. Linked post in comments.

[Cua @trycua]

1/ Introducing CUA-S1: a family of System One Models, small, specialized, and built for computer use.

Today we're open-sourcing CUA-S1-FORMS, the first in the family: https://t.co/J1frbEbZTQ

[Jσsh @JoshKuechly]

Great question. Yes, this is why a generalist classifier will eat the long-tail of tasks that aren’t economical to fine-tune for. But you only need 5-10k labeled samples to fine-tune something like GLiNER on most tasks. If you can’t find free training data for your use case on huggingface, you can use GPT 5.6 Sol or Opus to generate the 10k training labels. You’re effectively distilling the frontier model’s intelligence into the fine-tuned smaller model for that task.

[Jσsh @JoshKuechly]

Generalist classifiers will definitely get better. They will likely handle the long-tail 80% of classification tasks. But it’s not compute-intensive to build/train classifiers.

So my disillusionment with Jev is twofold:

- for generalist classifiers, open source will win
- for specialist classifiers, fine-tune will win

[Jσsh @JoshKuechly]

@lucataco @trycua Nice! Results?

[Jσsh @JoshKuechly]

@petergostev I think Jev is cool. But I’m much more excited about fine-tuning specialized open-weight classifiers that crush Jev and are 8-10x faster

[Jσsh @JoshKuechly]

Jev is great at zero-shot classification, but specialist classifiers will dominate commercial use cases.

@trycua tuned a tiny model that scored 99.7% on their form-filling eval. Hosted Jev scored 83.6%.

I tuned GLiNER 2.5 on a task in 51 minutes yesterday and it crushes Jev. And it's local. And 8.8x faster:

You too can do this. Linked post in comments.

[Jσsh @JoshKuechly]

@ShauryaCODE @typesafeai Yes, GLiNER can run locally on CPU. It’s very fast, but can slow a bit under heavy concurrency

[Jσsh @JoshKuechly]

Jev from @typesafeai is great, but way overhyped.

I fine-tuned an open-weight model on my Mac that beats Jev by almost 10pp, and is 8x faster.

It took less than an hour and runs on my local CPU. Yes, it's task-specific. Meaning it doesn't beat Jev on every task. But that's ok:

Jev is pretty good at zero-shot general classification, but a quick fine-tuning run allows GLiNER 2.5 (open weights, built by @fastinoAI) to beat it on a specific task as a specialized classifier.

In this experiment, I tested default GLiNER 2.5 vs Jev vs Fine-tuned GLiNER 2.5 on the Banking77 benchmark from @huggingface.

Default GLiNER 2.5 trailed Jev by ~9pp, despite being 10x faster.

After fine-tuning GLiNER 2.5 on ~10k training samples (this took about 51 minutes on my M4 Max), I retested.

Fine-tuned GLiNER 2.5 gained ~18pp of classifying accuracy on the task, beating Jev by ~9pp.

I'm betting that while Jev thrives as an easy plug-and-play endpoint for non-recurring classification tasks, most teams shipping to production will just take a little bit of time of front to fine-tune open-weight classifiers for tasks.

If you can spend an hour fine-tuning to get:
- 8-10x speed boost
- 10pt accuracy lift
- AND it can run locally on your hardware?

That seems like a no-brainer.

So yes, @rileybrown, it seems like you can fine-tune your own "jev" on your data :)

[Riley Brown @rileybrown]

Okay i want to fine tune my own jev based on my own data.

Is this possible? Will this be possible?

[Jσsh @JoshKuechly]

Here's the original post on my fine-tune that beat Jev in an hour by almost 10pp and ran locally 8.8x faster.
https://t.co/MxJRqbX86W

[Jσsh @JoshKuechly]

Jev from @typesafeai is great, but way overhyped.

I fine-tuned an open-weight model on my Mac that beats Jev by almost 10pp, and is 8x faster.

It took less than an hour and runs on my local CPU. Yes, it's task-specific. Meaning it doesn't beat Jev on every task. But that's ok:

Jev is pretty good at zero-shot general classification, but a quick fine-tuning run allows GLiNER 2.5 (open weights, built by @fastinoAI) to beat it on a specific task as a specialized classifier.

In this experiment, I tested default GLiNER 2.5 vs Jev vs Fine-tuned GLiNER 2.5 on the Banking77 benchmark from @huggingface.

Default GLiNER 2.5 trailed Jev by ~9pp, despite being 10x faster.

After fine-tuning GLiNER 2.5 on ~10k training samples (this took about 51 minutes on my M4 Max), I retested.

Fine-tuned GLiNER 2.5 gained ~18pp of classifying accuracy on the task, beating Jev by ~9pp.

I'm betting that while Jev thrives as an easy plug-and-play endpoint for non-recurring classification tasks, most teams shipping to production will just take a little bit of time of front to fine-tune open-weight classifiers for tasks.

If you can spend an hour fine-tuning to get:
- 8-10x speed boost
- 10pt accuracy lift
- AND it can run locally on your hardware?

That seems like a no-brainer.

So yes, @rileybrown, it seems like you can fine-tune your own "jev" on your data :)

[Jσsh @JoshKuechly]

@the_og_karan @urchadeDS @typesafeai https://t.co/RqZNAic5T4

[Jσsh @JoshKuechly]

@the_og_karan @urchadeDS @typesafeai https://t.co/NYmL2de1Ip

[Jσsh @JoshKuechly]

@the_og_karan @urchadeDS @typesafeai Yep! GLiNER 2.5 is apache-2.0. I used LoRA to fine-tune it on ~8k labeled examples locally on my Mac. Their repo has custom-data and fine-tuning guides. Claude/codex can help here too!

[Jσsh @JoshKuechly]

@Namidaka1 @typesafeai I've gotten burned by overfitting before lol so was pretty cautious to ensure zero overlap. My exact split:

Training run: 7,967 messages

Dev holdout (unseen): 965
Additional holdout (unseen): 1,071
Full benchmark (unseen): 3,080