Eldan

IT Manager | Local AI enthusiast

Building things and occasionally breaking them

[Eldan @EldanRing]

Introducing Winnow-12B.

Jev-style decisions, chat and vision in one llama.cpp-based server.

A Gemma 4 12B fine-tune running locally at 64K context + vision on my 16 GB RTX 5070 Ti (Q8).

Open weights: https://t.co/D4xvEIdK3H

Benchmarks + inference code below 👀👇

[Eldan @EldanRing]

New data + improved training pipeline are working wonders right now. I will be releasing a second decision-focused finetune soon!

[Eldan @EldanRing]

@airesearch12 Thanks for featuring Winnow-12B! It works great as a classifier and also supports regular chat with long context + vision. For anyone who wants to try it, here are the weights: https://t.co/D4xvEIdK3H

[Eldan @EldanRing]

@melvynx My fine-tuned Winnow-12B can do it! https://t.co/D4xvEIdK3H

[Eldan @EldanRing]

@4rcherhume I just released Winnow 12B! I'd love it if you could try it and give me your take.

https://t.co/D4xvEIdK3H

[Eldan @EldanRing]

@Doctorarii @4rcherhume It's out now!

[Eldan @EldanRing]

Introducing Winnow-12B.

Jev-style decisions, chat and vision in one llama.cpp-based server.

A Gemma 4 12B fine-tune running locally at 64K context + vision on my 16 GB RTX 5070 Ti (Q8).

Open weights: https://t.co/D4xvEIdK3H

Benchmarks + inference code below 👀👇

[Eldan @EldanRing]

Try Winnow-12B:

BF16 + Q8 weights and vision projector:
https://t.co/D4xvEIdK3H

Code, examples and full benchmarks:
https://t.co/m4lDY6pxKU

/v1/systemone for typed decisions.
/v1/chat/completions for chat + vision.

[Eldan @EldanRing]

Introducing Winnow-12B.

Jev-style decisions, chat and vision in one llama.cpp-based server.

A Gemma 4 12B fine-tune running locally at 64K context + vision on my 16 GB RTX 5070 Ti (Q8).

Open weights: https://t.co/D4xvEIdK3H

Benchmarks + inference code below 👀👇

[Eldan @EldanRing]

On my RTX 5070 Ti, Q8:

64K + vision, 15.01 GiB peak GPU memory.
55.5 tok/s short-prompt generation.
46.9 tok/s with a 62K multimodal prompt.

Four decisions over the populated 64K vision fixture: 143 ms (cached)

Exact settings + sample counts in the report.

[Eldan @EldanRing]

I’ve been building Winnow-12B, a Gemma 4 fine-tuned for Jev-style decisions, with chat + vision on the same llama.cpp server.

The Q8 candidate passed 64K + vision on my 16GB 5070Ti. Comparing it with Kev and Laya now. Numbers coming with today’s release.

[Eldan @EldanRing]

Updated Qubicl to 0.6 over the last few days! This weekend I’ve been working on a finetune. Release day tomorrow!!

[Eldan @EldanRing]

@matvelloso Me 👋 I’m finishing a Gemma 4 finetune with long-context and answers Jev-style questions, while retaining normal chat and vision. Same llama.cpp-based server, running at 64K + vision on my 5070 Ti. Open weights and code coming tomorrow!

[Eldan @EldanRing]

@jaredpalmer Very cool release! Exciting to see open decision models taking off. I’m working on a Gemma 4 12B post-train that combines Jev-style decisions with chat and vision in one llama.cpp-based server. It runs at 64K context with vision on my RTX 5070 Ti. Releasing tomorrow!

[Eldan @EldanRing]

@GitHubNext I've been doing this myself and gotten great results so far with Gemma 4 12B. Currently fine-tuning, and the results look promising.

[Eldan @EldanRing]

@madiator The Gemma 4 12B base model tested better than Qwen for me. I wonder if a fine-tune of that model would outperform this.

[Eldan @EldanRing]

Feels like I blinked and the AI landscape is no longer the same. We truly have crazy times ahead.

[Eldan @EldanRing]

Okay, Jev is pretty cool. Currently trying to get something similar running locally on my GPU. I'll release it if it's worth it!