Ty

Tech + AI
Own your weights ⚖️

[Ty @tysn_dev]

@omkarships No, abstraction and critical thinking with the knowledge of programming fundamentals is like a superpower compared to marketing larpers

[Ty @tysn_dev]

@ASvanevik Having trouble getting it working in a Dell server rn

[Ty @tysn_dev]

I need Astra level intelligence in my basement in 6 months

Deepseek
Nemotron
GLM
Qwen

I’m looking at you guys 🫪

[Ty @tysn_dev]

Don’t hmu.

I’m vibecoding all weekend.

[Rhys @RhysSullivan]

What if we are all unrestricted agents and this is our message board

[Ty @tysn_dev]

@aijoey @mr_r0b0t @chargerlab @storagereview @nvidia @NVIDIAAI @NVIDIARTXSpark Wow 👏

[Mau “Rules without Rulers” Ledford @krunkosaurus]

5th best agentic model in the entire world. And you can run it at home on 2 Sparks for $10k.

That's where AI is at this point in the timeline.

[Ryan Wright @IQReactorAI]

Guys and Girl nerds, the tiny 500m-1.2B models trained for a single task are taking off cause they are dirt cheap to run and do that task better then frontier models.

Us Local AI folks can run AND train these at home. (I will soon)

Actually ☝️🤓 For a training a highly specialized 1B model, 1,000 to 5,000 high-quality, varied examples are often enough to achieve frontier-level performance on that specific task. Not 50,000 to 1 million like with big models.

Checkout this blog article by @ liquidai where they talk about their tiny task models.

"LFM2-Extract – A 350M and 1.2B multilingual models for data extraction from unstructured text, like turning invoice emails into JSON objects.

LFM2‑350M‑ENJP‑MT –  A 350M model for bidirectional English ↔ Japanese translation.

LFM2‑1.2B‑RAG – A 1.2B model optimized for long‑context question answering in RAG pipelines.

LFM2‑1.2B‑Tool – A 1.2B model built for function calling and agentic tool use.

LFM2‑350M‑Math –  A 350M reasoning model for solving mathematical problems.

Luth-LFM2 – An additional community-driven series of French fine-tunes to enable general-purpose assistants for on-device chat."

Read more here: https://t.co/GqsOAhUD3C

[Ty @tysn_dev]

@NousResearch Curious on how you guys will keep up with runtimes as new methods/updates come out constantly for consumer hardware

[Lee Edwards @ledwards]

If this bill passes, with 100% certainty, this is the chain of events.

American recession. ➡️

Global recession ➡️

Chinese AI research continues. ➡️

Early adopters switch to Chinese models. ➡️

Everyone else follows. ➡️

AI usage continues, but with models US can’t regulate.

[Ty @tysn_dev]

AGI should make me dinner, do my laundry, fold my clothes, run errands for me, etc.

ASI should cure cancer, solve world hunger/peace, provide abundance.

Computer use and good multimodality is not AGI. These are stochastic LLM’s with deterministic harnesses. Tools for now.

[Ty @tysn_dev]

@dsjfnsdlkjfbsdl I have, it’s smart just too slow

[Ty @tysn_dev]

TP=3 just isn’t too worth for GB10’s imo. Granted I haven’t made recipes, but gains are small.

There are better mixes of models for 1x spark and 2x sparks and a big jump in performance to 4x.

Recommendations:

Single = Qwen3.6 35b-a3b
Dual = Deepseek V4-FV
Quad = Deepseek V4-FV

[Pierre-Luc @dallairedemers]

Ban vehicles that match or exceed horse capabilities.

[Ty @tysn_dev]

Single Spark models will become a staple for what people use!! No physical clustering going forward.

[PC Watch @pc_watch]

RTX SparkミニPC「ProArt GR1X」。120BのLLMをローカル実行 https://t.co/knlQDfchVC

[Ty @tysn_dev]

@exdayai It’s good just too slow. 100+ tok/s is starting to be the norm so a small MoE is needed.

[Yume_X @yume_arasaki]

If you have an RTX 3090 or 4090, @MiaAI_lab just shipped you a free massive upgrade in both speed and intelligence.

I will explain to you why this will make your Qwen 3.8 27B on your card, even better, and my flags for running it.

Qwen3.8-27B, EXL3 3.5bpw, DFlash2 speculative decode, RTX 4090. Single stream.

The kit is from MiaAI-Lab, EXL3 is turboderp's format.

I re-measured everything on my own card because the my first benchmarks seemed off. It turns out it really does run much faster.

WHY EXL3 IS A DIFFERENT ANIMAL

The old way (Q4_K_M) rounds each weight to the nearest 4-bit value independently. Every weight introduces its own rounding error. Those errors accumulate across millions of weights and causes drift (Slightly dumber).

EXL3 is a fundamentally different compression algorithm. Instead of rounding each weight on its own, it encodes the entire weight vector as a path through a constrained codebook and spreads the rounding error across dimensions using a Hadamard transform.

The result is that at the same bits per weight, more of the original model's intelligence is preserved.

The important part is this CAN ACTUALLY BE MEASURED.

The cleanest way to see that is KL divergence against a high-precision teacher. Lower means the quantized model thinks more like the original. On the malaiwah independent teacher-logit panel for GLM-5.3-Flash:

EXL3 4bpw: 0.0246 nats
Official FP8: 0.0206 nats
NVFP4: 0.0605 nats

EXL3 sits 0.004 nats behind native FP8 at half the size. NVFP4 at higher bit width is 2.5x further from the teacher. That panel is GLM-5.3-Flash, not Qwen 3.8. Cited as the mechanism, not as this run's data.

But the point stands: EXL3 is not just smaller, it is smarter per bit than the formats most people are running.

WHAT I MEASURED

I first measure 108 tok/s from a single run. After that number looked too good to be true. I reran it. It looks like after a warm up, the numbers are even better.

Basically, like people long thought, the RTX 3090 and RTX 4090 are actually superb AI computer cards. Hence why NVIDIA stopped shipping them with NVLINK since the 4090.

Short context ceiling (~2k in, 1016-token output, TTFT-separated): 135, 138, 153, 174, 133, 129 tok/s across 6 runs.

Sustained longform (2040-token essay): 105.4, 94.5, 98.4 tok/s

Short answer (504 tokens): 93.2 tok/s

The honest shape: ~130-150 tok/s at short context is the ceiling, ~94-105 sustained on longform.

The ceiling matters because that is what people feel in chat.

The old dense Q4_K_M on llama.cpp ran ~37 tok/s on this same card. (No MTP), with MTP about 60 tok/s

Sustained is roughly 2.5-3x. Ceiling is closer to 4x. Same model, different quantization and engine.

The multiplier comes from EXL3, the ExLlamaV2 engine, and DFlash2 together.

CONCURRENCY IS A RTX 4090 LANE.

Just like the old config on the 4090, the 24gb vram, means a long context can only hold one stream, and running concurrency requires to lower context length, because it runs fast it sort of makes up for it by being faster than slower GPU chips.

CONTEXT LADDER

The recipe doc measured prefill. I re-ran it with TTFT separated from decode, because decode is what you actually feel after the first token.

~5k in: decode 140 tok/s (TTFT 0.5s), needle HIT ~18k in: decode 85 tok/s (TTFT 0.3s*), needle HIT ~73k in: decode 28 tok/s (TTFT 1.8s), needle HIT ~146k in: decode 16 tok/s (TTFT 2.3s), needle HIT

(*0.3s at 18k is a prefix-cache hit from the paired pass. Cold prefill for reference: ~2,020 tok/s at 17k falling to ~508 at 153k.)

Needle hit at every depth, mine and the original 7/7. Retrieval is intact at max context. Speed is not: decode falls ~9x from short to max. Past ~50k tokens this stops being a chat tool and becomes a batch tool. At 146k it works, but nobody is typing interactively against 16 tok/s.

WHERE IT BROKE

The model's native context is 262k. The README says DFlash2 fits ~220k on a 24GB card.

My 200,704-token attempt failed with insufficient VRAM. Dropped to 168,960 and it booted.

The real ceiling is somewhere between 168,960 and 200,704 and I never tested that gap.

I jumped to a value that worked and called it done. That is ~32k tokens of context I left on the table.

One thing the numbers taught me: JSON tokenizes at ~1.5 chars/token, prose at ~3.9. "150k tokens of JSON" needs ~2.7x more filler than the same estimate in prose. Size by real tokens, not estimates.

THE UPGRADE

If you own a 4090 and you are running Q4_K_M on llama.cpp, you are leaving a good bit of speed and measurable intelligence on the table.

The same model, on the same card, with a better quantization and engine, goes from 60 tok/s to 130-150 at short context and 94-105 sustained.

The model also thinks closer to the original because EXL3 preserves more of the output distribution per bit than the old rounding method.

The recipe is in the first reply. Everything above came from one 4090 and one afternoon of re-measuring. The decode ladder especially needs independent numbers.

If your card gets different falloff, that is worth knowing.

Recipe and flags/ findings in reply 👇

[Ty @tysn_dev]

Me still running off my sparks in my basement:

[Polymarket @Polymarket]

BREAKING: ChatGPT is currently down for many users.

[Ty @tysn_dev]

@jakeharrisdev Wow great work! I’ll be trying this out this week for sure 🫡

[ashen @ashen_one]

we got muse spark 1.3 and gemini 3.8 flash today aka poop and fart came out today 😭