Cruz

God-fearing husband & father. AI Engineer. Fine-tuning + day-zero EXL3 & GGUF quants + benchmarks • Local LLMs • honest tok/s • fitting big models on small GPUs

[Cruz @ViC305]

🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!

🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳
📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀
💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁!
⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠!
🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟!
🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!)
𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯

And YES, the quantized weights are AVAILABLE NOW! 💚
━━━━━━━━━━━━━━━━━━━━
🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘!

Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.

One thing really stuck with me...

NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.

But there's a problem.

𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦.

That inspired me.

I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.

So I spent the last 72 hours pushing the limits.

And the results genuinely surprised me! 🔥

━━━━━━━━━━━━━━━━━━━━

🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬!

I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.

And when I saw the CODE results, I could barely believe it!

🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘

💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹

✅ 99.08% Top-1 Agreement
✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧!
✅ KL Divergence: 0.0186

🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬

✅ 96.04% Top-1 Agreement
✅ 97.98% Answer-Only Agreement
✅ KL Divergence: 0.0349

🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞

✅ 95.56% Top-1 Agreement
✅ 94.19% Answer-Only Agreement
✅ KL Divergence: 0.0477
📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬
🏆 94.52% Top-1 Agreement
📉 Mean KL: 0.0982
🎯 Median KL: 0.00019

𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯

These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
━━━━━━━━━━━━━━━━━━━━

⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞!

Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.

𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.

My latest vLLM measurements:
⚡ Decode: 35.5 tok/s
🚀 Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...

𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚

And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.

━━━━━━━━━━━━━━━━━━━━

💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦.

I have a simple goal.

𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦.

I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.

I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.

NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.

My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.

Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.

𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚.

━━━━━━━━━━━━━━━━━━━━

🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧!

The token-level fidelity results are incredible, but I'm not stopping there.

I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.

💻 HumanEval+ / MBPP+
🧠 MMLU-Pro STEM / MATH-500
🔬 SciCode / LiveCodeBench
🔧 BFCL / Multi-step tool chains
🌍 MGSM

𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦!

Those task-level results are coming next! 👀

━━━━━━━━━━━━━━━━━━━━

👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥...

@mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯.

Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!

We're putting the finishing touches on it and making sure everything is solid before releasing it.

More on that VERY soon. 😏

━━━━━━━━━━━━━━━━━━━━

🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨!

To @AWERXAI for supporting my work and helping make this contribution to the community possible.

To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.

To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.

To @mr_r0b0t for collaborating with me on what's coming next!

And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.

𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚

🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯
https://t.co/7CY9ByTjMu

🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
https://t.co/MTqzscYuff

𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞.
𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦.
𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.

𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥

[Cruz @ViC305]

@QwenAIFans Thanks for the shout out! Click on the 300x in the upper right hand corner to see it all move fast 🤓

[Cruz @ViC305]

@JuanAG_IT Isn’t that such a neat feature?!

[Cruz @ViC305]

🚨🌍 𝗤𝗪𝗘𝗡𝟰-𝗠𝗔𝗫-𝗣𝗥𝗘𝗩𝗜𝗘𝗪 𝗕𝗨𝗜𝗟𝗧 𝗠𝗘 𝗔 𝗙𝗨𝗟𝗟 𝟯𝗗 𝗘𝗔𝗥𝗧𝗛 𝗠𝗜𝗦𝗦𝗜𝗢𝗡 𝗖𝗢𝗡𝗧𝗥𝗢𝗟 𝗜𝗡 𝗝𝗨𝗦𝗧 𝟭 𝗛𝗢𝗨𝗥 𝗔𝗡𝗗 𝟰𝟳 𝗠𝗜𝗡𝗨𝗧𝗘𝗦! 🤯🚀
Not a mockup. Not a static demo.
𝗔 𝗥𝗘𝗔𝗟, 𝗜𝗡𝗧𝗘𝗥𝗔𝗖𝗧𝗜𝗩𝗘, 𝗣𝗨𝗕𝗟𝗜𝗖𝗟𝗬 𝗗𝗘𝗣𝗟𝗢𝗬𝗘𝗗 𝗔𝗣𝗣𝗟𝗜𝗖𝗔𝗧𝗜𝗢𝗡!
Introducing 𝗢𝗥𝗕𝗜𝗧𝗔𝗟. 🌎

I wanted to see just how far @Alibaba_Qwen Qwen4-Max-Preview could push agentic software engineering using @NousResearch's Hermes as my harness.

So I gave it an ambitious challenge: build a cinematic Earth intelligence platform combining satellite tracking, rocket launches, aircraft visualization and a fully interactive 3D globe.

𝗙𝗥𝗢𝗠 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗢 𝗟𝗜𝗩𝗘 𝗪𝗘𝗕𝗦𝗜𝗧𝗘:

⏱️ Started: 9:23 AM
🚀 Deployed and verified: 11:10 AM
🔥 Total: 𝟭 𝗛𝗢𝗨𝗥 𝟰𝟳 𝗠𝗜𝗡𝗨𝗧𝗘𝗦!
And look at what it built!
━━━━━━━━━━━━━━━━━━

🌎 𝗖𝗜𝗡𝗘𝗠𝗔𝗧𝗜𝗖 𝟯𝗗 𝗘𝗔𝗥𝗧𝗛

🌍 NASA Blue Marble imagery
🌗 Real Sun-driven day/night terminator
🌃 Nighttime city lights
☁️ Rotating clouds and atmosphere
🌙 Sun, Moon and Milky Way
🎥 Cinematic camera controls

🛰️ 𝗟𝗜𝗩𝗘 𝗦𝗔𝗧𝗘𝗟𝗟𝗜𝗧𝗘 𝗧𝗥𝗔𝗖𝗞𝗜𝗡𝗚

📡 589 objects loaded during testing
🚀 Starlink, GPS, ISS, weather and science
🧮 CelesTrak TLEs + SGP4 propagation
📊 Altitude, velocity and orbital telemetry
🎯 Select the ISS and lock the camera onto it!
🚀 𝗥𝗢𝗖𝗞𝗘𝗧 𝗟𝗔𝗨𝗡𝗖𝗛 𝗠𝗜𝗦𝗦𝗜𝗢𝗡 𝗖𝗢𝗡𝗧𝗥𝗢𝗟
⏳ 14 upcoming launches loaded during testing
🚀 SpaceX, ULA, Rocket Lab and more
📍 Launchpads, mission data and countdowns
🌍 Click a launch to fly to its launch site!
✈️ 𝗔𝗜𝗥 𝗧𝗥𝗔𝗙𝗙𝗜𝗖 𝗩𝗜𝗦𝗨𝗔𝗟𝗜𝗭𝗔𝗧𝗜𝗢𝗡
320 simulated aircraft on real airport routes when live OpenSky access is blocked by CORS. Clearly labeled as SIMULATED, not falsely presented as live data.
⚡ Time warp up to 300×
🔎 Global search and constellation filters
📱 Desktop and mobile support
🎛️ Full Mission Control HUD

━━━━━━━━━━━━━━━━━━

💻 𝗧𝗛𝗘 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥𝗜𝗡𝗚 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗥𝗘𝗔𝗟𝗟𝗬 𝗜𝗠𝗣𝗥𝗘𝗦𝗦𝗘𝗗 𝗠𝗘.
During that 1h47m session, the agent worked through:
✅ ~1,700 lines of TypeScript/React
✅ CesiumJS 3D Earth rendering
✅ Satellite orbital propagation
✅ Live API integration and CORS handling
✅ Three build/fix cycles
✅ Desktop and mobile browser QA
✅ ISS follow-camera verification
✅ GitHub repository + Pages deployment
✅ Starlink streaming and 2-hour cache hardening
And it delivered a working, published application with zero console errors in the reported final browser tests!
The stack: React + TypeScript + CesiumJS + Tailwind.

━━━━━━━━━━━━━━━━━━

🧠 𝗪𝗛𝗔𝗧 𝗔𝗕𝗢𝗨𝗧 𝗧𝗢𝗞𝗘𝗡 𝗨𝗦𝗔𝗚𝗘?
The Hermes session ledger recorded approximately:
⚡ 501 API calls
📥 6.07M input tokens
♻️ 51M cache-read tokens
📤 629K output tokens
🧠 381K reasoning tokens within that output
IMPORTANT: Those totals cover the entire shared Hermes session, including earlier CubeLab work. ORBITAL's exact token share wasn't separately logged.

But ORBITAL itself went from request to deployed application in a measured 107 minutes.

━━━━━━━━━━━━━━━━━━

What excites me isn't that an AI made a pretty interface.
It's watching a coding agent coordinate real data sources, 3D graphics, application architecture, debugging, testing and deployment in a single sustained workflow.
𝗙𝗥𝗢𝗠 𝗜𝗗𝗘𝗔 𝗧𝗢 𝗔 𝗪𝗢𝗥𝗞𝗜𝗡𝗚 𝗣𝗥𝗢𝗗𝗨𝗖𝗧 𝗜𝗡 𝗨𝗡𝗗𝗘𝗥 𝗧𝗪𝗢 𝗛𝗢𝗨𝗥𝗦. 🔥
Huge shoutout to @Alibaba_Qwen and @QwenDevs for Qwen4-Max-Preview, and @NousResearch for Hermes!

🌎 𝗧𝗥𝗬 𝗢𝗥𝗕𝗜𝗧𝗔𝗟:
https://t.co/iikGti5Xiu

💻 𝗙𝗨𝗟𝗟 𝗦𝗢𝗨𝗥𝗖𝗘:
https://t.co/Somu6CtXHX

Find the ISS, lock onto it, accelerate time and explore Earth!

𝗧𝗛𝗜𝗦 𝗜𝗦 𝗧𝗛𝗘 𝗔𝗚𝗘𝗡𝗧𝗜𝗖 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥𝗜𝗡𝗚 𝗙𝗨𝗧𝗨𝗥𝗘 𝗜'𝗩𝗘 𝗕𝗘𝗘𝗡 𝗪𝗔𝗜𝗧𝗜𝗡𝗚 𝗙𝗢𝗥. 🚀🌍

[Cruz @ViC305]

@rdevaul Thank you Rich! Much appreciated! 🙏🙏

[Cruz @ViC305]

@jmurillocode Thank you J! 🙏

[Cruz @ViC305]

@Blackfrost_AI Bravo on this model brother! I know youve taken every day since its release last week to get it right!! The community is lucky to have you sir!

[Cruz @ViC305]

@Maciej25956571 Follow me and stay tuned I’m releasing an inference engine for EXL3s with @mr_r0b0t shortly!

[Cruz @ViC305]

🔥 EXL3 IS NOT ONE QUANTIZATION RECIPE. QUALITY MATTERS.

While the discussion focuses on BF16 embeddings and memory overhead, I want to show the community what I’ve achieved with SAGE, my custom fidelity-first, mixed-K EXL3 quantization process.

🚀 Qwen3.8-27B — SAGE-REXL3

4.50 bpw | 17.3 GB
✅ 96.82% top-1 agreement vs BF16
✅ 0.0117 KL divergence

3.64 bpw | 14.7 GB
✅ 95.35% top-1 agreement vs BF16
✅ 0.0321 KL divergence

These are full-pack sizes, not numbers that exclude embeddings.

Both retain BF16 token embeddings and use a quantized EXL3 K6 lm_head.

The difference is how SAGE allocates precision across the model to preserve fidelity.

These are measured teacher-forced, full-vocabulary fidelity results against the original BF16 model, not downstream benchmark scores.

The format doesn’t determine the quality. The quantization methodology matters.

I’d love to see @UnslothAI benchmark both SAGE variants against other formats under identical conditions.

Let’s compare real quality at real memory footprints. 🔬

🔗 4.50 bpw:
https://t.co/U6MvNq63Qi

🔗 3.64 bpw:
https://t.co/eDbqgUQqh7

[Unsloth AI @UnslothAI]

Today, we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more. We’re working on even better version for Qwen4

Run on 12GB RAM.

GGUF: https://t.co/xIdNwm7CLQ
Blog: https://t.co/J2PwgMP6GZ

[Cruz @ViC305]

Appreciate the transparency around actual memory usage. I agree that full disk and RAM footprints should be included in any fair quantization comparison.
However, I think there’s an important distinction being overlooked: not all EXL3 quantizations use the same methodology or achieve the same fidelity.
I’ve developed SAGE, a custom tensor-wise mixed-K quantization process for EXL3, and recently published two Qwen3.8-27B variants:
SAGE-REXL3 4.50 bpw
17.3 GB total pack size (~16.1 GiB resident weights)
96.82% top-1 agreement vs BF16
0.0117 mean KL divergence
SAGE-REXL3 3.64 bpw
14.7 GB total pack size (~13.7 GiB resident weights)
95.35% top-1 agreement vs BF16
0.0321 mean KL divergence
These are teacher-forced, full-vocabulary fidelity measurements on held-out sequences against the original BF16 model, not downstream benchmark scores.
For clarity, both releases retain the 2.54 GB BF16 token embedding table, while the lm_head uses EXL3 K6 quantization.
So I’m not claiming to have eliminated BF16 embeddings. What I am demonstrating is that precision allocation throughout the remaining model matters significantly.
I’d also be interested in seeing experimental evidence for the hypothesis that BF16 embeddings absorb quantization damage from the output head. That’s an interesting theory, but it needs an actual ablation.
I’d love to see Unsloth include both SAGE variants in your benchmarks, using identical evaluation conditions and comparing actual total memory footprint against accuracy.
EXL3 is a quantization format, not a single fixed quantization recipe. Comparing one EXL3 implementation shouldn’t automatically characterize the entire format.
Let’s put different quantization methodologies through the same tests and see what the data says.
Models:
4.50 bpw:
https://t.co/U6MvNq63Qi

3.64 bpw:
https://t.co/eDbqgUQqh7

https://t.co/4mlypNFFa3

[Cruz @ViC305]

🎁🔥 𝗤𝗪𝗘𝗡𝟰 𝗛𝗔𝗗 𝗠𝗘 𝗪𝗔𝗬 𝗧𝗢𝗢 𝗘𝗫𝗖𝗜𝗧𝗘𝗗… 𝗔𝗡𝗗 𝗧𝗛𝗘 𝗪𝗔𝗜𝗧 𝗟𝗘𝗧 𝗠𝗘 𝗗𝗢𝗪𝗡!

So I decided to give the community a little gift instead! 🎁

I ran Qwen3.8-27B through my NEWEST SAGE-REXL3 quantization process to see just how small I could push it without sacrificing quality.

𝗔𝗡𝗗 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦?! 🤯

🔥 𝗟𝗔𝗥𝗚𝗘𝗥 — 𝟭𝟲.𝟬𝟴 𝗚𝗶𝗕
🏆 96.82% Top-1 agreement
🔬 0.0117 KL | 4.50 BPW

⚡ 𝗦𝗠𝗔𝗟𝗟𝗘𝗥 — 𝟭𝟯.𝟲𝟳 𝗚𝗶𝗕
🏆 95.35% Top-1 agreement
🔬 0.0321 KL | 3.64 BPW

Both measured against the original model!

And the smaller version fits a 24GB RTX 4090 with 262K Q4 KV cache, Prefix Cache AND DFlash2! 🚀

𝗦𝗢𝗠𝗘 𝗢𝗙 𝗧𝗛𝗘 𝗛𝗜𝗚𝗛𝗘𝗦𝗧-𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗤𝗨𝗔𝗡𝗧𝗦 𝗬𝗢𝗨’𝗟𝗟 𝗙𝗜𝗡𝗗 𝗔𝗧 𝗧𝗛𝗘𝗦𝗘 𝗦𝗜𝗭𝗘𝗦!

🤗 16 GiB version:
https://t.co/U6MvNq63Qi

🤗 13.7 GiB version:
https://t.co/eDbqgUQqh7

⚡ Quantized DFlash2 drafts:
https://t.co/17rO2qCCwm

𝗤𝗪𝗘𝗡𝟰 𝗖𝗔𝗡 𝗧𝗔𝗞𝗘 𝗜𝗧𝗦 𝗧𝗜𝗠𝗘… 𝗜’𝗠 𝗦𝗧𝗜𝗟𝗟 𝗖𝗢𝗢𝗞𝗜𝗡𝗚! 🔥

[Cruz @ViC305]

@u1tra_instinct @SamJWasserman @NousResearch @Teknium Phenomenal work boys!!

[Cruz @ViC305]

@u1tra_instinct @mr_r0b0t @NousResearch Thanks Keys! I worked hard on this one for the past few weeks and it was perfect timing for the hackathon 😎

[Cruz @ViC305]

Ever wanted a simple, quick way to sync your Hermes agents/bots across devices?

Introducing Hermes Roam, my @NousResearch Hermes Agent hackathon entry, now in the official Hermes plugin catalog!

One push, one pull, and your SOUL, memories, skills and config follow you to every machine. Everything is encrypted and stored on storage you own: an S3/R2 bucket or a private git repo. No service in the middle.

Main features:
• Secrets stay local: API keys and .env files never leave the machine
• No silent overwrites: an out-of-date push is refused until you pull
• Every version is kept and restorable
• Multi-profile: all your bots travel together
• Optional encrypted, read-only archive of your chat sessions
• Hands-free auto sync: push when a session ends, pull when the next one starts
• Linux, macOS and Windows

The video is a real run across my VPS and my Windows laptop 👇

hermes plugins install vcruz305/hermes-roam
https://t.co/FnEndLmKd9

https://t.co/qBoSwPEYbT

[Cruz @ViC305]

@Alibaba_Qwen Super excited to test this out!!

[Cruz @ViC305]

Woahhh Qwen4-Max Preview is now available on API!! 🚀🚀 I knew we would get something soon!! Going to give this new Max model a go for sure!! 🔥

[Qwen @Alibaba_Qwen]

Qwen4 is coming.

Deeper knowledge. Richer world understanding. Stronger coding and agentic capabilities.

Built for real-world tasks — from complex coding and knowledge work to front-end builds and professional software workflows.

Qwen4-Max is now in preview on Alibaba’s Token Plan. You can also get early access to the new model on QwenWork, Qoder, Qwen APP, and QwenStudio.

Put it to the test on a real problem. Show us what you build. 👀

Token Plan: https://t.co/LOB2Jdt3Sk

[Cruz @ViC305]

@NeoAIForecast @NousResearch Thank you my brother!!

[Cruz @ViC305]

@mr_r0b0t @NousResearch

[Cruz @ViC305]

@wikiwayne @NousResearch Thank you Wayne!

[Cruz @ViC305]

@witcheer @NousResearch Appreciate you!!

[Cruz @ViC305]

@witcheer @Teknium 👀

[Cruz @ViC305]

Fair concern, but tell me you haven’t watched the demo video without telling me you didn’t watch it as that's not how it works lol. Roam never syncs .env, auth.json or any credential file, and secret-looking keys are stripped from config.yaml. API keys never leave the machine (the demo shows the VPS key appearing 0 times on the laptop).

What does sync (SOUL, memories, skills) is encrypted on your machine with AES-256-GCM before upload, to a bucket or git repo you own. No third-party service ever sees it.

It also passed Nous's security review to get into the official plugin catalog.