opensource.wtf

Opensource software is changing and I am build cool opensource projects for where its going.

[opensource.wtf @OpensourceWtf]

@Tono_Ken3 Seems more like an experiment than a practical model since Q3.6 27b outbenches it.

[opensource.wtf @OpensourceWtf]

@_imdawon Maybe wait for the next Mistral drop since they said those were trained for math instead of coding so maybe they'll research better.

[opensource.wtf @OpensourceWtf]

@MiaAI_lab 🤣

[opensource.wtf @OpensourceWtf]

@Tech2Wild 35B also requires more memory. The only advantage it has is responsiveness.

[opensource.wtf @OpensourceWtf]

@mjkzys @ItsmeAjayKV Have you had any luck using 4b in workflows?

[opensource.wtf @OpensourceWtf]

@ItsmeAjayKV Well you could go back to the ooga booga days of having a bash script run your flow and manually loop it with Ralph

[opensource.wtf @OpensourceWtf]

We need to get open source models to be one click install.

[Ahmad @TheAhmadOsman]

Local AI was never this EASY

> Install ODS
> Let it detect your hardware
> It will download the best model for your hardware
> And then start local inference and Open WebUI for you

With ODS, you can

> Add voice, agents like Hermes, workflows, RAG, search, image generation, and more
> Manage the whole stack from one dashboard

Now your PC, Mac, or Linux box is a private AI server

No cloud required
No subscription required
Your prompts and data stay on your machine unless you choose otherwise

We're gonna make Local AI The Default

[opensource.wtf @OpensourceWtf]

@TheAhmadOsman More tools like this to help out newbies.

[opensource.wtf @OpensourceWtf]

@Rose__Celestial @Youssofal_ @ivanfioravanti There's no quality loss. It just tries to precompute what the next likely tokens are. Your CPU does the same thing when programs hit an if statement, it evaluates both at the same time to speed things up.

[opensource.wtf @OpensourceWtf]

@M_Chimiste @Tech2Wild Yeah its like 2020 when I sold my 4 year old mining GPUs in profit.

[opensource.wtf @OpensourceWtf]

@M_Chimiste @Tech2Wild I would not recommend training on a spark either but at that point what advantage does it have if you're just looking for memory bandwidth to push tokens? You can probably get over 40 tokens per second on Qwen 3.6 27b on that M4 compared to less than 20 on the Spark.

[opensource.wtf @OpensourceWtf]

@Tech2Wild @M_Chimiste 2) From my experience in GPU mining, you need to factor in resale. You want to target the highest end of consumer grade the hardware because that has the best secondary market. I rather have a Mac that will hold its value so I can flip for the next platform if no training.

[opensource.wtf @OpensourceWtf]

@Tech2Wild @M_Chimiste There's two issues with this analysis:

1) You're comparing total ram when you need to factor in bandwidth. I personally do the math against the 128gb MBP which for 5k has around twice the memory bandwidth of 5k worth of DGX Spark. If you only care about inference then M.

[opensource.wtf @OpensourceWtf]

I'm still pissed off no one's fixed the trademark and IP loop hole yet that we have to have another extension. Good job, @Shpigford!

[Josh Pigford @Shpigford]

Knockoff is now live!

Filter out the knockoff crap brands on Amazon.

Sorry to brands like WNPETHOME, EHEYCIGA, YXYL, LU&MN, JOYIN, TOMY, GODONLIF, YOOJEE, LINGTENG, LANEIGE, VISCOO, BIODANCE, COOFANDY, BALENNZ, TOSY and LUENX.

https://t.co/9mLk0EAsfG

[opensource.wtf @OpensourceWtf]

@Youssofal_ I benchmark an average of 60-68 tps on realistic work loads.

[opensource.wtf @OpensourceWtf]

@Youssofal_ Great work, upgrading to it now.

[opensource.wtf @OpensourceWtf]

@0xSero I think its worth it for high level design tasks because it feels like they sacrificed the conversational experience with opus with it getting so much less readable between 6 and 8. Good for pair programming, totally not worth it for doing long running tasks.

[opensource.wtf @OpensourceWtf]

The barriers to local open LLMs is coming down every week.

[Alok @analogalok]

I can't afford a $2,000 GPU is officially a dead excuse.

yesterday I showed you how to unlock an enterprise grade 16GB NVIDIA GPU for $0.

your immediate response? "How do I run BIGGER models?" "How do I push massive context windows without OOM crashes?"

Fine. Let’s double the compute.

Today, we are securing a Dual GPU cluster with 32GB of VRAM for free.

At standard cloud rates (~$0.40/hr for dual cards), Kaggle is handing you roughly $12 of free, heavy duty GPU compute every single month.

And instead of compiling C++ from source like yesterday, we are dropping in pre built CUDA binaries. You can have a massive 26 Billion parameter model running in literally under 2 minutes on this free ubuntu instance.

The Specs you are getting for $0:

- 2x NVIDIA Tesla T4 GPUs (turing architecture)
- 32GB GDDR6 VRAM Total
- 5120 CUDA Cores
- 30 GBs of Massive RAM
- 60 GBs of Storage
- Optional single p100 (pascal architecture, 16gb vram)

But here is the engineering reality: Throwing multiple GPUs at a Local LLM doesn't magically make it faster. If you don't understand how data moves across a motherboard, you will completely choke your inference engine.

Here is a 60 second masterclass on multi GPU parallelism in llama.cpp for you to experiemtn.

When you split a model across two GPUs, you have two choices:

1. Tensor Parallelism (Flag: -sm tensor) This mathematically slices every single neural layer in half. Both GPUs compute simultaneously.
Because each card only holds half the equation, they must constantly synchronize their math for every single token.

2. Pipeline Parallelism (Flag: -sm layer) This is the default. It chops the model sequentially. GPU 0 gets layers 1-20, GPU 1 gets layers 21-40.
It acts like a factory assembly line. GPU 0 processes the prompt, hands the data over the PCIe lane exactly once, and goes idle while GPU 1 finishes the math.

Benchmark on both and post your numbers!

In today's tutorial, I’ve built a clean Python dictionary configuration block so you can seamlessly experiment with the heavy hitting flags:

-ngl 99 (Offload all layers to both GPUs)
-c 250000 (Push a massive 250k context window)

With 32GB of VRAM and Pipeline Parallelism, you can easily load DeepMind's Gemma 4 26B A4B MoE, ingest entire codebases into the prompt, and never see an Out of Memory core dump. I could easily manage 62 tokens/sec decode without any additional optimizations.

No compilation time. No credit card. Zero excuses.

Bookmark this post right now so you don't lose this multi GPU configuration workflow. Don't forget to share it so it reaches every GPU poor person struggling with compute.

I’ve put together a brand new, cell by cell Kaggle notebook specifically for this Dual T4 setup.

The link to the free Notebook is in the comments below. Let me know what massive models you're spinning up today.

[Alok @analogalok]

I can't afford a $2,000 GPU is officially a dead excuse.

yesterday I showed you how to unlock an enterprise grade 16GB NVIDIA GPU for $0.

your immediate response? "How do I run BIGGER models?" "How do I push massive context windows without OOM crashes?"

Fine. Let’s double the compute.

Today, we are securing a Dual GPU cluster with 32GB of VRAM for free.

At standard cloud rates (~$0.40/hr for dual cards), Kaggle is handing you roughly $12 of free, heavy duty GPU compute every single month.

And instead of compiling C++ from source like yesterday, we are dropping in pre built CUDA binaries. You can have a massive 26 Billion parameter model running in literally under 2 minutes on this free ubuntu instance.

The Specs you are getting for $0:

- 2x NVIDIA Tesla T4 GPUs (turing architecture)
- 32GB GDDR6 VRAM Total
- 5120 CUDA Cores
- 30 GBs of Massive RAM
- 60 GBs of Storage
- Optional single p100 (pascal architecture, 16gb vram)

But here is the engineering reality: Throwing multiple GPUs at a Local LLM doesn't magically make it faster. If you don't understand how data moves across a motherboard, you will completely choke your inference engine.

Here is a 60 second masterclass on multi GPU parallelism in llama.cpp for you to experiemtn.

When you split a model across two GPUs, you have two choices:

1. Tensor Parallelism (Flag: -sm tensor) This mathematically slices every single neural layer in half. Both GPUs compute simultaneously.
Because each card only holds half the equation, they must constantly synchronize their math for every single token.

2. Pipeline Parallelism (Flag: -sm layer) This is the default. It chops the model sequentially. GPU 0 gets layers 1-20, GPU 1 gets layers 21-40.
It acts like a factory assembly line. GPU 0 processes the prompt, hands the data over the PCIe lane exactly once, and goes idle while GPU 1 finishes the math.

Benchmark on both and post your numbers!

In today's tutorial, I’ve built a clean Python dictionary configuration block so you can seamlessly experiment with the heavy hitting flags:

-ngl 99 (Offload all layers to both GPUs)
-c 250000 (Push a massive 250k context window)

With 32GB of VRAM and Pipeline Parallelism, you can easily load DeepMind's Gemma 4 26B A4B MoE, ingest entire codebases into the prompt, and never see an Out of Memory core dump. I could easily manage 62 tokens/sec decode without any additional optimizations.

No compilation time. No credit card. Zero excuses.

Bookmark this post right now so you don't lose this multi GPU configuration workflow. Don't forget to share it so it reaches every GPU poor person struggling with compute.

I’ve put together a brand new, cell by cell Kaggle notebook specifically for this Dual T4 setup.

The link to the free Notebook is in the comments below. Let me know what massive models you're spinning up today.