Local Inference Lab 🧪

We are a local AI-focused lab researching and developing optimized performance for vLLM on SM12x hardware for open models.

[Local Inference Lab 🧪 @YourLocalAILab]

With today's release of Jovian Judgement R37, DeepSeek V4.1 Flash has full support with blazing speeds.

Decodes of 500 tok/s and prefills of 20k tok/s for single stream workloads on 4 RTX 6000 Pros. ⚡️

To learn more point your agent at our GitHub here: https://t.co/uNojQBFIZR

[Local Inference Lab 🧪 @YourLocalAILab]

This is what we love to see! 🦾

[Harrison Kinsley @Sentdex]

@Sd_hector omp, vllm, and some rtx pro 6ks w/ the help of @YourLocalAILab recipes to make the models go fast!

[Local Inference Lab 🧪 @YourLocalAILab]

Thank you for all the support! Our first official post about our work went pretty far!

Stay tuned. We have a lot more to share with you all that will help improve your local AI experience.

We're just getting started. 🦾

[Local Inference Lab 🧪 @YourLocalAILab]

With today's release of Jovian Judgement R37, DeepSeek V4.1 Flash has full support with blazing speeds.

Decodes of 500 tok/s and prefills of 20k tok/s for single stream workloads on 4 RTX 6000 Pros. ⚡️

To learn more point your agent at our GitHub here: https://t.co/uNojQBFIZR

[Local Inference Lab 🧪 @YourLocalAILab]

@djnmrjnvc Yes you could have four GPUs in a TP=4 vLLM config and the 5th one for a smaller model, or even video / image generation.

[Local Inference Lab 🧪 @YourLocalAILab]

With today's release of Jovian Judgement R37, DeepSeek V4.1 Flash has full support with blazing speeds.

Decodes of 500 tok/s and prefills of 20k tok/s for single stream workloads on 4 RTX 6000 Pros. ⚡️

To learn more point your agent at our GitHub here: https://t.co/uNojQBFIZR

[Local Inference Lab 🧪 @YourLocalAILab]

@sam_sonivalabs We do have that setup with this container. You can use it on Qwen Flash Next.

[Local Inference Lab 🧪 @YourLocalAILab]

@adriancantrill Qwen Flash Next is great for 2x RTX 6Ks!

[Local Inference Lab 🧪 @YourLocalAILab]

@Mayhem4Markets 🙌

[Brandon Music @BrandonMusicKy]

Great work from the team @YourLocalAILab . Highest numbers, rigorous, and empirically tested.

[am.will @LLMJunky]

500 toks/s at home 🤯

The thing that bothers me the most about local AI, and too few talk about this, is the fact I've already dropped obscene amounts of money and I just simply don't have enough vram.

I have two RTX 6000 pros and it's pretty rare I can find a model that I actually want to run.

It seems like four is the minimum honestly. This is gonna end up costing me over $30,000 more.

It's true that smaller models are getting way better, and I'm not discounting these relatively tiny models

But at the end of the day when there is a significantly better larger model, you're always going to want to run that.

I'm getting serious fomo.

Anyway, these guys are cracked.

[Local Inference Lab 🧪 @YourLocalAILab]

@AI_with_Eric 😂😂

[Local Inference Lab 🧪 @YourLocalAILab]

@HealthRanger Thank you! 🫡

[Local Inference Lab 🧪 @YourLocalAILab]

@Mayhem4Markets 🙌

[Local Inference Lab 🧪 @YourLocalAILab]

Be sure to tune in!

[Markets & Mayhem @Mayhem4Markets]

It's Friday already?! 😱

Time flies when everything is accelerating.

We're talking about AI security, the latest in local and much more.

Panel: @philipbankier @LLMJunky @secemp9

Guests: @ZackKorman @BrandonMusicKy @msuiche

Join us at 4:00 pm ET!

https://t.co/a21BggLohK

[Harrison Kinsley @Sentdex]

watching this model's release while one of my rtx pro 6ks is dead is painful! Looks very good!

[Brandon Music @BrandonMusicKy]

Now trying to get deepseek v4.1flash to run on my rtx pro 6ks. Really excited to try this one. Following a deliberate process to make sure there are no max logit error and zero relative L2., and cosine similarity is appropriate. Running in eager. Nowtime for graphs and MTP lol

[Markets & Mayhem @Mayhem4Markets]

DeepSeek-V4.1 Flash is pushing KV cache compression to levels once thought impossible.

This is actually insane.

And I love it.

[Local Inference Lab 🧪 @YourLocalAILab]

This is an excellent read on why more companies are investing in local AI, but not to leave the cloud. Instead to change their relationship with the cloud.

[Markets & Mayhem @Mayhem4Markets]

The shift to more local AI is real, but the destination is more of a hybrid approach than abandoning the cloud entirely. https://t.co/T2WbB78xRS

[Local Inference Lab 🧪 @YourLocalAILab]

Great article!

[Brandon Music @BrandonMusicKy]

https://t.co/pulg93Q8Gl

[Local Inference Lab 🧪 @YourLocalAILab]

@Mayhem4Markets 🙌