Eugenious 👨🏿‍💻

Sage | Software Savant | Azure | Intellectual Athlete | Invested | 🇳🇬 | Taurus | Technologist | DevOps | Martial 🎨 | root |

[Eugenious 👨🏿‍💻 @Akademic_2]

Side quest ☑️

[Memphis Jookin @MemphisJookin]

To clear up any confusion…. Here’s the REAL version of that AI Jookin video that keeps goin viral everywhere 😂🔥

[Eugenious 👨🏿‍💻 @Akademic_2]

nvtop

[Eugenious 👨🏿‍💻 @Akademic_2]

Google & AWS Veteran: What Top Tier Software Architects Do Differently https://t.co/uhZ5K6snb6 via @YouTube

[Eugenious 👨🏿‍💻 @Akademic_2]

@justjalen It will be an interesting transition.

[Eugenious 👨🏿‍💻 @Akademic_2]

We will be forced to redefine where we gain our meaning. When “work” is no longer a necessity but a choice.

[Eugenious 👨🏿‍💻 @Akademic_2]

“Adaptation out of context”

[Eugenious 👨🏿‍💻 @Akademic_2]

Also kudos to @OpenWebUI, impressed with the built features. The built in RAG (knowledge) is a nice touch.

[Eugenious 👨🏿‍💻 @Akademic_2]

Open source LLM’s are good for humanity. No one entity should hold all this power.

[Eugenious 👨🏿‍💻 @Akademic_2]

Code
Fight
Dance
Music

Basically it’s all engineering.

[Eugenious 👨🏿‍💻 @Akademic_2]

Multigenerational play.

My son benefits from the efforts of his ancestors.

[Eugenious 👨🏿‍💻 @Akademic_2]

Claude Code > Open Claw

For local models imo

[Eugenious 👨🏿‍💻 @Akademic_2]

The pen is nice

[Eugenious 👨🏿‍💻 @Akademic_2]

Excellence takes on many forms, and I’m rooting for all of them.

[Eugenious 👨🏿‍💻 @Akademic_2]

Respect a good plan and actualization.

[Eugenious 👨🏿‍💻 @Akademic_2]

This is a solid overview of designing distributed systems for scale.

[The Tech Prophet (Amospikins) @Amospikins]

Why Your App Works for 10 Users but Dies for 10,000.

[Ahmad @TheAhmadOsman]

You don’t pick an Inference Engine

You pick a Hardware Strategy
and the Engine follows

Inference Engines Breakdown
(Cheat Sheet at the bottom)

> llama.cpp
runs anywhere
CPU, GPU, Mac, weird edge boxes
best when VRAM is tight and RAM is plenty
hybrid offload, GGUF, ultimate portability
not built for serious multi-node scale

> MLX
Apple Silicon weapon
unified memory = “fits” bigger models
than VRAM would allow but also slower than GPUs
clean dev stack (Python/Swift/C++)
sits on Metal (and expanding beyond)
now supports CUDA + distributed too
great for Mac-first workflows, not prod serving

> ExLlamaV2
single RTX box go brrr
EXL2 quant, fast local inference
perfect for 1/2/3/4 GPU(s) setups (4090/3090)
not meant for clusters or non-CUDA

> ExLlamaV3
same idea, but bigger ambition
multi-GPU, MoE, EXL3 quant
consumer rigs pretending to be datacenters
still CUDA-first, still rough edges depending on model

> vLLM
default answer for prod serving
continuous batching, KV cache magic
tensor / pipeline / data parallel
runs on CUDA + ROCm (and some CPUs)
this is your “serve 100s of users” engine

> SGLang
vLLM but more systems-brained
routing, disaggregation, long-context scaling
expert parallel for MoE
built for ugly workloads at scale
lives on top of CUDA / ROCm clusters
this is infra nerd territory

> TensorRT-LLM
maximum NVIDIA performance
FP8/FP4, CUDA graphs, insane throughput
multi-node, multi-GPU, fully optimized
pure CUDA stack, zero portability

(And underneath all of it:
Transformers → model architecture
layer → CUDA / ROCm / TT-Metal
→ compute layer)

What actually happens under the hood:

> Transformers defines the model
> CUDA / ROCm executes it
> TT-Metal (if you’re insane) lets you write the kernel yourself

The Inference Engine is just
the orchestrator (simplified)

When running LLMs locally,
the bottleneck isn’t just “VRAM size”

It isn’t even the model

It’s:
- memory bandwidth (the real limiter)
- KV cache (explodes with long context)
- interconnect (PCIe vs NVLink vs RDMA)
- scheduler quality (batching + engine design)
- runtime overhead (activations, graphs, etc)

(and your compute stack decides all of this)

P.S. Unified Memory is way slower than VRAM

Cheat Sheet / Rules of Thumb

> laptop / edge / weird hardware → llama.cpp
> Mac workflows → MLX
> 1–4 RTX GPUs → ExLlamaV2/V3
> general serving → vLLM
> complex infra / long context / MoE → SGLang
> NVIDIA max performance → TensorRT-LLM

[Eugenious 👨🏿‍💻 @Akademic_2]

Setting up my local AI setup at home. Currently using a spare laptop running Proxmox, RHEL, Ollama.

Need to swap out the laptop for hardware that can support larger models (VRAM constraint).

Recommendations?

[Eugenious 👨🏿‍💻 @Akademic_2]

Cash Does Not Rule Everything Around Me

[Zain Shah @zan2434]

Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see.

@eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)

[Eugenious 👨🏿‍💻 @Akademic_2]

Order 66