Alec Mackanic

local-first AI

[kache @yacineMTB]

Our models are way over parameterized, our learning algorithms are still incredibly slow, our machines are too power hungry, and our understanding of intelligence is too shallow. The path goes much, much further. It terrifies me how early we are

[Alec Mackanic @alecmack]

5.6 is cooking through my backlog, sure it’s slow, but it one shots any prompt that has enough context. Plus ui work is actually useable in prod. Unreal leap. Its ability to steer through logic is incredible.

[Alec Mackanic @alecmack]

"Fix this code" was the administration's reason for disabling Fable.
It's not a new concept that those in charge try to control what they know nothing about.
The seemingly ethical frontier labs are currently the best defense against bad actors, and yet the government is constraining them.

Again reemphasizing the urge for open source AI, developed by and maintained by those with humanity's best intentions.

[SuperGemma @0xSUPERGEMMA]

The official website of the Supergemma Foundation is now live 🚀

https://t.co/DYRiy8SETh

The detailed roadmap is also ready and will be available on June 2nd at 9:00 AM EST.

Thank you for waiting so long.

This is just the beginning of long journey.

[Alec Mackanic @alecmack]

We have to pump these owner numbers up. Closed-source AI's main priority are their investors (hence the absurd valuations). They will always push the boundary of what the users are willing to pay and what to actual cost is (maximize margins). These valuations are driven by what insiders see as the most profitable endeavor in history.

Inference will drop eventually, right now large labs are bloating models since they continue to see benchmark improvements, but these large models are incredibly inefficient from an architecture/transformer standpoint.

If open-source can stay close, it forces these labs to keep prices low. This in turn helps open-source use their leading models to further drive local development to stay in the race.

There is a reality where every person can have a true assistant(s) running locally, completely private, with indiscernible capabilities from a human. Keep pushing.

[Sandro @pupposandro]

Nvidia is the most-used hardware for Local AI

[Alec Mackanic @alecmack]

Based on current performance, GPT-6, Claude 5, and Gemini 4 will be agi for 99% of the population.

[wd 🇵🇹🔺 @populartourist]

In a dream nearby

[Alec Mackanic @alecmack]

Cannot stand the doubters - it’s a combination of the entitled and those who have always been “better”, and cannot fathom that rocks and electrons can render their “talents” useless.

Maybe, just maybe, it was never that impressive 🤔

They hate the notion that EIQ and social aptitude will genuinely win out over all.

[Alec Mackanic @alecmack]

Why isn’t every relevant world power combining the brightest to solve UBI and economic prosperity ?

The system is not capable of handling 60+ % unemployment

[Alec Mackanic @alecmack]

Inbound massive week for AI

All leaders are trying to pull away to keep vc cash flowing

Thankfully open source is on their heels.

AGI will genuinely be general meaning for all - it already works 2+ hours straight for you if you know what you’re doing.

And pretty soon u won’t even have to know what you’re doing.

[Alec Mackanic @alecmack]

Who's got the timeline for an open model release that has great fundamental programming knowledge/capabilities, basic logic, and average research ability (<24GB)?
Feel like with the right context, an agent should be able to iteratively work through a codebase, fill its own knowledge gaps through internet searches, and validate its work through the results (reminds you like how every other programmer used to work, right??).
Yeah I know models like these exist, but they continue to fail in seemingly simple places.

Are we waiting on better transformer architecture? Compression algorithms? The tooling is there, seems like the models are still lacking.
I'm sick of relying on usage rates that change every day.

[Alec Mackanic @alecmack]

Testing local 27B inference on a single RTX 5090:

HauhauCS Qwen3.6 27B Uncensored Q6_K_P GGUF + Qwen3.6 DFlash
draft on latest Luce DFlash/DDTree.

OpenAI-compatible server, TQ3 KV, FA window 2048, DDTree
budget 16.

Results:

- 900-token runs avg ~179 tok/s
- Best run ~195 tok/s
- 13.4K-token prompt completed in 9.4s end-to-end
Exploring different harnesses and prefill improvements next.

[Alec Mackanic @alecmack]

Testing local 27B inference on a single RTX 5090:

HauhauCS Qwen3.6 27B Uncensored Q6_K_P GGUF + Qwen3.6 DFlash
draft on latest Luce DFlash/DDTree.

OpenAI-compatible server, TQ3 KV, FA window 2048, DDTree
budget 16.

Results:

- 900-token runs avg ~179 tok/s
- Best run ~195 tok/s
- 13.4K-token prompt completed in 9.4s end-to-end
Exploring different harnesses and prefill improvements next.

[Alec Mackanic @alecmack]

Tested Qwen3.6-27B locally on RTX 5090 with Luce DFlash vs ik_llama.cpp speculative decoding.

Setup:

Target: Qwen3.6-27B Q4_K_M GGUF
GPU: RTX 5090
Runtime: WSL2
Server: OpenAI-compatible /v1/chat/completions

DFlash results:

Standalone binary: 512 tokens in 3.968s — 129.05 tok/s
Acceptance: 512 / 1360
Avg commit/step: 6.02
API path: ~59–73 tok/s depending on request length
Streaming: ~60 tok/s, first token ~0.54s
1024-token API run: 72.68 tok/s

ik_llama.cpp results:

Qwen3-1.7B draft: ~30 tok/s, ~11.8% acceptance
Qwen3.5-2B draft: ~15–17 tok/s, ~1.8–2.6% acceptance

Couldn’t reproduce the 154 tok/s ik_llama claim on my setup.

[Alec Mackanic @alecmack]

Got Qwen3.6-27B running locally on RTX 5090 via Luce DFlash in WSL2.

Setup:

Qwen3.6-27B Q4_K_M GGUF target
Qwen3.5-27B DFlash draft
OpenAI-compatible server at localhost:8000/v1

Results:

Short default no-thinking request: 88 tokens in 0.630s - 139.78 tok/s
Explicit thinking enabled: 256 tokens in 2.090s - 122.48 tok/s
Long run: 4,096 tokens in 49.042s - 83.52 tok/s sustained
Long-run acceptance: 4,096 / 15,872 = 25.8%
Avg commit/step: 4.13 tokens

Next test: ik_llama.cpp + speculative decoding with Qwen3-1.7B as the draft. I’m seeing Qwen3.6-27B Q4_K_M jumping from ~26 tok/s to ~154 tok/s on a 4090 with ~85% acceptance. Curious how it scales on a 5090.