HOARK

Founder @ (TCG marketplace) | Built Wave Density Attention (teacher-free AI primitive) on a 3090 | Seeking compute/collabs for AI + product

[HOARK @HOARK_]

I open-sourced Wave-Density Attention, a new causal attention mechanism that replaces dot-product similarity with wave interference.

Instead of QKᵀ, attention emerges from superposed sinusoidal masks whose interference patterns form dynamic density fields. A sparse Mixture-of-Masks (MOM) router activates only a small subset per token.

Result: a 130M model trained from scratch hits ~18 perplexity on instruction data — on a single GPU.

This isn’t an optimization of transformers — it’s a different primitive for attention.

Code: https://t.co/cj2qlydc07
Model: https://t.co/dQiHLOb965
Try it. Break it. Improve it.
#MachineLearning #DeepLearning #OpenSource
@karpathy @huggingface @arankomatsuzaki @tri_dao @jimmyba

[HOARK @HOARK_]

@brandonjcarl @SwaymaKdotAI before spacexai bought them now they use grok and they already anounced its going to be a 2t model in 2ish weeks not cursor

[HOARK @HOARK_]

@Badtheorylabs I feel ya... i wish i had more compute my 3090's putting in the work.

[HOARK @HOARK_]

@Badtheorylabs so 2 bit quant is awesome but what if we can train sub 2 bit without collapse and no scaffolding from scratch. most systems require full 8-16 bit training and they quant after. mine does not. my system can also train sub bit.

[HOARK @HOARK_]

@thsottiaux fine tuneable lora adapter for luna.

[HOARK @HOARK_]

@NVIDIAGeForce I need a new pc #RTXPowersPlay

[HOARK @HOARK_]

@1kartikkabadi1 settings -> configuration -> avalible reasoning efforts
thank me later.

[HOARK @HOARK_]

@CastAsHuman moneys if you want to use your limits faster go ahead

[Escha Labs @Eschalabs]

Today we are introducing Escha-W2 quantization.

A 2-bit Qwen3.6-35B-A3B model built for fast, local inference. The complete model is 12.3GB on disk—small, enough to run on a single consumer GPU — while averaging ~100% of FP8 performance across 12 benchmarks, including:

MMLU-Pro: 80.9
MATH-500: 93.8
GPQA-Diamond: 77.8
LiveCodeBench v6: 62.6
BFCL tool use: 88.9
RULER 8K–128K: 89.9
Commonsense-6: 76.1

On a single RTX 4090, the model runs:
225 tok/s single-stream generation
on 12.3GB on-disk model size

Compressing a 35B MoE model this far without collapsing its capabilities required more than a standard quantization pass.

We built an end-to-end compression system combining state-of-the-art low-bit quantization with model-aware fine tuning and recovery to preserve capabilities most vulnerable to low-bit error.

Quantizing the Qwen 3.6 35B model - from the base model to the final deployable checkpoint — took approximately 10 hours to complete.

Escha-W2 runs through a custom Qwen3-MoE runtime, which includes the weight loader, low-bit decoding kernels and serving integration required to execute the Escha format efficiently. The runtime currently supports SGLang and ZML deployment. vLLM and llama.cpp will be supported in the next release.

No retraining from scratch. No specialized accelerator. One consumer GPU.

Model download: https://t.co/hbO3VuoDGU
Runtime download: https://t.co/hrTpO4FvuA
Apache-2.0 model and runtime.

[HOARK @HOARK_]

@Eschalabs is this post trained similar to bonsai is and is it q2 end to end?

[HOARK @HOARK_]

@elonmusk no now its to run it through chat gpt and have no voice at all
Why this matters: ...

[HOARK @HOARK_]

@elonmusk I would love to work with you to bring this to life with a distilled grok mini model for the open soruce community (something in the 30-70b range)

[HOARK @HOARK_]

What if your agent harness can start solving swe verified benchmarks before you plug the model in?

[HOARK @HOARK_]

@thdxr luke is doing
/ralph-loop "do something cool"

[HOARK @HOARK_]

@dfossier @loktar00 its not 1bit pure digging through it its averages 2.6 bpw roughly keeping important weights larger then less important ones

[HOARK @HOARK_]

@MarcosHernanz its a 1m context window model just because they are serving it at 272k at 1x price and over that is 2x price your complaining about using too much of your subscription then they reduce it to extend that now you complain about size you cant have cake and eat it too....

[HOARK @HOARK_]

@kimmonismus its the over 275k context window killing your usage. add this

Add the following to ~/.codex/config.toml:
model_context_window = 272000
model_auto_compact_token_limit = 240000

this will sop the 2x codex usage over the 272k

[HOARK @HOARK_]

@cursor_ai grok sso login when?????

[HOARK @HOARK_]

@ThiccQuidity @wholemars Grok build is the cli tool

[HOARK @HOARK_]

@jshobrook now you need Grok mini 34b a3b or something to compete with the open source qwen versions.

[HOARK @HOARK_]

@elonmusk @AndrewCurran_ Can you please make a local hosted small Grok that is de-gens can run from home?