DMNSH

Resisting the entropy

[DMNSH @dmnsh001]

More code will be written from today to 2028 than possibly ever written before

[DMNSH @dmnsh001]

let's bet on next release score (in spirit of world cup)

[Arena.ai @arena]

Big news: Kimi-K3 by @Kimi_Moonshot is now #1 in the Frontend Code Arena with 1679 pts, surpassing Claude Fable 5.

This is a 17-place jump from Kimi-k2.6 (#18 -> #1).

In Frontend, Kimi-K3 ranked #1 in 6 of 7 domains: Brand & Marketing, Reference-Based Design, Data & Analytics, Consumer Product, Simulations, and Content Creation Tools, landing #2 only in Gaming behind Fable 5.

The full model weights will be released by July 27.

Congrats to the @Kimi_Moonshot team on this major milestone!

[DMNSH @dmnsh001]

@xeophon look at the mid-training data for vision😅

[DMNSH @dmnsh001]

Prefill and decode seperation at arch level

[NVIDIA AI @NVIDIAAI]

We took a 30B model and split it in two to write tokens in parallel instead of one at a time.

Introducing Nemotron-Labs-TwoTower: a diffusion language model from NVIDIA Research adapted from Nemotron-3-Nano-30B-A3B. Here’s how it works: one half holds the context, the other writes the tokens, with both reusing the pretrained model instead of training a new one from scratch.

We found it kept 98.7% of the original model’s quality at 2.42× faster generation.

[DMNSH @dmnsh001]

The ablations and benchmark gains are reported in the paper:

https://t.co/gjwFsjuUwJ

[DMNSH @dmnsh001]

Regular RL merges multiple signals into one entity to compute advantage scores. This may limit the model's answering styles.

Vector Policy Optimization (VPO) is a nice improvement in which the rollout scoring is randomized to diversify answering styles.

First, within each prompt, the model is asked to generate three different answers using a delimiter. Meaning the output will be <answer-1>, <answer-2>, and <answer-3>. Authors explain: "Under standard independent sampling, diversity arises only from stochastic decoding applied to a fixed conditional distribution, producing small variations around whichever mode the policy has concentrated on." meaning being able to attend previous answers, steers the model toward new solution spaces.

Now say we have 8 rollouts, each with 3 different answers, the next interesting difference is the generation of K (assume 4) different weights as "judges". Each weight set is applied to answers within all rollouts. And the maximum value generated across judges averaged per rollout:

R1: rollout 1 - A11, A12, A13: three answers of R1
4 Judge weights: J1, J2, J3, J4 (each being something like [0.65, 0.11, 0.24] summing to 1 and generated at random)
reward_vector (rv): gives the vector showing reward of different signals

Per rollout:
R11 <- max([rv(A11)*J1, rv(A12)*J1, rv(A13)*J1])
R12 <- max([rv(A11)*J2, rv(A12)*J2, rv(A13)*J2])
R13 <- max([rv(A11)*J3, rv(A12)*J3, rv(A13)*J3])
R14 <- max([rv(A11)*J4, rv(A12)*J4, rv(A13)*J4])

(the * here is dot product)

The reward for R1 will be the average of [R11, R12, R13, R14] each being the maximum score for different judge preferences. After scoring, the rest is GRPO.

Each judge will score differently and if the answers are not diverse enough then overall score will be lower.

[DMNSH @dmnsh001]

@gpusteve Are there any performance comparisons among different quantizations for KV cache specifically? Also has the accuracy been tested against agentic benchmarks?

[DMNSH @dmnsh001]

@malikwas1f Any metrics on speed and token generation needed for each model?

[DMNSH @dmnsh001]

@CodeByNZ buttposted the job

[DMNSH @dmnsh001]

@asusevski what is the role in which you get paged at 2AM? Could be a specific area that is heavily automated. Sounds like a devops-ish role to me

[DMNSH @dmnsh001]

leaner turns, longer trajectories on smaller models is the theme

[DMNSH @dmnsh001]

On consumer hardware, a dense 14-32B model generates tokens as fast as a 80-300B MoE model but the MoE models store more knowledge and needs less tokens for the same tasks; making it faster/better IN PRACTICE

[DMNSH @dmnsh001]

Different family models all starting with "Okay, let's...", is this some kind of convergence?

[DMNSH @dmnsh001]

~1 year ago: Qwen3 writes full essay before every tool call

Today: Qwen 3.x variants drop 1 reasoning sentence before tool calls

Next couple of months: Optimal harness + omni capabilities at all sizes

[DMNSH @dmnsh001]

I suspect early AI labs monitored VRAM usage of dense models and said fuck it we will make MoEs work

[DMNSH @dmnsh001]

@rot13maxi Too expensive to tune

[DMNSH @dmnsh001]

@sama Dostoevsky talked about this.

[DMNSH @dmnsh001]

@jayfarei Say you want the model to learn to optimize its harness use by using sub-agents, what would the "grounded outcome" look like to decide the reward? Count of sub-agents?

[DMNSH @dmnsh001]

@willccbb Looking at open models, there are some low hanging fruit when it comes to harness use. Like most open models won't even read the AGENTS.md file or instead of reading a portion of the file, they just read the whole thing and blow the context.

[DMNSH @dmnsh001]

Cursor V3 is a conspiracy to eliminate manual coding.