pipi.md

AI & Software Engineering

[Sebastian Raschka @rasbt]

MiMo-V2.6 is "simply" the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average).

With "simple," I mean a classic Grouped Query Attention (GQA) with Sliding Window Attention (SWA) at a tiny 128-token window size.

So, that underlines one of the points I've been trying to make in recent months: most of the progress still comes from the data and post-training recipe improvements. Fancy attention variants are just mostly efficiency tweaks.

What are some of the training data improvements and recipe improvements? The MiMo team shared a pretty detailed technical report. Lots to carefully digest there, but in short, there are a few things that stood out:

1. An increase in agent tasks; also training across different harnesses (the average DeepSWE pass@1 accuracy on held-out harnesses improved from approximately 50% -> 66%).

2. Better reward signals: they replaced a simple correctness verifier with an agentic grader that looks at the execution traces as well.

3. Large RL batches (1,568 prompts × 16 rollouts = 25,088 trajectories) and 2.7–3.7 billion training tokens per update (unclear, though, what the predecessor used).

[pipi.md @0xpipimd]

The original 4096 × 4096 weight stays frozen. LoRA learns a low-rank update on top of it.

That simple factorization is the core idea behind LoRA.

Original paper:
https://t.co/vrHb8I22L0

Hugging Face PEFT docs:
https://t.co/GM0HR3Ow9J

[pipi.md @0xpipimd]

LoRA makes a lot more sense once you look at one matrix.
Suppose a model has a 4096 × 4096 weight matrix.
Training the whole matrix means:
16,777,216 trainable parameters.
Now use a rank-8 LoRA update.

[pipi.md @0xpipimd]

Instead of training that full matrix, LoRA learns two smaller matrices:
4096 × 8
and
8 × 4096
Together: 65,536 trainable parameters.
That is 256× fewer trainable parameters for this one weight matrix.

[pipi.md @0xpipimd]

Interesting architecture if you are building agents that can actually take actions.

https://t.co/urpIcvkE59

[pipi.md @0xpipimd]

Putting a GitHub token or API key directly into an agent's environment is convenient.
It also means the agent can read the key.

Infisical's open-source Agent Vault takes a different approach.

[pipi.md @0xpipimd]

It can also substitute placeholder credentials when a tool expects a token to be present.

For a stronger security guarantee, outbound traffic should be restricted so the agent cannot simply bypass the proxy.

[pipi.md @0xpipimd]

One thing to know before trying it: session data stored with a public repo can be public too.

Entire redacts detected secrets, but the project explicitly describes that redaction as best-effort.

https://t.co/Pm0QphodUe

[pipi.md @0xpipimd]

When an agent changes 12 files, the final diff tells you what changed.

It usually does not tell you how the agent got there.

Entire is an open-source CLI that records coding-agent sessions alongside your Git history.

[pipi.md @0xpipimd]

It can capture prompts, responses, files touched, tool calls and checkpoints, then associate that context with the relevant commits.

You can also return to an earlier checkpoint and resume the work instead of starting the agent from scratch.

[pipi.md @0xpipimd]

What I like about this release is that the optimized code is public, so the changes can actually be inspected and tested.

Research: https://t.co/4ilIHdOjOe

GitHub: https://t.co/AzMXxW1uJK

[pipi.md @0xpipimd]

Claude optimized more than 30 open-source biomolecular models in under four weeks, and Anthropic has released the code.

The work covers structure prediction, protein design, protein language modeling and genomics.

[pipi.md @0xpipimd]

The GitHub repo contains 36 drop-in optimization kits.

For the structure-prediction models, Anthropic reports roughly 4× average speedup in the faster mode. Some kits also provide an exact mode that produces identical outputs while running faster.

[pipi.md @0xpipimd]

If RLHF and DPO have mostly been implementation terms to you, this looks like a good place to understand the machinery underneath them.

Course: https://t.co/Iios0hQIml
Textbook: https://t.co/XGvey9uIVJ

[pipi.md @0xpipimd]

Stanford's Machine Learning from Human Preferences course starts tomorrow.

I was going through the syllabus, and it goes much deeper than just RLHF and DPO.

[pipi.md @0xpipimd]

Bradley–Terry models, random utility models, Bayesian estimation, Fisher information, active preference elicitation, RLHF, DPO, assistance games, bandits, aggregation and fairness.

There is also a free Stanford textbook behind the course, with slides and problem sets.

[pipi.md @0xpipimd]

That avoids overflow from trying to compute exp(x) when a logit is very large.

A tiny implementation detail, but an important one for making softmax numerically stable in floating-point arithmetic.

SciPy docs: https://t.co/N2m8C69rDF

[pipi.md @0xpipimd]

One small numerical trick shows up everywhere in ML code, and it is easy to miss why it matters.

When computing softmax, subtract the largest logit first:

softmax(x) = softmax(x - max(x))

The probabilities do not change.

[pipi.md @0xpipimd]

The same factor introduced by the shift cancels between the numerator and denominator.

But numerically, something useful happens.

The largest exponent becomes e^0 = 1, and every other exponent is at most 1.