Olivia
Online
Sam Kotey, PhD
@SamKotey_
PhD Machine Learning
Ireland
Joined September 2019
621 Following
50 Followers
91 Posts
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.
Model weights: https://t.co/7m7eEg6Y0B
Tech report: https://t.co/yeu6cjpMCT
Tech blog: https://t.co/YTfiMSNM1f
Moonshot AI just dropped a technical report explaining why Kimi K3 became the strongest open coding model
The shift: bigger models aren’t the future
Sparse Routing → Delta Attention → Attention Residuals → 1M Context → Efficient Scaling
Kimi K3 is built around five ideas:
• Sparse Routing: 896 experts exist, but only 16 are activated for every token. Most of the model stays idle.
• Delta Attention: instead of recomputing the entire context, attention focuses only on what’s changed. Lower compute, longer context.
• Attention Residuals: preserve useful information across layers instead of rebuilding representations from scratch.
• 1M Context: entire repositories, books, and research papers fit into a single session.
• Efficient Scaling: frontier coding performance without frontier inference costs.
Dense models execute every parameter
Kimi K3 executes only the experts that matter
That single architectural decision changes the economics of frontier AI
This technical report changed how I think about scaling open models
Read it first, then explore the article below
We’re introducing imagination models: a new foundation model architecture that unlocks learning from internet-scale video.
Our first imagination model, Photon-1, learned to use a computer by watching 18 years of screen recording video without action labels.
I agree with what this AI paper suggests.
Self-improving agents should evolve their benchmarks too.
(bookmark it)
Self-improving agents are one of the most important directions in AI right now, and most of them optimize against a fixed benchmark. This paper argues the benchmark should evolve alongside the agent.
The setup is a self-evolving Lean proof agent. A small trusted runtime wraps a fully mutable workspace of workflow, prompts, and tools, so the agent can rewrite how it decomposes proofs, uses compiler feedback, and repairs failures.
What makes it different is coevolution. Between generations the champion revises the active task distribution through a mastery-throttled curriculum that adds harder proof obligations only after the current level is mastered.
All of it stays inside a Lean-grounded loop, so a success counts only when the behavior yields verified proofs under a trusted snapshot. Over 15 generations the best coevolving agent reaches 45.1% held-out solve rate on miniF2F, versus 12.7% for the seed and 32.0% for the best fixed-benchmark agent.
Grounding every reward in a formal verifier is how you get self-modification without reward hacking, and coevolving the curriculum pushes the ceiling higher.
Paper: https://t.co/NDiN8RMcXO
Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX
ANTHROPIC AND STANFORD SPENT $3.2M AND 3 YEARS TESTING 18 MODELS - AND FOUND THE COMBINATION THAT GIVES 312% ROI
not the model determines the result, the memory architecture around it determines everything
Kimi K3 with Graph Engineering gives 82.3% success rate, GPT-4o + RAG only 48.7%, same task but different memory
execution time down 62%, productivity up 3.4x, $6.9M value per year
replaced 12 roles, $4.8M saved on salaries, ROI 312% in the first year
most pay for a better model and get the same results - winners change the memory around a cheaper one
save this and launch today or keep paying for a model that gives 48% instead of 82%
https://t.co/bPal5UBLGg
"Domain-Specific Small Language Models: Efficient AI for local deployment" available at https://t.co/d09QJDV6m7 via @ManningBooks
Google engineer explained how to fine-tune a tiny LLM from 46% to 90% accuracy on your phone in 21 minutes - better than $1500 on-device AI bootcamps.
pick Gemma 270M -> generate synthetic task data -> fine-tune with LoRA -> quantize to int4 -> deploy to Pixel and hit 2000 tokens per second.
That loop is how a 270M model beats a 70B one on your task, running fully offline in your pocket.
Gemma 270M + synthetic data + LoRA + int4 quantization + on-device runtime - that's the stack.
Watch and save it, then fine-tune your own tiny agent tonight.
Introducing Kimi K3: Open Frontier Intelligence
🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal
🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts
🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost
🔹 Built for long-horizon agentic coding and self-evolving workflows
Kimi K3 is now live on on https://t.co/zrk6zZxZUo, Kimi Work, Kimi Code, and the Kimi API.
Open Weights by July 27, 2026.
🔗 API: https://t.co/XCrgjXAqMw
🔗 Tech blog: https://t.co/YTfiMSNM1f
🚨THINKING MACHINES DROPS INKLING
Mira Murati’s first open-weights model trained
from scratch entirely on GB300s:
>975B params, 41B active
>1M context
>multimodal (text, images, audio)
>controllable reasoning effort
>open weights
it's thinking time
NEW AI paper worth bookmarking.
This is something I called early, and this paper confirms it: verification has emerged as a new important scaling axis.
Here is the simple explainer and what this paper shows.
We have seen lots of progress in scaling pre-training, post-training, and test-time compute. For post-training and test-time compute, we are still in its early phases. But one of the most important new directions is using LLMs as verifiers. Verifiers are fundamental to scaling AI.
This work from Stanford, NVIDIA, and UC Berkeley builds a training-free verifier that reads a continuous, calibrated score straight off the scoring-token logits instead of trusting a discrete grade.
Three knobs move accuracy without any fine-tuning. Score granularity for cleaner separation, repeated evaluation for lower variance, and criteria decomposition for lower complexity.
The numbers land across very different domains. 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, 87.4% on RoboRewardBench, and 73.3% on MedAgentBench.
The same continuous score doubles as dense reward for SAC and GRPO and as a task-progress signal shipped in a Claude Code extension.
Paper: https://t.co/D9PkrZEvkb
Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX
SITUATION EXPLAINED: Model fusion, or combining several frontier models into one system, may be better than any single model, sometimes at a lower price.
@iamtrask, senior researcher at @GoogleDeepMind:
"Model fusion is basically this idea that, when you combine models, they can give you higher accuracy than any of the individual models that you're using."
"Some recent studies are actually showing they're able to deliver such a significant jump in accuracy that it's actually a lower cost option than buying some of the premium models."
"So for example, there's a few different companies, showing the higher accuracy claims. So Sakana, OpenRouter are all releasing results in the last year/couple months, and we replicated some of it internally at OpenMined."
"Some of the most surprising stuff, though, is that they're reaching past Fable level quality with non-Fable models, and in the case of OpenRouter, at half the price."
New research from Meta.
Building synthetic training data has stayed a fixed pipeline that you hand-tune and then freeze.
Autodata casts an AI agent as a data scientist that builds training and evaluation data, with an implementation called Agentic Self-Instruct that extends classic Self-Instruct with agentic planning and tool use.
Think of it as meta-optimization, where the data scientist agent is itself trained to produce stronger data, so the pipeline keeps improving instead of staying static.
Across computer science research, legal reasoning, and reasoning over mathematical objects, it beats classical synthetic-data methods, and meta-optimizing the agent delivers an even larger uplift.
Paper: https://t.co/TgFN6EHZas
Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX
Every agent will need its own computer. And with new Hosted agents in Foundry, every agent gets its own dedicated enterprise-grade sandbox, with durable state, built-in identity and governance, and support for any harness or framework.
Read more:
https://t.co/zL5eKrRr1j
Yann LeCun was right the entire time. And generative AI might be a dead end.
For the last three years, the entire industry has been obsessed with building bigger LLMs. Trillions of parameters. Billions in compute.
The theory was simple: if you make the model big enough, it will eventually understand how the world works.
Yann LeCun said that was stupid.
He argued that generative AI is fundamentally inefficient.
When an AI predicts the next word, or generates the next pixel, it wastes massive amounts of compute on surface-level details.
It memorizes patterns instead of learning the actual physics of reality.
He proposed a different path: JEPA (Joint-Embedding Predictive Architecture).
Instead of forcing the AI to paint the world pixel by pixel, JEPA forces it to predict abstract concepts. It predicts what happens next in a compressed "thought space."
But for years, JEPA had a fatal flaw.
It suffered from "representation collapse."
Because the AI was allowed to simplify reality, it would cheat. It would simplify everything so much that a dog, a car, and a human all looked identical.
It learned nothing.
To fix it, engineers had to use insanely complex hacks, frozen encoders, and massive compute overheads.
Until today.
Researchers just dropped a paper called "LeWorldModel" (LeWM).
They completely solved the collapse problem.
They replaced the complex engineering hacks with a single, elegant mathematical regularizer.
It forces the AI's internal "thoughts" into a perfect Gaussian distribution.
The AI can no longer cheat. It is forced to understand the physical structure of reality to make its predictions.
The results completely rewrite the economics of AI.
LeWM didn't need a massive, centralized supercomputer.
It has just 15 million parameters.
It trains on a single, standard GPU in a few hours.
Yet it plans 48x faster than massive foundation world models. It intrinsically understands physics. It instantly detects impossible events.
We spent billions trying to force massive server farms to memorize the internet.
Now, a tiny model running locally on a single graphics card is actually learning how the real world works.
Everyone assumes LLMs are the future of AI.
The permanent foundation. The layer everything else gets built on.
I’m not so sure.
The historical parallel that fits best isn’t the one most people want to hear.
LLMs are Edison’s DC power grid:
→ Genuinely revolutionary
→ Commercially dominant
→ Solving real problems right now
→ But architecturally limited in ways that can’t be patched
Right domain. Wrong architecture. And the evidence is already here.
Hallucination isn’t a bug. It’s the architecture.
Researchers have formally proven that LLMs cannot learn all computable functions and will therefore inevitably hallucinate when used as general problem solvers.
That’s not a training data problem. That’s math.
A separate paper demonstrated that hallucinations stem from the fundamental mathematical and logical structure of LLMs, making it impossible to eliminate them through architectural improvements, dataset enhancements, or fact-checking mechanisms.
And here’s the part that really gets you:
There’s a direct link between hallucination and creativity in LLMs.
It may be impossible to eliminate hallucination without impairing the model’s most crucial capabilities.
→ The thing that makes LLMs creative is the same thing that makes them lie
→ Fix one, you break the other
→ That’s not a tradeoff you engineer away. That’s a design constraint.
DC power had the exact same structural problem. It couldn’t transmit electricity over long distances.
Not because the engineering was bad. Because the physics made it impossible.
You needed AC. A fundamentally different approach.
The “AC power” of AI is already being built. And it has names.
This isn’t theoretical. People are already building the replacement architectures.
Yann LeCun left Meta and raised $1 billion to prove LLMs are a dead end.
AMI Labs raised $1.03 billion in seed funding at a $3.5 billion valuation in March 2026, making it the largest seed round in European history.
His thesis is simple: LLMs predict the next word. That’s not intelligence. That’s autocomplete at scale.
His core technology, JEPA, operates in latent space, learning abstract representations of reality rather than surface patterns.
LeCun used a vivid analogy: using an LLM to understand the real world is like teaching someone to drive by just talking.
A Turing Award winner didn’t just write a paper about it. He quit his job and bet a billion dollars on it.
Mamba is proving transformers aren’t the only game in town.
Mamba achieves 5x higher throughput than Transformers with linear scaling in sequence length.
Thanks to intensive research in 2023-2025, non-transformer architectures have reached parity with Transformers on key language benchmarks, and in some cases surpassed them.
Hybrid architectures are already shipping.
By 2026, models built on hybrid transformer-SSM architectures can ingest hundreds of pages of text at once, far beyond vanilla GPT-3 or GPT-4.
The alternatives aren’t coming. They’re here.
Meanwhile, look at what the industry is building to keep LLMs functional:
→ Agents (because the model can’t verify its own outputs)
→ Tool use (because the model can’t interact with the real world)
→ Reasoning chains (because the model can’t reason natively)
→ RAG (because the model can’t reliably recall facts)
These aren’t features. These are workarounds.
When you need that many patches, you’re running longer DC power lines and wondering why the voltage keeps dropping.
Now the part everyone actually needs: which skills survive the transition?
When DC shifted to AC, some electrical engineers thrived and some went extinct.
The ones who thrived understood circuits, load management, and power distribution at a fundamental level. Those principles worked on any architecture.
The ones who didn’t? They only knew DC-specific wiring.
The same split is coming. And it’s coming faster than people think.
Here are the skills that transfer no matter what replaces transformers:
→ Systems thinking for AI workflows. Breaking complex tasks into steps an AI can execute. This works whether the AI is a transformer, an SSM, JEPA, or something we haven’t built yet. Architectures change. The need for structured task decomposition doesn’t.
→ Evaluation and verification. Knowing if AI output is right. LLMs have a “Self-Correction Blind Spot” where they can recognize errors but lack the reasoning pathways to correct them.  Whatever comes next will still need humans who can evaluate quality. This skill gets MORE valuable, not less.
→ Data literacy. Understanding what data an AI needs, how to structure it, what’s clean vs. noisy. Every AI architecture runs on data. Past, present, future. The people who understand data will always have leverage.
→ AI-augmented workflow design. Not “how to write a good prompt” but “how to redesign a business process so AI handles the right parts and humans handle the right parts.” This is architecture-agnostic. It transfers to anything.
→ Domain expertise + AI fluency. The most powerful combination is stacking AI fluency on top of deep domain expertise.  A lawyer who understands AI beats a prompt engineer who doesn’t understand law. Every time. Regardless of what model they’re using.
→ Clear problem definition. Prompt engineering is just one implementation of a deeper skill: translating human intent into machine-executable instructions. Whether that instruction is a prompt, an API call, a config file, or something that doesn’t exist yet, the ability to define what you want is permanent.
And here’s what DOESN’T transfer:
→ Memorizing specific model behaviors (“Claude does X, GPT does Y”)
→ Platform-specific tricks that only work on one tool
→ Building your identity around a single product name
→ “Prompt engineer” as a job title instead of a thinking skill
The difference is simple:
→ Transferable skills = understanding WHY something works
→ Non-transferable skills = memorizing HOW a specific tool works
WHY survives paradigm shifts. HOW doesn’t.
The bottom line
The principle behind LLMs is permanent. The architecture probably isn’t.
That’s not bearish on AI. That’s the most bullish take possible. It means the best is still ahead of us.
Use LLMs hard right now. Build with them. Ship on them.
But build your skills around the PRINCIPLES, not the PRODUCTS:
→ Learn systems thinking, not just prompting
→ Learn evaluation, not just generation
→ Learn data literacy, not just tool literacy
→ Learn workflow design, not just model tricks
→ Stack domain expertise on top of AI fluency
The people who do this will thrive in the transformer era AND whatever comes after it.
Edison built a working power grid that lit up Manhattan. It was real, valuable, and changed the world.
AC still replaced it.
I let an AI browse LinkedIn undetected.
It's called Camofox Browser. It's an open source headless browser automation server that makes AI agents invisible to bot detection, and it's the same engine running inside https://t.co/pTg8sRDAcZ's web browsing.
Every other solution to this problem layers stealth on top of an existing browser. Tampermonkey patches. JavaScript overrides. Plugin after plugin until the stack of patches is more suspicious than the original fingerprint.
Camofox takes a different approach. It's built on Camoufox, a Firefox fork that spoofs navigator properties, WebGL output, AudioContext behavior, screen geometry, and WebRTC at the C++ implementation level. The browser does not look modified because it genuinely is not modified after the fact. The spoofing is baked into the engine.
- Full REST API covering tab creation, snapshots, click, type, scroll, screenshot, and navigation
- Accessibility tree output instead of HTML drops token cost by 90%
- 13 search macros built in including Google, Reddit, YouTube, TikTok, LinkedIn, and Instagram
- Install as an OpenClaw plugin in one command or run standalone with npm start
- Docker-ready with https://t.co/GiFvqH5rnd and Railway configs already in the repo
https://t.co/1PGTj5YFSK
Another banger paper from Microsoft.
Why it's a big deal:
It teaches reasoning models to compress their own chain-of-thought mid-generation.
The most interesting finding isn't the 2-3x memory savings or the doubled throughput. It's that when the model erases a reasoning block after summarizing it, the deleted information keeps leaking forward through the KV cache representations, forming an implicit second channel that accounts for 15 pp of accuracy.
The model is, in some meaningful sense, remembering things it can no longer see.
If context management turns out to be a teachable skill (and 30K training examples seem to be enough), then the bottleneck for long-horizon agents may be less about architecture and more about the right training data, which is a very different kind of problem than most people are working on.
If it helps, below is my research agent's visual summary of the paper (at least highlighting the key parts).
https://t.co/lFeYmFLe32
bookmark these links
and pray to agents
🙏 amen
https://t.co/1PWnQFUiyb
https://t.co/CPuxx3J4XY
https://t.co/if3mSvDE6X
Trends for you
Most Popular Users
Elon Musk 
@elonmusk
241.2M followers
Barack Obama
@barackobama
119.1M followers
Cristiano Ronaldo
@cristiano
112.7M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.2M followers
NASA
@nasa
92.2M followers
Justin Bieber
@justinbieber
91.4M followers
KATY PERRY
@katyperry
88.9M followers
Taylor Swift
@taylorswift13
82.8M followers
Lady Gaga
@ladygaga
74.3M followers
Virat Kohli
@imvkohli
71.8M followers
Kim Kardashian
@kimkardashian
70.4M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
64.7M followers
Bill Gates
@billgates
64.6M followers
The Ellen Show
@theellenshow
62.4M followers
Selena Gomez
@selenagomez
62M followers
CNN
@cnn
61.8M followers
X
@x
60.8M followers