Simon

Doing small things that can change the world

[Simon @Simon02ZJ]

Interesting !

[Avi Chawla @_avichawla]

RAG vs. Jev + RAG, clearly explained!

In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector DB.

When a query arrives, the system embeds it and retrieves the top-k chunks with the closest vectors.

Many production setups add a reranker after retrieval.

The reranker compares the query with each retrieved passage, improves their ordering, and keeps the highest-scoring results.

Those passages are then placed in the context window, and the LLM generates an answer from them.

The problem is that ranking and answerability are different questions.

A passage can be more relevant than the other candidates while still containing weak, incomplete, or merely adjacent evidence.

The LLM receives it anyway and may produce a plausible answer from context that never supported one.

Jev + RAG keeps the retrieval stage as is but changes what happens before generation.

Instead of using a conventional reranker at this stage, Jev can judge every retrieved candidate against a typed question such as:

"Does this passage help answer the query?"

The query becomes the shared state, while the retrieved passages become individual candidates. Jev evaluates them together and returns a probability for each one.

Application code then applies a threshold.

↳ Candidates above the threshold continue to the LLM.
↳ Candidates below it are removed from the context window.

The same Jev request can also judge whether the remaining evidence is sufficient to answer the query.

If answerability falls below the threshold, the application can skip the LLM and return "not in the documents."

Jev does not replace the embedding model, vector database, or generation model. Retrieval still sets the cap because Jev cannot recover a passage that never entered the candidate set.

The diagram below depicts the complete flow.

- RAG retrieves a broad candidate set.
- Jev reranks and gates those candidates.
- The LLM generates an answer.

If you want to dive deeper, I have also written a hands-on guide showing how to build a Jev-style model with open models, entirely locally.

Read it below.

[Simon @Simon02ZJ]

You have to read this article!

[David George @DavidGeorge83]

https://t.co/AXVmndxtk3

[Simon @Simon02ZJ]

Interesting article!

[Sydney Runkle @sydneyrunkle]

https://t.co/VVDiN1rvRU

[Simon @Simon02ZJ]

Nice work!!

[Akshay 🚀 @akshay_pachaar]

Another insane Jev use case!

Jev makes it incredibly cheap to evaluate and classify agent runs at scale.

And finally, someone open-sourced a self-improving memory layer that can put that capability to work across agent harnesses.

It turns your agent sessions into a compounding knowledge layer, where every successful run can make future agents smarter across:

- Codex
- Claude Code
- Cursor
- OpenCode and 20+ more

Beacon by @asymptotelabs continuously builds a shared history across your agent harnesses and uses Jev to identify the runs worth learning from.

It then turns the best workflows, corrections, and debugging patterns into reusable skills.

GitHub repo: https://t.co/sfdu9P1bfx.

(don’t forget to star it ⭐)

Most agent runs are messy.

They contain exploration, failed commands, dead ends, and one-off fixes that should never become permanent memory.

So Beacon preserves the full session history, while Jev helps decide what should be promoted, reviewed, or discarded.

The recording below shows this in action.

Beacon found 579 sessions across 5 coding-agent harnesses and normalized them into one consistent history.

From there, Jev surfaces the lessons worth keeping and makes them available across your agent stack.

- A pattern learned in Cursor can carry into OpenCode.
- A lesson from Claude Code can improve the next Codex run.

Every successful run adds to the shared knowledge layer, making future agents smarter.

If you want to dive deeper into Jev, I also wrote a breakdown of how it works.

The article is quoted below.

[Simon @Simon02ZJ]

Just do it!

[Sam Altman @sama]

“give yourself a lot of shots to get lucky” is even better advice than it appears on the surface.

luck isn’t an independent variable but increases super-linearly with more surface area—you meet more people, make more connections between new ideas, learn patterns, etc.

[Simon @Simon02ZJ]

Great!

[elvis @omarsar0]

Must-read paper from Google on self-improving agent harnesses.

If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks.

This paper shows how to prevent that.

Of five harness-evolution methods compared on agentic workspace tasks, RRSI scored the lowest on the tasks it evolved against and highest on all three out-of-distribution benchmarks.

Automated harness evolution proposes edits to prompts, control flow, tools and memory, keeps the ones that raise the score, and repeats.

The authors show this overfits the training tasks. Meta-Harness reached 93.0 on the Harvey LAB evolve split but gained only 0.3 to 1.5 points on JobBench, GDPval and APEX-Agents.

RRSI adds regularization on both sides of the loop.

The proposer gets an edit budget that shrinks over time and is pushed toward directions it has not tried. A critic rejects benchmark-specific edits, and a pruner removes edits that are too small, too costly or no longer useful.

RRSI scored 90.5 on the evolve split and gained 3.5 to 4.7 points on the three held-out benchmarks. In the ablation, unregularized evolution used 3.80M tokens per trial against 2.42M for RRSI. With Gemini 3.5 Flash, RRSI raised Terminal-Bench 2.1 from 64.6 to 78.7 and carried a 2.2-point gain over to SWE-bench Verified.

Paper: https://t.co/SlfjDg96VI

Chat with Paper: https://t.co/gBotiH6Jfq

[Simon @Simon02ZJ]

Great work

[Alex Veremeyenko @alex_verem]

Build your own harness.

This is a seriously good paper from NVIDIA on self-improving agent harnesses.

The paper is called SoL-Pi, out of NVIDIA, NTU and MIT.

Every coding agent runs inside a harness. That's the layer that reads files, runs commands, manages memory and talks to the model. Humans tune it by hand, reading long execution logs and turning what they notice into code changes.

They handed the job to an AI instead.

A research AI watched a coding agent work, proposed changes to the harness, tested them, and kept only what saved tokens without hurting results. It ran about 150 research directions across roughly 500 environments, more than 3,000 runs and over 60,000 agent-environment interactions.

Four fixes survived.

1. The agent edits a file and runs the test in one call instead of two.
2. It compresses its memory when a subtask finishes, and only if that's cheaper than keeping everything.
3. It sends big tool outputs in full twice, then swaps them for a 1KB excerpt with a handle to pull the rest on demand.
4. A cheap model summarizes build and test logs, and a verifier rejects the summary if it drops evidence.

None of that touches the model. It's all in the software around it.

On EdgeBench, a 51-task benchmark, token traffic dropped 44.7 to 49% and API cost fell by about a third against Pi, the harness it was built on. Against the native harnesses it came in 50% cheaper than Codex on GPT-5.6 Sol and 54.3% cheaper than Claude Code on Opus 5. Scores held at roughly 94% of the baseline.

They built all of it on GPT-5.6 Sol, then dropped it onto Opus 5 with zero changes. It still used 44.7% fewer tokens.

The cost conversation so far has been about cheaper models. This paper says a big chunk of the waste sits in the software around the model, and an AI finds it faster than we do.

Their next move is running the search again with the cheaper harness, so a cheaper agent funds the search for an even cheaper one.

[Simon @Simon02ZJ]

Nice work!

[Josh Rosen @JoshARosen]

https://t.co/97od35Ae48

[Simon @Simon02ZJ]

Agree

[Jeffrey Katzenberg @jeffreykWNDR]

The World is Changing: AI For Creativity

By Jeffrey Katzenberg

A few months ago, I sat in my office in Silicon Valley and watched as a tech founder showed me something extraordinary. On the screen was a fully realized, beautifully lit, well-composed animated scene. It was stunning and it made me feel exactly what I felt in 1986 watching Luxo Jr. That was the first time I watched a computer-animated 3D character take a breath and seem, against all reason, to have life. It left me in awe.

Later that day, I received a text from an artist I've known for thirty years, 350 miles to the south, in the city where I spent most of my career. After seeing a similar video, she texted: "Is this the end of us?" My answer was, "Certainly not.”

I have spent the better part of the last decade in Silicon Valley, but the heart of my career has been in Hollywood. Being deeply connected to both worlds means I have deep loyalties to each and a responsibility to speak honestly to both.

In 2023, I said that these new AI tools would cut the time and cost of producing world-class animation by as much as ninety percent within three years. Some colleagues were alarmed, many were furious.

There is growing fear and resistance surrounding AI within the creative community. I deeply understand it, because I've spent countless hours walking through animation studios watching gifted artists bent over their desks, rebuilding a single second of film for the tenth time because the ninth version wasn't quite right. I've sat in screening rooms where four years of people's labor played out in minutes, and I knew the name of every person that had spent countless hours bringing those images to life. The creative process is a calling, there's really no other way to describe it. From the outside some see resistance. From the inside, it is love.

People do not fight this hard for things they don't care about. The pushback coming out of Hollywood represents the collective effort of people who are deeply passionate about their craft.

Is History Repeating Itself?

The history here is more complicated than either side may realize. In 1906, the most famous composer in America, John Philip Sousa, published an essay titled “The Menace of Mechanical Music." He warned that the phonograph would become "a substitute for human skill, intelligence and soul."

Sousa's fight was not really about the machine, it was about money. The machines were playing his compositions, and the men who built them weren't paying him a cent. His campaign helped create the Copyright Act of 1909. He did not stop the technology. He changed the terms under which it could use his work.
A hundred years ago, sound came to the movies. We remember it now as a miracle, and it was. What we forget is who paid for it. Before sound, tens of thousands of musicians made their living in the orchestra pits of movie houses, scoring every film live, every night, in towns all over the world. When the soundtrack arrived, the work of one composer and one orchestra was recorded for a film that went into thousands of theaters. The union fought back with everything it had, taking out newspaper ads across the country warning against the menace of "canned music," one of them showing a mechanical man tearing the strings out of a harp while an angel wept.

They were not fools, and they were not Luddites. They were right. Those pit jobs did not come back. And yet (this is the part we have to be brave enough to admit), sound gave us the movie musical, the modern score, sfx, sound design, audio engineering, and an art form vastly larger than the one it disrupted. And it helped keep Hollywood in the forefront of world entertainment for the rest of the century and into the next. The loss was real. And yet the art form expanded.

This is a story that has been told over and over again. To resist technology is to risk irrelevance. Just look at Kodak or Blockbuster. To embrace technology is to open doors of new possibility. Just consider Apple and Netflix.

What I Learned From Walt Disney

In the mid-1980s, I was tapped to lead Disney's animation division at a moment when the studio was at an inflection point. Animation wasn't just another business unit. It was the soul of the company, a medium revered because of Walt's genius and his passion. But the production system was cumbersome and unforgiving. A single movie was 125,000 individual hand-drawn and painted cels, photographed one frame at a time. Every revision carried a cost measured in months. These degrees of difficulty shaped the kinds of stories we could tell.

We found our way forward in an unexpected place: Walt himself. The Disney archives held astonishing recordings of Walt explaining his creative process. His own writings. His notes and storyboards. Work product captured at every stage of his process. This was truly a gift. Listening, reading, sitting with the work itself, we heard him talk about character, about emotion, about how an audience feels when a character truly comes alive. He talked about making bold choices and refining a scene until it genuinely moved people. We didn't hear a word about pencils or paintbrushes. In fact, Walt was famous for being a technologist, forever hunting for state-of-the-art tools, often inventing them himself to achieve the images he saw in his head. But he never defined animation by the tools. He defined it by whether the audience believed the character. His principles were timeless. The tools were not.

That realization changed everything. We co-developed the Computer Animation Production System (CAPS) with a young Northern California company called Pixar, replacing hand-painted cels with CGI. In The Little Mermaid, the final scene shimmered with a dimensionality and light that the old process simply couldn't achieve. In Beauty and the Beast, the ballroom sequence moved with a cinematic sweep that placed the audience inside the emotion of the moment. In Aladdin, the Cave of Wonders felt vast and alive, and the Magic Carpet became an intricate, compelling character all its own. In The Lion King, the stampede carried a scale and intensity that raised the emotional stakes beyond anything we'd done before. Technology didn't diminish the craft, it expanded the canvas. It gave artists more room to create.

A decade later, the canvas expanded again. When Disney released Pixar's Toy Story, it wasn't simply a technical milestone. It was proof that a fully computer-animated film could carry real emotional weight, that it could make audiences laugh, cry, and believe. At DreamWorks, we made the difficult decision to sunset hand-drawn animation and become a fully computer-animated studio. It was the right thing to do, but it was not without pain. It cost talented people their place in an industry where they had worked their whole lives. Some made the leap to the new tools and did the finest work of their careers. Some never did.

Tools are never the point. The instruments change with every generation. What endures is taste and imagination. The magical ability to make an audience feel. One of the greatest storytellers of our generation, George Lucas, succinctly captured the eternal essence of this issue: “It’s not the how, it’s the why.”

A Distinction With a Difference

I asked one of the leading AI models a question that has been challenging me for months. What is the difference between reasoning and creating? Its answer changed how I think about almost everything happening in this industry. It said . . .

Reasoning and creating are two distinct cognitive modes, though they also work together. Reasoning is fundamentally evaluative and analytical. It operates on what already exists: facts, premises, evidence. It moves toward a conclusion that was in a sense already implied by the input. Reasoning is constrained by logic and truth. Its goal is to arrive somewhere correct, not to invent somewhere new.

Creating is fundamentally generative. It produces something that didn't exist before. And crucially, there's no single right answer waiting to be found. A blank page has infinite valid responses. Creation involves choices that can't be fully justified by logic alone. Taste, intuition and vision fill the gap where deduction runs out.

Reasoning is what Silicon Valley has been perfecting. Creating is what Hollywood has been practicing for more than a century. AI today operates almost entirely on the reasoning side of the line. It can deduce, evaluate, optimize, and pattern-match brilliantly. And while it can create, there is a real distinction to being creative. What it doesn’t yet have is those things that make us human: empathy, devotion, serendipity, the kind of creativity that comes from a person trying to say something only they could say. When the bot generates a piece of art, it is not trying to communicate anything. It is statistics, not soul; it is emulating things that have been done. By contrast, human creativity isn’t about repeating patterns of zeros and ones; it is about doing something new.

One day, AI may close this gap. Three years ago, the leaders building AI would have called what they are achieving today, improbable, if not impossible. Impossible is no longer improbable.

Today, the line between reasoning and creating is real. Even the leading technologists acknowledge we are not there yet. There is no scientific path to crossing this divide that anyone in the field can articulate today. Understanding that gap is where we will find common ground.

A Path Forward

In 2016, I closed one chapter in Hollywood with the sale of DreamWorks and opened another in Northern California, co-founding WndrCo. We’ve backed more than 50 founders building the next generation of technology and watched how breakthroughs in Silicon Valley emerge, first as experiments, then as platforms, and finally as infrastructure that reshapes entire industries. It's worth remembering that the last great revolution in animation also came from the north. Pixar was a Northern California company, forged not in the conventions of the Hollywood studio system, but in the technological breakthroughs of Silicon Valley. I've spent years on both sides of this bridge. For sure, I don’t have all the answers (take Quibi, for one!). But, from my past and present vantage points of my long career, here is what I see . . .

Brilliant people in Northern California building this technology have made something extraordinary. They have earned the right for the rest of us to be, if not believers, at least optimistic that what comes next will be remarkable. But they have not made an artist. The tools are powerful, but they are not what makes a story matter. That knowledge lives 350 miles to the south, inside people whose life's work has informed the very models you are building. The right path forward includes them by design, with credit, with consent, and with compensation. Build this with the storytellers. Not on top of them. Taste is not something that can be synthesized, it is uniquely human.

At the same time, Hollywood needs to accept that AI is not going away. The energy they are spending trying to make it disappear is energy they are not spending deciding the terms on which it will exist. And the terms are everything. The north needs something from it that they cannot build and cannot buy: creativity. The kind that takes a blank page and conjures a single right answer where there was none and has held audiences for a century. Without it, the most powerful reasoning engine ever invented will still be missing the only thing that makes a story worth telling.

The artists who learn to wield these new instruments will do things the engineers never dreamed of. They always have. Edison invented the motion picture but made terrible movies. It took Chaplin, Lloyd, Keaton and so many others to make movies emotional. Now, the canvas is about to expand yet again. We should decide now that we intend to paint on it.

There are so many valuable lessons in history. This has happened many times before, and it was never settled by the technology. It was settled by the terms. Sousa did not stop the phonograph; he helped write the law that made sure composers got paid. And two years ago, when the writers and the actors walked out, they were fighting for the very things Sousa was fighting for in 1906. Consent, compensation, the basic recognition that human creative work has a price that must be paid. The terms of that fight are still being negotiated, but the principle is older than any of us.

The tools-versus-no-tools argument is a trap. First, we must all agree that there should be terms. Then we can have the crucial debate about what fairness requires.

What I Learned From Steve Jobs

Years ago, Steve Jobs said, "It's in Apple's DNA that technology alone is not enough. It's technology married with the liberal arts, married with the humanities, that yields us the result that makes our hearts sing." He was describing a device. But he could just as easily have been describing this tale of two cities.

What I See Coming Soon

As the barriers and the costs come down, more films will get made, not fewer. Studios will get to take more risks. There will be more seats at the table, and very soon entirely new forms of storytelling. In the 1980s, animation was dismissed as a niche corner of the business. Today it is one of the most beloved and profitable forms of storytelling in the world. In live action, filmmakers like Steven Spielberg, James Cameron and Peter Jackson embraced new visual tools not as shortcuts, but as instruments, and expanded cinema in the process. Every time storytelling has met a genuine technological shift, from synchronized sound to color to computer animation, it has redefined the boundaries of the medium and grown larger in the process.

Assuredly, I don’t have all the answers, but I am confident that the creative opportunities will expand yet again. How we come through this is a choice. The north has the new tools. The south has the creative soul. The best future will draw on the best of both worlds.

[Simon @Simon02ZJ]

@jeffreykWNDR Agree

[Simon @Simon02ZJ]

@AlanShao111 150刀