Jev for RAG, clearly explained!
Hybrid search gives you a shortlist. It does not decide which passages contain evidence, which are merely adjacent, or whether the evidence is strong enough to answer.
That missing judgment is where Jev fits.
Jev sits between retrieval and generation. It does not replace BM25, embeddings, or the LLM. It evaluates the candidates they produce before those candidates enter the context window.
→ Retrieve wide
Combine dense and keyword search, then merge the results with reciprocal rank fusion. This gives Jev a broad candidate pool, such as the top 20 passages shown in the visual.
Retrieval still sets the ceiling. If the right passage is missing from this shortlist, Jev cannot recover it.
→ Judge every candidate together
Send the query as Jev’s state. For each candidate, ask a typed yes-or-no question such as “Does C7 help answer this query?”
Jev evaluates every question in one packed request and returns a calibrated probability for each passage. This avoids making a separate model call for every query-passage pair.
→ Let code apply the threshold
Your application compares each probability with a threshold. Candidates above it continue to the LLM. Everything below it is removed before generation.
Jev makes the fuzzy judgment. Code remains responsible for the actual decision.
→ Gate the entire answer
The same request can check whether the retained passages make the query answerable. It can also flag signs of prompt injection inside a candidate.
If answerability falls below the threshold, the application can skip the LLM and return “not in the documents.” The injection score should remain a filtering signal, not a security boundary.
The result is a cleaner division of work.
Hybrid search retrieves broadly. Jev reranks, filters, and decides whether sufficient evidence exists. The LLM writes only from the passages that survive.
Jev’s value here is not simply moving passages up or down a list. It turns relevance into an explicit probability that your application can inspect, threshold, and act on.
To summarise:
- Retrieval finds the candidates.
- Jev decides what deserves context.
- The LLM writes the grounded answer.
----
I also built an open-source project showing how to use Jev as a judge for AI observability with Comet Opik.
It evaluates support traces for groundedness, request coverage, action honesty, and helpfulness, then records the results as an auditable experiment.
You can explore the project here: https://t.co/XVmpg6ldHT
My article on how Jev works is quoted below.
AI Agent 要怎么写测试,还不会过度测试、为测试而测试 ?
@anshnanda 分享了自己写在 AGENTS.md 顶部的三条规则,值得一看!
1. 严禁在写完代码之后再补单元测试。 事后补写的单测(尤其是 Agent 生成的)往往只是把实现逻辑复述一遍:代码怎么写的,测试就怎么断言。它验证不了正确性,只制造“有测试覆盖”的假象。dex 举例的那种“检查常量里包含某子串”正是这类无意义的同义反复。
2. 首选 E2E 测试作为唯一的验证机制,且结束时必须产出可验证、可复现的产物。 他要的是从真实用户路径出发验证功能确实能用,而判据不是“测试绿灯”,而是一份能留档、能重跑、能被人检查的证据(如输出文件、录屏、报告)。这把验证从“测试通过”升级为“拿出证据���。
3. 如果确有必要隔离测试某个系统,先穷举它所有可能的失败方式,再动手写代码。 这是失败模式驱动的前置测试:测试的价值在于预先写下“哪些情况会坏”的假设,让测试承载设计意图,而不是事后镜像实现。
AGENTS.md 中要加入顶部的原文:
- NEVER write unit tests after you write code.
- Highly prefer E2E tests as the sole testing mechanism. Use them to verify complex features work. At the end of E2E tests, produce a verifiable and repeatable artifact.
- If you must test a system in isolation, FIRST write all the ways it could fail, THEN write the code.
Another insane Jev use case!
Jev makes it incredibly cheap to evaluate and classify agent runs at scale.
And finally, someone open-sourced a self-improving memory layer that can put that capability to work across agent harnesses.
It turns your agent sessions into a compounding knowledge layer, where every successful run can make future agents smarter across:
- Codex
- Claude Code
- Cursor
- OpenCode and 20+ more
Beacon by @asymptotelabs continuously builds a shared history across your agent harnesses and uses Jev to identify the runs worth learning from.
It then turns the best workflows, corrections, and debugging patterns into reusable skills.
GitHub repo: https://t.co/sfdu9P1bfx.
(don’t forget to star it ⭐)
Most agent runs are messy.
They contain exploration, failed commands, dead ends, and one-off fixes that should never become permanent memory.
So Beacon preserves the full session history, while Jev helps decide what should be promoted, reviewed, or discarded.
The recording below shows this in action.
Beacon found 579 sessions across 5 coding-agent harnesses and normalized them into one consistent history.
From there, Jev surfaces the lessons worth keeping and makes them available across your agent stack.
- A pattern learned in Cursor can carry into OpenCode.
- A lesson from Claude Code can improve the next Codex run.
Every successful run adds to the shared knowledge layer, making future agents smarter.
If you want to dive deeper into Jev, I also wrote a breakdown of how it works.
The article is quoted below.
Jev와 비교했을 때 Kev가 압도적으로 혁신적인 이유:
1. 단순 '예/아니오'를 넘어 '선택지별 가중치와 점수' 반환
- 기존 Jev가 주로 단순 "예/아니오" 식의 이진 판단 위주였다면,
- Kev는 객관식 다중 선택과 1~5점 척도 평가까지 한 번에 처리함.
- 단순히 정답 하나만 딱 고르는 게 아니라, 각 선택지별로 정밀한 '확률 가중치'를 뽑아줌.
- 예: 복합 문의가 들어오면 [반품 47%, 배송 28%, 결제 25%]처럼 미묘한 비율까지 1번의 연산으로 완벽히 분석해 냄.
2. 여러 질문을 동시에 던져도 '상호 간섭 0' (독립 격리)
- "담당 부서", "긴급 대응 필요 여부", "고객 불만 점수"를 한 번에 물어봐도,
- 각 질문이 서로 영향을 주지 않도록 완벽히 분리된 상태에서 단 '1번의 Forward Pass'로 수십 밀리초 만에 동시 판별함.
3. 폐쇄형 유료 서비스 vs 100% 내 맘대로 학습하��� 오픈소스
- Jev는 입력 데이터 보관 정책 논란이 있고 커스텀이 불가능한 폐쇄형 API였음.
- Kev는 터보레포 창시자 Jared Palmer가 만든 Apache 2.0 완전 오픈소스임.
- 사내 업무 데이터 수백 개만 넣으면 우리 회사만의 규칙에 맞게 초고속 미세조정(LoRA) 가능.
4. 내 맥북에서 로컬 구동 (비용 0원 & 데이터 유출 제로)
- 0.8B, 4B, 9B 모델 가중치가 모두 공개되어 있음.
- 비싼 클라우드 API나 데이터 유출 걱정 없이, 내 로컬 환경(Mac M시리즈/CUDA)에서 무료로 무제한 구동 가능.
🔗 깃허브: https://t.co/3nAvNo844p
Si ya el código está resuelto, nada te impide expandir tu stack, estudiar cloud, software architecture, devops, y seguridad, es lo que diferencia la verdadera ingeniería de software del vibecoding.