John Lear and his coordinates to Huge Buried UFO [38°37'40.0"N 113°40'40.0"W]
Ex-CIA Pilot Said US Buried Huge UFO That Was Too Big To Move, Corroborating Coulthart Claim.
10 agent evals every AI engineer should know
1) golden set
a frozen set of cases you run after every prompt model or tool change
use it as your baseline to see whether the system improved or quietly broke
OpenAI Evals helps you build repeatable benchmark sets and compare model changes
→ https://t.co/dr1GZlC75R
2) llm as judge
a second model scores open ended answers against a written rubric
use it when there is no exact output to compare against
OpenEvals provides ready made evaluators for LLM applications
→ https://t.co/S2yhnByFIP
3) rubric scoring
score correctness tone safety and cost separately
one quality number hides the problem
DeepEval helps you create custom metrics and score every dimension independently
→ https://t.co/q9Z6Xmixia
4) trajectory eval
grade the path the agent took
not only the final answer
AgentEvals checks agent actions decisions and tool calls across the full trajectory
→ https://t.co/0oziAl54az
5) tool unit tests
test every tool with fixed inputs and outputs
no model in the loop
MCP Inspector helps you inspect and test MCP servers tools and responses separately
→ https://t.co/IVmt5qpWIN
6) regression suite
replay previous runs against every new prompt model or toolset
then compare the results
Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI
→ https://t.co/zxi2PuWuhe
7) a b in prod
split real traffic between two versions and compare actual outcomes
GrowthBook provides feature flags controlled experiments and product analytics
→ https://t.co/DGlE3JjDD3
8) human review
sample real runs and let a person grade them honestly
use it to calibrate your automated judge
Argilla helps teams collect human feedback review outputs and build better datasets
→ https://t.co/QHWb7skWjr
9) shadow run
let the candidate run on real traffic while its output is shown to nobody
use it before a risky rollout
Langfuse helps you trace production runs compare candidates and monitor eval results
→ https://t.co/IrhDf38tRn
10) red team
attack the system before somebody else does
jailbreaks prompt injection data leaks and tool abuse
Garak scans LLM systems for vulnerabilities and unsafe behavior
→ https://t.co/w8ObyW4ZKv
offline evals tell you it works
online evals tell you it still works
you probably do not need all ten today
start with the two that would have caught your last outage
bookmark this
Beginning July 20, Claude Fable 5 will be included in all Max and Team Premium plans, at 50% of limits.
Pro and Team Standard users will continue to have access to Fable via usage credits, and will receive a one-time $100 credit.
Demand for Fable has been challenging to predict, which is why we rolled it out to subscription plans in stages, extending access several times as we secured additional capacity.
Karpathy said something you'll regret ignoring:
"Remove yourself as the bottleneck. Maximize your leverage. Put in very few tokens, and a huge amount of stuff happens on your behalf."
Loop engineering is the exact thing that does that.
In a hand-run session, the operator handles two things:
- deciding what the agent runs next
- and checking its output before the next step
Both are manual, and both decide how far the agent gets on its own without the operator.
Loop engineering moves both steps into the system.
A core operating structure surrounds the loop, and the diagram below depicts it.
- A schedule decides what to run
- Loop is the maker that produces the work
- A separate checker agent grades the output
- A file on disk holds the state they both read.
The loop runs until either done, max iterations, or an exhausted budget.
Here are some practical engineering considerations:
1) A model grading its own output justifies what it already did instead of catching where it failed.
That's why a separate checker's findings return to the maker as the next instruction. And the cycle repeats until the checker finds nothing left to fix.
2) A loop with no stop condition burns tokens, and the cost climbs fast once sub-agents and long runs add up.
That's why the exit must be set before the loop runs, not while it is running.
A simple exit could be:
↳ fix only the major issues, run one final pass, and stop after two loops, with "all tests pass and lint clean" as the rule that ends it.
3) State has to live on disk, not in context.
The model forgets everything between runs, so an MD file or a knowledge graph holds what is done and what is still open.
Each run reads it and writes back to it, which lets a loop pick up again after days.
4) The lower the verification bar, the safer the loop.
Boring, repetitive checks like a stale version string or a missing test are trivial to verify, so a loop runs them with little risk while the operator is away.
Judgment-heavy work is loopable too, but only as far as the checker can confirm the result.
Let's look at how an unattended loop fails in two ways.
1) It reports done when nothing is actually verified.
The separate checker exists to prevent it, but it merges code faster than anyone reads it, so over weeks, the team stops understanding its own codebase while every check stays green.
Green tests say the code passed the tests, not that anyone knows what shipped. Someone still has to read what the loop merges.
2) The checker keeps a running loop honest, but it only catches failures inside a run.
The harness around the loop, like the prompts, tools, and checks wrapped around the model, still drifts and breaks in production as models change.
That repair loop is usually run by hand based on observability traces.
My co-founder wrote a detailed walkthrough (with code) on making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it cannot recur.
Read it below.
The market is down, but the vibes were way up ⬆️ in San Diego last night
I took this candid picture as we were prepping for the main pose, to capture the buzz in the room, with the cross-chain community we’ve become a part of and done our best to contribute to in SoCal
Shoutout once more to @VDizzle777 for hosting yet another awesome @Pizza_DAO Pizza 🍕 Day Event.
(And to @ozthecryptogoat & a few others who caught me and posed! 🤣)
Imagine a world where you can spend BTC to buy groceries, pay for gas, pay an Uber, or treat someone to a fancy tea party.
No banks, no borders, just freedom.
That is the power of blockchain technology, and that is exactly why I joined this movement.
A huge thank you to Laszlo Hanyecz the legend who dared to make the first move back in 2010, spending 10,000 BTC on two pizzas and changing history forever. We stand on the shoulders of that courage every time we celebrate Bitcoin Pizza Day. 🫡
And a special thank you to @Pizza_DAO & partners @brave, @ensdomains, @WrldPizzaChamps for honouring this iconic moment and bringing the global crypto community together through the universal language of pizza. What a way to keep the legend alive! 🍕🔥
This box is a reminder that Kisumu was part of the history.
pov:would probably make it a mintable NFT on
@RarePizzas
Alguien acaba de pulverizar a toda la industria de assets 3D.
Lo que en 2023 ocupaba a un artista 3D senior durante tres semanas, hoy lo haces tú solo en una tarde de sábado sin abrir Blender una sola vez.
El stack completo:
→ Concept art generado en 30 segundos con nano banana o GPT image
→ Conversión imagen → malla 3D con Hunyuan3D, Tripo o Meshy
→ Limpieza de topología + auto-rigging en Meshy o Tripo
→ Animaciones instantáneas vía Mixamo
Te dejo de prueba un capibara hecho íntegramente con este pipeline. Modelado, riggeado, animado. Game-ready para Unity o Unreal.
Cero artistas contratados. Cero líneas de código. Una tarde.
La barrera de entrada a la industria del videojuego acaba de caer al suelo.
@heygurisingh fact and fact. another fact is that I've shipped two enterprise apps with my agentic system has shipped full-stack/front/back end eng in the last 90 days. Code is a commodity. product builders are here to stay, and there is room for both.