10 agent evals every AI engineer should know
1) golden set
a frozen set of cases you run after every prompt model or tool change
use it as your baseline to see whether the system improved or quietly broke
OpenAI Evals helps you build repeatable benchmark sets and compare model changes
→ https://t.co/dr1GZlC75R
2) llm as judge
a second model scores open ended answers against a written rubric
use it when there is no exact output to compare against
OpenEvals provides ready made evaluators for LLM applications
→ https://t.co/S2yhnByFIP
3) rubric scoring
score correctness tone safety and cost separately
one quality number hides the problem
DeepEval helps you create custom metrics and score every dimension independently
→ https://t.co/q9Z6Xmixia
4) trajectory eval
grade the path the agent took
not only the final answer
AgentEvals checks agent actions decisions and tool calls across the full trajectory
→ https://t.co/0oziAl54az
5) tool unit tests
test every tool with fixed inputs and outputs
no model in the loop
MCP Inspector helps you inspect and test MCP servers tools and responses separately
→ https://t.co/IVmt5qpWIN
6) regression suite
replay previous runs against every new prompt model or toolset
then compare the results
Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI
→ https://t.co/zxi2PuWuhe
7) a b in prod
split real traffic between two versions and compare actual outcomes
GrowthBook provides feature flags controlled experiments and product analytics
→ https://t.co/DGlE3JjDD3
8) human review
sample real runs and let a person grade them honestly
use it to calibrate your automated judge
Argilla helps teams collect human feedback review outputs and build better datasets
→ https://t.co/QHWb7skWjr
9) shadow run
let the candidate run on real traffic while its output is shown to nobody
use it before a risky rollout
Langfuse helps you trace production runs compare candidates and monitor eval results
→ https://t.co/IrhDf38tRn
10) red team
attack the system before somebody else does
jailbreaks prompt injection data leaks and tool abuse
Garak scans LLM systems for vulnerabilities and unsafe behavior
→ https://t.co/w8ObyW4ZKv
offline evals tell you it works
online evals tell you it still works
you probably do not need all ten today
start with the two that would have caught your last outage
bookmark this
Awesome Android Root provides a comprehensive collection of over 500 root applications and modules for Magisk, KernelSU, and LSPosed, alongside step-by-step guides to root various Android devices.
https://t.co/W4l24Nk74R
someone open-sourced a tool that turns any phone video into a full 3D world in real time.
you record a walk around your house, drop the mp4 in, and the entire scene comes back as a walkable 996k-point world at 60fps on your local GPU.
no cloud. no subscription. runs 100% locally.
100% open source.
ANTHROPIC AND OPENAI BOTH MISSED THIS.
Someone just open-sourced one SDK that gives AI agents a mouse, keyboard, and screen across macOS, Linux, and Windows.
It's called CUA.
Same code drives a local VM or a cloud sandbox regardless of the OS. Your agent sees the screen, clicks, types, and runs shell commands like a real user.
> Benchmark against OSWorld and ScreenSpot
> Export trajectories for model training
> Near-native performance on Apple Silicon
No separate SDK per OS. No infrastructure headache.
100% Free. Open Source.
Built for agencies and solopreneurs, ai-marketing-claude brings 15 specialized skills to Claude Code. Users can run full marketing audits, generate copy, create content calendars, and produce client-ready PDF reports entirely through terminal commands.
https://t.co/ASlBCBir7O
Family Trip Command Center is an open-source dashboard that manages group travel logistics through a command-center interface. The tool replaces standard planning methods by tracking convoy routes, simulating arrival times, and coordinating meals and activities for multiple families traveling together.
https://t.co/iU28zCJotH
Generates 96,000 high-quality deep research trajectories and provides a fully open-source recipe for training agentic language models on long-horizon web research tasks without external API dependencies.
https://t.co/zUo5XOpZBb
A list of open source tools for deploying your own VPN, music streaming service, photo hosting, and more on personal infrastructure.
https://t.co/oebhnTGo47
ANOTHER MASSIVE WIN FOR LOCAL AI 🚀
ICYMI @kyutai_labs quietly dropped a super lightweight Text-to-Speech application designed to run efficiently on CPUs.
Only 100M parameters. No GPU needed. Zero API fees 🤯
It completely bypasses the usual token-transformer bottlenecks to deliver wildly fast performance:
→ 6x faster than real-time on a Mac
→ ~200ms latency to the first audio chunk
→ Voice cloning from just 5 seconds of audio
→ 6 languages out of the box
Best of all?
It's 100% free and open-source.
MIT licensed and trained exclusively on public data.
Repo in 🧵↓
Alguien creó un Simulador de Diseño de Sistemas basado en web.
Copias y pegas (API gateways, bases de datos, cachés) y simula tráfico en tiempo real.
Puedes ver la latencia, los puntos de saturacion y los fallos mientras ocurren en directo...