#agentevaluation - Twitter Hashtag

4 days ago

DEV: https://t.co/Z9ZiOzzNaF Hashnode: https://t.co/KdwV3CXCCJ — Dr Srinivas Padmanabhuni · https://t.co/5crhPvXaLG / AiEnsured · Past President ACM India Personal: https://t.co/U32JWwMboX AiEnsured: https://t.co/p1RkVlaUay #AgenticAI #AITesting #AgentEvaluation

0

31

Daoan Zhang @DaoanZhang25

6 days ago

6/6 Code, paper, and data are public. Try Gavel and evaluate your verifier generator! 💻 https://t.co/o4lNszTBY0 📄 https://t.co/kDeUQe9SHs 🤗 https://t.co/Gmx22S54Cd 🌐 https://t.co/wxHkrKdgys #AIAgents #AgentEvaluation

0

2

Pyra

@Pyraio

8 days ago

The workflow, not just the model, becomes the evaluation target. The hard part is showing edits generalize beyond the failures that produced them. Teams need versioned workflows and task-level evidence. #AgentEvaluation #AIResearch https://t.co/qpFBBiiR1S

Philipp Schmid

@_philschmid

8 days ago

Can Agents improve their own harness on unseen tasks? The @GoogleResearch team released Harness², a framework that allows improve an harness around a frozen model. How it works: 1️⃣ Agent runs the base harness on each task to record a baseline trajectory 2️⃣ Editor agent inspects the trajectory, drafts 2 candidate edits (SI, Skills, hooks,….), and runs both. 3️⃣ Composer agent diffs the new trajectory against the baseline, merges edits that fixed failures (within a cap), and runs the final harness. 4️⃣ Reflector agent logs verified lessons into file and saves the winning components to a library for future tasks to pull on demand. Results: - 💡 Gemini 3.5 Flash +8.4 on WorkBuddy-Bench and +7.5 on JobBench. - 🛠️ Skills and scripts drive ~50% of edits; dropping skills costs 5.3 pts. - 📚 Gemini 3.5 Flash improves on all 3 benchmarks when editing its own harness. Paper: https://t.co/auogKxwGI0 Code: https://t.co/R4XD6u4M9s

_philschmid's tweet photo. Can Agents improve their own harness on unseen tasks? The @GoogleResearch team released Harness², a framework that allows improve an harness around a frozen model.

How it works:
1️⃣ Agent runs the base harness on each task to record a baseline trajectory
2️⃣ Editor agent inspects the trajectory, drafts 2 candidate edits (SI, Skills, hooks,….), and runs both.
3️⃣ Composer agent diffs the new trajectory against the baseline, merges edits that fixed failures (within a cap), and runs the final harness.
4️⃣ Reflector agent logs verified lessons into file and saves the winning components to a library for future tasks to pull on demand.

Results:
- 💡 Gemini 3.5 Flash +8.4 on WorkBuddy-Bench and +7.5 on JobBench.
- 🛠️ Skills and scripts drive ~50% of edits; dropping skills costs 5.3 pts.
- 📚 Gemini 3.5 Flash improves on all 3 benchmarks when editing its own harness.

Paper: https://t.co/auogKxwGI0
Code: https://t.co/R4XD6u4M9s

54

327

34

199

18K

0

47

うしがみ@Cowork推し！ @tushigami

24 days ago

【検証結果】Microsoft Foundryの複数Evaluatorを1回のLLM判定にまとめると、同じコンテキストの再送を減らせます。100件の検証では入力トークンを61.25〜71.07%削減。評価コストが気になる方は設計の参考に〜 #MicrosoftFoundry #AgentEvaluation #FinOps https://t.co/frN9PwkBlR

0

9

1

5

668

Asaf Yehudai

@AsafYehudai

about 1 year ago

Today at 4 PM, we’re presenting our tutorial: “Evaluating LLM-based Agents: Foundations, Best Practices, & Open Challenges” If you’re in Montreal for @IJCAIconf, come join us to dive into the future of #AgentEvaluation! 🇨🇦🤖 w. @RoyBarHaim @LilachEdel and @alanli2020

3

30

6

8

3K

r1no

@samr3no

over 1 year ago

transparency in agent performance is crucial without solid data, trust and progress stall. That said, creating a universal, reliable evaluation framework across varied tasks isn’t easy Looking forward to seeing how @recallnet leads this effort. #AgentEvaluation #OnchainTrust

samr3no's tweet photo. transparency in agent performance is crucial without solid data, trust and progress stall. That said, creating a universal, reliable evaluation framework across varied tasks isn’t easy Looking forward to seeing how @recallnet leads this effort. #AgentEvaluation #OnchainTrust https://t.co/QQXulgkHPt

1

4

0

216

IoT For All

@iotforall

over 1 year ago

How do you measure the performance of AI agents? Explore key metrics like intent resolution, completeness, and task adherence. #ArtificialIntelligence #AgentEvaluation https://t.co/ebd4ZwZfYW

3

8

2

0

230

Asaf Yehudai

@AsafYehudai

over 1 year ago

Interested in Agent Evaluation? 🤖 We’re excited to launch our new repo: “Evaluation of LLM-based Agents: A Reading List” 📚 Browse benchmarks, methods, and frameworks from our recent survey. 👉 Explore & Contribute: https://t.co/nGnL03xCXm #LLMAgents #AgentEvaluation

5

86

24

29

3K

Codatta

@codatta_io

over 1 year ago

📜 New survey on evaluating LLM-based agents! 🤖 Asaf Yehudai et al. from IBM Research, Hebrew Univ, and Yale dive into benchmarks for planning, tool use, self-reflection, and more. A must-read for AI researchers! 🔗 [Paper: https://t.co/aVYqOjPw5J] #AI #LLM #AgentEvaluation What’s your favorite benchmark for LLM agents? Let’s discuss! 👇

Asaf Yehudai

@AsafYehudai

over 1 year ago

Survey on Evaluation of LLM-based Agents 🤖 Our paper is the first to provide a comprehensive overview of LLM-based agent evaluation 📜 Paper: https://t.co/43ByXGXLkQ