Praneeth

Data Scientist

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

There are three primary grading strategies we can use when evaluating an LLM agent…

Human evaluation is the definitive source of truth for measuring an agent’s performance. We can start the human evaluation process with quick manual inspections (with researchers or internal users). In fact, the creators of Claude Code even began their evaluation process using internal tests with Anthropic employees. These simple tests capture high-level trends in agent capabilities, but we need to be more rigorous to collect high-quality human feedback at scale.

We start by creating a guideline / rubric that defines key quality dimensions and how they should be evaluated. Then, we calibrate the human evaluation process by collecting feedback, measuring agreement between humans, and refining our evaluation guidelines / process to improve agreement. Some tasks are more objective, but evaluation is usually subjective–human evaluators rarely agree with each other without investing significant effort into calibration.

Although it is our source of truth, human feedback can be difficult to scale, so automatic evaluation techniques are often adopted to make experimentation more efficient. We can roughly categorize automatic evaluation techniques into two groups:

1. Code-based: heuristic checks that can be captured in a Python function.
2. Model-based: open-ended checks that are based on an LLM judge.

Code-based graders include techniques like string matching and assertions on the agent’s outcome or transcript. For example, we can check if an agent’s final answer matches the ground truth or run predefined test cases for a coding problem. These are deterministic quality checks one can implement / execute with a Python function.

Code-based graders are efficient, reproducible, and easy to debug, but they also struggle to capture subjective aspects of agent behavior. For example, checking whether code written by an agent passes test cases can be handled with a code-based grader. Judging whether the code is sufficiently clean or elegant, on the other hand, is subjective. To grade such open-ended and subjective criteria, we need model-based graders.

The idea of a model-based grader is simple: we just prompt an LLM to evaluate some aspect of quality for us. This approach, known as LLM-as-a-Judge, is universally used in the evaluation process for both conventional LLMs and agent systems. To ensure the LLM judge is accurate, we should calibrate it by measuring agreement with human experts.

Multiple graders. No one grading technique is perfect. LLM judges are flexible but non-deterministic and subject to bias. Code-based graders are narrow. Human evaluation is expensive and difficult to tune. Therefore, we should use all of these techniques in tandem instead of relying on one of them.

Human evaluation serves as the core evaluation signal for an agent system. We can calibrate model-based graders based on their agreement with humans and use them to scale the evaluation process (while monitoring agreement with humans). Code-based graders can also be used for more deterministic or narrow checks that do not require model-based graders.

[Amy Pechacek @pechacek_amy]

1. https://t.co/nHpAGp4jwM

No code automation

On this site, there are bots to automate basically anything on the internet.

https://t.co/bmWFCeQMyT

[Soumith Chintala @soumithchintala]

Maybe I can finally use matplotlib now without spending half a day googling the exact syntax and options!

[TensorFlow @TensorFlow]

⚡️ Accelerating TensorFlow 2 performance on Mac

@Apple’s new Mac-optimized TensorFlow 2.4 fork lets you speed up training on Macs, resulting in up to 7x faster performance on platforms with the new M1 chip!

Learn how ↓ https://t.co/ggaQeduCWl

[Papers with Code @paperswithcode]

🎉 Papers with Code partners with arXiv! Code links are now shown on arXiv articles, and authors can submit code through arXiv. Read more: https://t.co/kO6zhWAWGH

[deva katta @devakatta]

This is brilliantly explained, applies to everyone of us!!

[Kaggle @kaggle]

'How to use Kernels and Forums to Win Kaggle Competitions' | A live presentation from #KaggleDays Beijing by Jian Qiao. [WATCH] (English subtitles) https://t.co/6vC3voO4SJ

[Goku Mohandas @GokuMohandas]

🔥 Excited to release practicalAI 2.0 - https://t.co/uobL1jvG0I
- 📚 Illustrative ML lessons in @TensorFlow 2.0 + Keras
- ⚒️ Build robust models using the functional API w/ custom components
- 📦 Train using simple yet highly customizable loops to build products fast

[ISRO @isro]

13 commercial satellites from USA successfully placed in their designated orbits
#PSLVC47

[Josh Gordon @random_forests]

Here are the slides from TF World (was so good to see everyone!):

https://t.co/N3gLj34mIS

[Jeremy Howard @jeremyphoward]

If you're interested in using @pytorch on free Colab TPUs, here are some notebooks to get you started
https://t.co/WvT7zW8xw9

[Jeremy Howard @jeremyphoward]

"PyTorch is becoming the deep learning framework of choice at Microsoft" - Big statement from @Microsoft at @PyTorch devcon

[François Chollet @fchollet]

I just wrote an extensive TensorFlow 2.0 + Keras overview, targeted at deep learning researchers: https://t.co/k694J95PI8

Hope you will find it useful! Let me know if you have any feedback.

[Sergey Levine @svlevine]

Want to learn deep RL? My deep RL course now has a permanent course number (CS285) and is being offered this semester:
https://t.co/tAGzTDfvUc
Lecture videos here (so far, we've gotten through most of model-free RL, model-based RL coming up next): https://t.co/u4z5kQCuVo

[PyImageSearch @PyImageSearch]

@fchollet I will be! All new tutorials will use tf.keras (was just waiting for the official TF 2.0 release). My Deep Learning for Computer Vision with Python book will be updated to cover TF 2.0 in early 2020 as well 😃

[François Chollet @fchollet]

TensorFlow 2.0.0 release is out now: https://t.co/s9tudRs6Kn

Huge milestone for the machine learning ecosystem. Congratulations to the hundreds (yes) of people who have contributed!

[Josh Gordon @random_forests]

TensorFlow 2.0 has been released! Congrats everyone, and thanks for making this possible.

Release notes: https://t.co/NSgSiswauO

[Jeremy Howard @jeremyphoward]

I believe this is the first time a dataset like this has been released - this is really exciting!

[Praneeth @praneeth26]

Docker compose tutorial for beginners by example [all you need to know] https://t.co/YLWXxt6024

[AI at Meta @AIatMeta]

We are open-sourcing a state-of-the-art deep learning recommendation model to help #AI researchers and the systems and hardware community develop new, more efficient ways to work with categorical data. https://t.co/kQL0pD8Akf @PyTorch