Zeno

AI evaluation platform

[Zeno @try_zeno]

We've teamed up with @AiEleuther to make it super easy to visualize your evaluation results in Zeno!

Try it out the next time you run a benchmark: https://t.co/XLWQIj6bUM

[TwelveLabs (twelvelabs.io) @twelve_labs]

@a13xba @cmuhcii @a13xba will give a presentation about @try_zeno, an interactive AI evaluation platform for exploring, debugging, and sharing how your AI systems perform. (co-founded with @a_a_cabrera)

https://t.co/FqoLXwrUHJ

[Zeno @try_zeno]

We just sent out the first issue of Zeno's Notes, our newsletter on AI evaluation.

In case you're not on the recipient list yet, read it here: https://t.co/D6UCKpqFVs

[Alex Cabrera @a_a_cabrera]

Lots of predictions of synthetic data for AI being big this year. Decided to look at the OG Alpaca dataset:

https://t.co/ab2x6PK6vs

Impressive for being GPT-4 generated w/ 1 prompt, but begs the question of how to generate diverse, OOD data

[Slator @slatornews]

Researchers from @CarnegieMellon, BerriAI explore the translation capabilities of Google’s Gemini and suggest Gemini Pro could be a valuable tool for MT.
@SNAT02792153 @yu_zichun52802 @AashiqMuhamed @tianyue_01 @a13xba @a_a_cabrera @krrish_dh @XiongChenyan
https://t.co/WuL24c0xOW

[Alex Cabrera @a_a_cabrera]

In case you missed it over the break - you can now visualize the outputs of any Eleuther LM Eval Harness run in @try_zeno with one command!

𝚙𝚢𝚝𝚑𝚘𝚗 𝚜𝚌𝚛𝚒𝚙𝚝𝚜/𝚣𝚎𝚗𝚘_𝚟𝚒𝚜𝚞𝚊𝚕𝚒𝚣𝚎.𝚙𝚢

[Graham Neubig @gneubig]

Google’s Gemini recently made waves as a major competitor to OpenAI’s GPT. Exciting! But we wondered:

How good is Gemini really?

At CMU, we performed an impartial, in-depth, and reproducible study comparing Gemini, GPT, and Mixtral.

Paper: https://t.co/S3T7ediQLa
🧵

[Zeno @try_zeno]

💠

[Greg Brockman @gdb]

evals are surprisingly often all you need

[Shuyan Zhou @shuyanzh36]

Since the initial release, we have significantly improved the usability of WebArena, accuracy of the evaluation, and provided interactive result analysis with @try_zeno

I am attending #NeurIPS2023 , say hi 👋 if you are interested in AI agent, code gen and their evaluations!

[Alex @a13xba]

Since some of you might be wondering whether Mamba 2.8B can serve as a drop-in replacement of some of the larger models, we've compared the Mamba model family to some of the most popular 7B models in @try_zeno

Report: https://t.co/1qyHTETcfS

🧵 1/5

[Graham Neubig @gneubig]

Recently there were some great results from the new Mamba architecture (https://t.co/0yD3bjGD7y) by @_albertgu and @tri_dao. We did a bit of third-party validation, and
1. The results are reproducible
2. Mamba 2.8B is competitive w/ some 7B models (!)
3. Mistral is still strong

[MMitchell @mmitchell_ai]

Useful thread wrt understanding Google's new Gemini reported results.

[Alex Cabrera @a_a_cabrera]

Google just released 𝑮𝒆𝒎𝒊𝒏𝒊, their long-awaited GPT-4 competitor.

Their report shows comparison across multiple common benchmarks, but 𝐡𝐨𝐰 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐫𝐞 𝐭𝐡𝐞𝐬𝐞 𝐫𝐞𝐬𝐮𝐥𝐭𝐬?

🧵 on potential issues with the benchmark scores

[Alex @a13xba]

Awesome blogpost by our friends @huggingface and @AiEleuther and a demonstration of how @try_zeno can be used to systematically spot issues with benchmark results. Give it a read!

Also:
Zeno Report: https://t.co/mm1kLssDc1
Zeno Project: https://t.co/x9XI5Q9Qed

[Teknium 🪽 @Teknium]

Lots of info here on the several problems benchmarks can face

[Alex Cabrera @a_a_cabrera]

We loved collaborating with the @huggingface and @AiEleuther teams to investigate the odd behavior on the DROP benchmark! Check out the blog post and supporting Zeno report & project:

Report: https://t.co/LeK3yjujmg
Project: https://t.co/JEHX4gzRR7

[Clémentine Fourrier 🍊 is off till Dec 2026 (🪂) @clefourrier]

⚠️ We are removing DROP from the Open LLM Leaderboard!

With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱

https://t.co/OExqyuD9RE

[Alex Cabrera @a_a_cabrera]

𝐂𝐚𝐧 𝐀𝐈 𝐝𝐨 𝐲𝐨𝐮𝐫 𝐭𝐚𝐱𝐞𝐬? Probably not 💸

Quick @try_zeno report on @danielgross' benchmark of tax Qs. LLMs struggle with any math, and it's hard to validate text answers without external references

Report: https://t.co/zQROuxvER6
Explore: https://t.co/cOH1QEpw9j

[Zeno @try_zeno]

Happy Thanksgiving!

If you don't have one already, there's some turkey on Zeno: https://t.co/RsKUW1YxHV

[Zeno @try_zeno]

Zeno now supports 3D 🧊 data!

We've uploaded over 1M @allen_ai ObjaverseXL models to a Zeno project to showcase how you can explore 3D data in a Zeno Project: https://t.co/qbYsobtkkP

[Zeno @try_zeno]

Zeno now supports 3D 🧊 data!

We've uploaded over 1M @allen_ai ObjaverseXL models to a Zeno project to showcase how you can explore 3D data in a Zeno Project: https://t.co/qbYsobtkkP