Top Tweets for #DecentralizedArena

almost 2 years ago

🚀🚀🚀 Finally releasing #DecentralizedArena #DeArena during the amazing week of #COLM2024! Imagine a new paradigm of AI evaluation🤔: Can we have all LLMs govern themselves for a faster, smarter, and unbiased system to benchmark LLMs (and even future superintelligence models)? 🤖 Reflect the current LLM benchmarking system Let's be realistic: traditional accuracy/dataset-based benchmarking systems have broken in the #LLM era. Human judgment and democracy seem to be the only trustworthy oracles, hence, the great Chatbot Arena https://t.co/nAXB0z5mUv. But lately, people become obsessed with GPT-4-based evaluation 😶. Sure, it's fast, and yes, you can publish more, but we're all aware of the risks and biases of trusting a single (or a few) model. Numerous papers have pointed out these biases (e.g., https://t.co/U2nMkv0bAS; https://t.co/BoavmQ4svk; https://t.co/Ow6DPgQecq) A natural question is that, can we train a better judge model to approximate human judgments? It's doable, and some folks are trying (e.g., Prometheus https://t.co/wGZvyJknKM, SFR-LLaMA https://t.co/9DmluoAx3J). But here's the kicker: democracy doesn't fit in a single model. That's like asking the king to write the constitution for a democracy. Weird, right? 🏟️ Enter the era of #DeArena Instead, why not have all LLMs judge themselves? It's automated, scalable, and—get this—less biased. It's democracy at scale, powered by collective LLM intelligence. Welcome to a more transparent, autonomous, and reproducible future in AI evaluation. 🔧 Overcome the technical challenges There’s no free lunch when it comes to democracy. Using all LLMs to judge each other sounds compute-heavy (N^3 complexity for pairwise judgments with N models). Here comes the core technical novelty of DeArena, which is a sorting-based algorithm to sample pairs of models to compare and maximally distinguish models with close capabilities. It uses a coarse-to-fine-grained process to reduce the complexity to Nlog(N)! More details are in our blog https://t.co/abiYdOMFKB and upcoming papers. For perspective, this novel algorithm can rank a new LLM in an incredible fast way. With a practical estimation, it takes about 1 week for Chatbot Arena to collect enough human votes to rank o1 models https://t.co/6Iw0Xxbhcq , while #DeArena can take about 2 days to rank the o1 with 100 LLMs. 🚀 Future of #DeArena: Scalable Oversight for Superintelligence? At first glance, #DeArena is a fast version of Chatbot Arena (much faster of course), but what makes it extremely exciting is that it pictures a future for scalable oversight and benchmarking superintelligence systems. When humans can no longer reliably evaluate models, we’ll need to (or can only?) trust a democratic system where all LLMs participate in their own governance. Already, #DeArena is showing it can rank LLMs on extremely challenging questions—those requiring expert-level human judgment, which, let’s face it, is both expensive and scarce. 🌐 Open-source commitment and involvement Exciting times ahead! Stay tuned as we continue to expand #DeArena’s capabilities, covering even more evaluation dimensions while improving the algorithm for greater scalability and precision. 🚀 In the coming months, we'll be releasing more results, papers, and open-source tools to help the community better benchmark LLMs across any user-defined dimensions and understand their true boundaries. This is just the beginning of a more democratic and robust AI evaluation paradigm. 🔮 Join us on this journey to shape the future of AI oversight and benchmarking—together, we’re redefining how we trust and measure intelligence. 🌍💡

Maitrix.org

@MaitrixOrg

almost 2 years ago

With @llm360, we're thrilled to release ⚔️Decentralized Arena built on Collective LLM Intelligence 🤖🤖 An automated "Chatbot Arena" for democratic LLM benchmarking on any fine-grained dimensions, e.g.: 🧮(Math) algebra, geometry, probability ... 💡(Reasoning) symbolic, social ... 🧪(Science) biology, chemistry ... ⌨️(Coding) python, C++, SQL ... ❓any other dims you care about! 🔥95% correlation with Chatbot Arena in "Overall" dimension (on 50+ models) Key idea: 👉Chatbot Arena ⬅️Wisdom of the human crowds 👉Decentralized Arena ⬅️Wisdom of the LLM crowds With @llm360, we're working to make Decentralized Arena fully transparent and reproducible Blog: https://t.co/AmBQqtMztO Leaderboard: https://t.co/0a9O520PQO More interesting results: 1/

24K

Top Tweets for #DecentralizedArena

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users