El tema de los benchmarks públicos es que muchos modelos han aprendido de forma directa o indirecta el contenido necesario para poder pasar esos benchmarks (problema de Data Leakage). Pero si en tus tareas rutinarias tienes bien definidos tus objetivos, eso será más fiable
Prueba de crearte un dataset de queries y outputs de tareas concretas. Puedes evaluar modelos y buscar el que te da mejores resultados. Los benchmarks públicos te pueden servir como heurística pero es mucho mejor si te creas tu propio dataset de evaluación.
Me está costando mucho saber qué modelos usar. Glm5.3-flash como orquestador y ds4.1-flash como subagente funciona.
Pero luego le pido a gemma4 conectarse por ssh a una máquina y cambiar una config, y lo hace perfectamente.
Supongo que lo ideal es tener tu propio benchmark para no fiarte de los números benchmaxxeados. Tenéis vosotros algún benchmark propio?
si eres parte de NaN, esta noche se duerme poco 👀 nuevos modelos muy punteros, nuevo sistema de créditos por tokens, y un reset gratuito que a dia 10 del mes también se agradece. joder, esto del tokenmaxxing es adictivo, cuidao chavales
Being an AI Research Engineer is fun, but it is also addictive. You must balance, when you must dedicate your time to theoretical knowledge and when you must jump onto more applied things
Кто-то должен был это сделать, somewhat accurate.
Uses words sparingly – Ну да, шутка детства – говорим мы тихо потому как едва рот открываем, либо зубы простудишь либо проредят.
The science of aging with one of its top researchers, Vadim Gladyshev @gladyshev_lab, and optimism for slowing biological aging
with transcript, open-access, w/ links to his many recent papers
https://t.co/dZS3SSKpcn
RAG is officially cooked.
IBM built a retriever that hallucinates 65x less than fine-tuned RAG systems.
Right now, every company is using Retrieval-Augmented Generation (RAG) to stop their AI from hallucinating.
The idea is to give the AI a private database to read so it doesn't make things up.
But there is a massive structural flaw.
Standard RAG blindly grabs documents based on vector similarity and forces the AI to read them. If the retriever pulls irrelevant garbage, the AI doesn't stop.
It just hallucinates with citations.
IBM just dropped a paper that changes the math entirely.
They built a new retrieval architecture that hallucinates 65x less than state-of-the-art fine-tuned RAG systems.
65x is the difference between a fun prototype and a production-grade enterprise system.
Instead of blindly dumping context into the model and hoping for the best, their system actively filters and validates the retrieved data before the generation step even begins.
It cuts out the noise. It forces absolute precision.
For businesses, this is the holy grail.
Zero hallucinations means zero liability. It means autonomous agents can finally be trusted with sensitive company data without a human babysitting every output.
@samu2kdotcom Also, many companies require strict policies to be enforced in terms of where the model is located. An example is Mistral which is becomimg widely popular just because it is located inside the EU