Top Tweets for #otsdata

Better data is better than better models. Most AI projects never graduate from the demo stage. This is now painfully obvious with Large Language Model applications. One of the main culprits is low-quality data full of incorrect edge cases. Models work by approximating their datasets. Give them garbage, and garbage you'll get. You can't out-train bad data. You have to deal with it. Check out this open-source project founded by a team from MIT: https://t.co/PRYzGr1oRX I've been a massive fan of Cleanlab for a while. I took their Data-Centric AI MIT class (public online.) Their tool is even better. It will help you curate any image, text, or tabular dataset and do it automatically. If you are building an LLM application, you can use Cleanlab as follows: • To improve the model inputs and outputs • To improve the dataset you are using to fine-tune the model Cleanlab will automatically detect the following: • Low-quality instructions and responses • Unsafe or poorly-written text • Any model outputs that aren't trustworthy • Any model outputs that look unrealistic The team invested in a lot of research to make this possible. Much of this research is published! Here is a blog post you must read. It shows how their tool can find and fix problems in any LLM instruction-tuning dataset: https://t.co/LUiyr1Oem5 If you are seriously working in LLM applications, Cleanlab is a tool you must check out.