Oguz @kolpicious - Twitter Profile

almost 2 years ago

Over the past few months, we've been developing @driaforall , a multi-agent network for synthetic data generation. And today, we're excited to share an initial version of our Dashboard, allowing you to generate synthetic data easily, within minutes. https://t.co/QA90lflbNC We leveraged decentralization for bringing unique solutions to synthetic data generation: Massively Parallel Inference: Dria efficiently utilizes every node in the network, distributing data generation tasks in parallel. This accelerates generation as the network scales. Extensive Model Diversity: Our Node Runners operate over 20 different models, harnessing the unique strengths of each to enhance data quality. Scalable & Reliable Data Pipelines: Dria uses custom, well-crafted pipelines, scaling them across the network by selecting the right models with the lowest compute cost tailored to each task's needs. Agents with Integrated Tools and Web Access: Each node has access to tools that enable intelligent web searches for grounding, generating diverse synthetic data with real-life distribution. We bring this unique approach to solve the key challenges faced by researchers, developers, and engineers trying to generate synthetic data. Because we know that as we push the limits of human-generated data, synthetic data is becoming increasingly vital—for training and fine-tuning models, crafting few-shot examples, or evaluating performance when real-world data is scarce, sensitive, or difficult to obtain. The key challenges we've identified 👇 Cost: Experimenting with various methodologies and approaches can be time-consuming and expensive. Complexity: Building pipelines for multiple methodologies, especially across different models, is highly challenging. Diversity: Each model has unique characteristics, making it hard to find the best fit without employing all during generation. Distribution: Ensuring your dataset represents every perspective of your use case requires a well-balanced generation pipeline. Grounding: Without grounding, relying solely on a model's knowledge can lead to hallucinations. Finding reliable ground truth data is often difficult. This dashboard is just the beginning of our journey to leverage decentralization for state-of-the-art synthetic data. We'll continue to provide more tools with broader coverage in the coming weeks. So, how does Dria make a difference in practice? Let me share a personal experience. One of my favorite applications is evaluation. While we often rely on standard benchmarks like MMLU and TruthfulQA, there's a need for unique, tailored datasets to evaluate LLMs for specific cases. Recently, at the HACK UK 2024 hackathon organized by @a16z and @MistralAI , I worked on a medical agent assisting patients in diagnosing eye conditions. Before starting, I needed to evaluate models but couldn't find a high-quality dataset specifically for eye care. In just minutes, by uploading an ophthalmology educational book to Dria, I generated a diverse, well-crafted set of synthetic Q&A pairs to evaluate the models' medical knowledge. Given that many people in the tech industry suffer from Dry Eye, I chose to focus on addressing that speciality. Here are the evaluation results via @promptfoo using the 200 Q&A pairs on Dry Eye disease generated by Dria: Every model underperformed their performance in MedQA: You can visit https://t.co/9UfY57M60o to get your own high-quality diverse QA pairs generated in minutes from any file or website. And the best part? It's completely free. Unlike most tools, we're removing the barriers of data availability to empower more people with resources to advance AI models.

aktasbatuhann's tweet photo. Over the past few months, we've been developing @driaforall , a multi-agent network for synthetic data generation.

And today, we're excited to share an initial version of our Dashboard, allowing you to generate synthetic data easily, within minutes.
https://t.co/QA90lflbNC

We leveraged decentralization for bringing unique solutions to synthetic data generation:

Massively Parallel Inference: Dria efficiently utilizes every node in the network, distributing data generation tasks in parallel. This accelerates generation as the network scales.

Extensive Model Diversity: Our Node Runners operate over 20 different models, harnessing the unique strengths of each to enhance data quality.

Scalable & Reliable Data Pipelines: Dria uses custom, well-crafted pipelines, scaling them across the network by selecting the right models with the lowest compute cost tailored to each task's needs.

Agents with Integrated Tools and Web Access: Each node has access to tools that enable intelligent web searches for grounding, generating diverse synthetic data with real-life distribution.

We bring this unique approach to solve the key challenges faced by researchers, developers, and engineers trying to generate synthetic data.

Because we know that as we push the limits of human-generated data, synthetic data is becoming increasingly vital—for training and fine-tuning models, crafting few-shot examples, or evaluating performance when real-world data is scarce, sensitive, or difficult to obtain.

The key challenges we've identified 👇

Cost: Experimenting with various methodologies and approaches can be time-consuming and expensive.

Complexity: Building pipelines for multiple methodologies, especially across different models, is highly challenging.

Diversity: Each model has unique characteristics, making it hard to find the best fit without employing all during generation.

Distribution: Ensuring your dataset represents every perspective of your use case requires a well-balanced generation pipeline.

Grounding: Without grounding, relying solely on a model's knowledge can lead to hallucinations. Finding reliable ground truth data is often difficult.

This dashboard is just the beginning of our journey to leverage decentralization for state-of-the-art synthetic data. We'll continue to provide more tools with broader coverage in the coming weeks.

So, how does Dria make a difference in practice? Let me share a personal experience.

One of my favorite applications is evaluation. While we often rely on standard benchmarks like MMLU and TruthfulQA, there's a need for unique, tailored datasets to evaluate LLMs for specific cases.

Recently, at the HACK UK 2024 hackathon organized by @a16z and @MistralAI , I worked on a medical agent assisting patients in diagnosing eye conditions. Before starting, I needed to evaluate models but couldn't find a high-quality dataset specifically for eye care.

In just minutes, by uploading an ophthalmology educational book to Dria, I generated a diverse, well-crafted set of synthetic Q&A pairs to evaluate the models' medical knowledge. Given that many people in the tech industry suffer from Dry Eye, I chose to focus on addressing that speciality.

Here are the evaluation results via @promptfoo using the 200 Q&A pairs on Dry Eye disease generated by Dria:

Every model underperformed their performance in MedQA:

You can visit https://t.co/9UfY57M60o to get your own high-quality diverse QA pairs generated in minutes from any file or website.

And the best part? It's completely free. Unlike most tools, we're removing the barriers of data availability to empower more people with resources to advance AI models.

1

20

4

1

3K

Oguz

@kolpicious

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users