New Short Course: Getting Structured LLM Output!
Learn how to get structured outputs from your LLM applications in this course, built in partnership with @dottxtai, and taught by @willkurt, a Founding Engineer, and @just_cameron , Developer Relations Engineer.
It's challenging for software to automatically parse through an LLM's freeform text outputs. Structured outputs—like JSON—solve this by converting natural language into consistent, clear, data that a machine can read and process. This course teaches you how to generate structured outputs while building several use cases, including a social media analysis agent.
You’ll learn about structured outputs and efficient ways to generate outputs in your defined schema or format. You’ll begin by using structured output APIs, then use re-prompting libraries like “instructor” to generate structured output. Finally, you’ll learn how constrained decoding works; this is a very clever technique in which constraints are applied on each subsequent token generated, blocking any tokens that don’t fit your defined schema.
In detail, you’ll:
- Learn why structured outputs are important, how they allow for scalable software development, and the different approaches to generate them, including vendor-provided APIs, re-prompting libraries, and structured generation.
- Build a simple social media agent using OpenAI’s structured output API, learn how to define a model's desired structured output using Pydantic, and perform basic programming with your outputs, such as importing structured data into a data frame using pandas.
- Learn how to use the open-source library "instructor," which checks the structured output of the model and re-prompts the model until it validates the desired output, and explore the limitations of this approach.
- Understand how structured generation by the “outlines” library works by modifying LLM logits, on a per-generated-token basis based on the desired format, to give a particular output structure.
- Learn how regular expressions, which outlines works with, are represented as finite-state machines, and how they can be used to develop a range of structured outputs beyond JSON.
By the end of this course, you’ll have broadened your knowledge of the approaches you can use to get structured outputs from your LLM applications.
Please sign up here: https://t.co/3k53vgEFj3
A new paper, "Let Me Speak Freely" has been spreading rumors that structured generation hurts LLM evaluation performance.
Well, we've taken a look and found serious issues in this paper, and shown, once again, that structured generation *improves* evaluation performance!
We’ve been cooking with @huggingface and just released a Rust port of Outlines’ structured generation.
👉 Faster compilation
👉 Lightweight library (poke @vllm_project)
👉 Bindings in many languages
We can’t wait to see what you will build! Blog 👇
https://t.co/6GaeugXRC0
Introducing @dottxtai
The solution for AI's fundamental flaw: unstructured answers are great for humans but not for businesses.
Finally, LLMs can speak computers.
👉 Structured generation beats GPT-4
Using structured generation, phi-3 achieves 95.5% accuracy when it only achieves 86% without structured generation.
More importantly, it beats GPT-4 (93.5%) by a whopping 2 percentage point. 🔥🔥🔥
Made some updates to our previous post on grammar-structured generation: https://t.co/BWxYWofBei
This version drives home the scalability point by using CodeGemma and its exceptionally large vocabulary. We can handle cases like this for large grammars using sequential Python.
⚠️ Efficient grammar-structured generation was achieved internally
You know from @OutlinesOSS that it is possible to achieve virtually free structured generation for regular expressions. However, the flexibility of context-free grammar is often needed in practical use cases. 🧵
✨ Structured generation improves performance ✨
🔎 We evaluate 8 models on the GSM8K benchmarks using (regex) structured generation and base models
👉 Structured generation systematically outperforms base models.
👓 https://t.co/DkDqvCwpft
⚡️ Speed up LLM inference by 5x. ⚡️
We introduce a new framework, coalescence, that makes structured generation several times faster than standard generation.
Coalescence is very flexible, and raises unexpected questions 🧐
https://t.co/uiIqj9nRVa
I’m available for contract work starting Jan 2. I work in the Bayesian space, author of @blackjax_mcmc and core member of @AesaraDevs
In my previous gig I worked on scaling Bayesian inference on a big HMM to 1.7 trillion samples per month using @AesaraDevs.
😊 RTs appreciated
A deep dive into the technical architecture we use at @MeetAmpersand to estimate and serve 200,000+ fully Bayesian time series models on AWS. Our models are built using @AesaraDevs.
@jeffreydenos@BrandonTWillard@AbunijemAla and PY Aquilanti.
https://t.co/obsRTmHxjo
I'm pleased to announce that the #rstats package 'strand' is now on CRAN. It's for running realistic simulations of factor-based investment strategies. Feedback and questions welcome!
https://t.co/gUP3w3gE1S