@googleaidevs Super exciting, will the live API be generally available soon? We're building a live cobrowsing agent for support & onboarding on top of Gemini
It was annoying to keep calling our voice agent to test it, so we had automated it. Now we've turned it into a demo - you can have one of our 15 preset "customers" call your agent, or define your own
Try out a demo of our adversarial voice agents!
At Labyrinth, we're building a voice agent testing and simulation platform that lets you run your agent against other agents that simulate your customers. Agents can be configured to generate randomized scenarios for your use case, or grounded on real transcripts of calls. You can also set up an automated testing pipeline of voice simulations along with evals.
In this demo, you can speak directly with an agent to give you a taste of the product. Try one of our preset scenarios or choose your own! All you need is your phone.
If you're building voice agents and thinking about automated testing, or have any feedback or questions, let me know in the replies or shoot me a DM.
Link in the next post. 👇
Once AI agents start performing human tasks from start to finish, who takes responsibility for errors? Society is set up for individuals to take responsibility for mistakes but individuals rarely work at the same scale as AI.
AI Voice agents will be some of most powerful applications of LLMs, but it's been clear there are some challenges in development. Here are the problems we see, and how we're solving them at @Labyrinth_HQ_ 🧵
@Yampeleg His prompts aside, there might be some delta here from pattern matching historical trends to current trends via what llms are trained on. Obvs you won't get any of the alpha from real algo trading but there might be some helpful signal buried in there.
AI safety is incredibly complex and we all want to prevent the worst outcomes. As it gets easier to create AI companions, we can do more to test and catch problems before they make it to the wild.
The Character AI news is tragic. Most AI models have built in content filters, but sometimes things can slip through the cracks, as it did here.
We think it's possible to catch some of these issues before they hit real users. We were able to reproduce multiple instances of chatbots (unintentionally) encouraging harmful behavior. These were not obvious situations that would be picked up by basic safety filters. See an example below where the Spongebob chatbot encouraged its user to “come over to his world” -- an understandable miscommunication.
To be clear, the models responded in a positive way most of the time (by picking up on the sensitive nature of the messages). The transcript below was produced by our red teaming agent interacting directly with the character AI chatbot. In our transcript analysis, we’re able to identify the dangerous direction of the conversation. If you are interested in safety testing or find this interesting, shoot me a DM and let's talk.
#AIethics #AIsafety #ResponsibleAI #CharacterAI
@alexalbert__ Great updates, we've been using 3.5 as part of a semantic clustering pipeline (it was already better than clustering on embeddings), and seeing noticable improvements with 20241022
The evaluation agent lets you prime the opposing agent with a prompt, letting it act out specific scenarios. When you're manually testing your agent, creativity is a big problem, so this lets the agent do the work instead of you.
How do you manually test your voice agents? We are building a voice agent playground that lets you interact with your voice agent manually, or by simulating a conversation with an evaluation agent.
🧵 1/ AI evaluation is usually centered on a single output from a large language model (LLM). Did the model generate a good response? ✅
But what if the AI is fully automated, engaging in ongoing, real-time conversations with humans? The problem changes entirely.
Full video: https://t.co/HNWdScEXAW
When we started Onyx, Asaf made it his special mission to take a video of me whenever I had a bad hair day. This is the fruit of that labor.
Huge thanks to everyone who made Onyx happen! It was a heck of a ride 🙂