Inspired Cognition

[Inspired Cognition @inspiredco_ai]

Check out this new example of using Inspired Cognition's Critique text evaluation API to evaluate chatbots!

[Graham Neubig @gneubig]

There are so many chatbots nowadays, it’s hard to keep up!

To help out, we made an open source tool for automatic comparison of chatbots, and created a report on LLaMa, Alpaca, Vicuna, ChatGPT, Cohere, etc.!

Report: https://t.co/n3JfKYcIXB
Browser: https://t.co/kOK4YsNHBq

🧵⬇️

[Inspired Cognition @inspiredco_ai]

Finally, if you want to try it out yourself, try out our power analysis widget (featured in the video at the top of the thread): https://t.co/1XEYvUOZgB

You can just plug in the system accuracies and confidence levels, then get on to data creation!

[Inspired Cognition @inspiredco_ai]

To compare two AI systems on a task that you care about, you'll often need to create test data. This takes time and money, and often you find yourself asking "how much is enough?" Let us introduce "power analysis" a method that helps answer this question!

[Inspired Cognition @inspiredco_ai]

Then you run a power analysis, and it tells you how much data you need! If you want more details on how the statistics actually work, you can take a look at this nice resource here: https://t.co/ahQz0stVIE

[Inspired Cognition @inspiredco_ai]

We're joining the Japanese NLP society's meeting this week! Find our CTO @odashi_en to discuss our work on building reliable NLP systems.

言語処理学会 #NLP2023 に参加しています！CTOの@odashi_en にお声がけください。弊社の信頼できるNLPシステム構築への取り組みについてお話します。

[LangChain @LangChain]

A rather large community driven release today!

🦜🔗v0.0.106

📃Lots (5!) of document loader additions/improvements!
🍰PromptLayer <> ChatOpenAI integration
🧑‍⚖️Critique (evaluation) integration
🔱Fake embedding class (for dry-runs)
🌲Chroma server support

👇

[Inspired Cognition @inspiredco_ai]

Check out this example of how the Critique text evaluation toolkit can be incorporated into @langchain to evaluate document-augmented question answering! https://t.co/nPpPzxo2BA

[Inspired Cognition @inspiredco_ai]

Excited for the integration of Critique into @try_zeno! You can easily and visually compare different systems for text generation, seeing how they perform along different axes and examining individual high- and low-performing examples. Check the online demo!

[Alex Cabrera @a_a_cabrera]

Finding a library, figuring out the API, and setting up compute for ML evaluation is a hassle.
Instead, you can use @inspiredco_ai's Critique service to calculate metrics for generative NLP eval in 💠 Zeno!
Check out a demo comparing translation models!
https://t.co/laKC9oQ9rL

[Graham Neubig @gneubig]

Building systems with prompts is often ad-hoc, trying a few prompts, looking at the results, and guessing which one is good in practice.

To make this more systematic, I made a "prompt gym" notebook to compare different prompts, models, and metrics: https://t.co/TMcMRgVG5n

[Inspired Cognition @inspiredco_ai]

If there’s anything missing, any questions, or if you’d like to discuss with us about your next AI project please get in touch via:
🐦 Twitter messages or DMs
🤖 Discord https://t.co/uNTdlf09YS
💬 The chat or email buttons on the AI Guide https://t.co/XFglQvBNdS

[Inspired Cognition @inspiredco_ai]

There’s so much information about AI online that if you want to build something with AI, it can be hard to know where to start or which of many approaches to take…

Introducing the 💥Inspired AI Guide💥, a curated guide that walks you through step-by-step https://t.co/BFxcy0Yu9Y

[Inspired Cognition @inspiredco_ai]

Whew, that’s a lot of content! But we’ve tried to curate the information down to the most important things to know in AI-driven app development. We hope it’s helpful to you, and…