Nico Tonozzi

work: engineering @freeplay_ai, @height_app, @twitter and @sourcegraph
fun: skiing, running and biking in the mountains and desert

[Nico Tonozzi @ntonozzi]

This is the only benefit of prompt-crafting prompts -- they show humans what they can and should be specifying in the prompts. It lets users see how underspecified their initial prompts are, shortcutting a bunch of trial and error you usually need to go through in production.

[Shreya Shankar @sh_reya]

not only do people want to just see what the LLM can do, but they also don't fully know what they are supposed to say in the prompt, or what answer they want (they won't know it until they see it).

I think of a prompt as a form you need to fill out to submit your task to the LLM, and the fields of this form are unknown and dynamic (i.e., task-specific). a prompt writing tool can make these fields more known

[Wyatt Walls @lefthanddraft]

Claude is thrown into confusion when I tell it to roleplay an overly censorious chatbot and it tries to refuse

Catches itself mid sentence

[Nico Tonozzi @ntonozzi]

I built this! It's so useful to be able to understand and limit your LLM spending by feature or prompt.

[Ian Cairns @cairns]

The boring parts of LLM ops are important. Like managing spend. 💸

Our team @freeplay_ai cooked up some new usage dashboards and spend management controls to help.

Now development teams can easily track and control AI costs across providers, and how money gets spent.

[Ian Cairns @cairns]

The AI Builders meetup is going on the road!

Come join us in LA for Tech Week on 10/14. We'll be hosting with our friends @ThisIsArtium and @SiliconVlyBank.

There will be demos from local startups & big companies -- come see what folks are building in LA. 🌴🤖

These events are always packed, RSVP below.

https://t.co/SuN8EuD45j

[Nico Tonozzi @ntonozzi]

https://t.co/jbUjIu8ere has gotta be the best post of the decade

[Nico Tonozzi @ntonozzi]

How has OpenSearch not been sued out of existence when it's obviously decompiled Elastic plugins?

[Jo Kristian Bergum @jobergum]

So Amazon dropped OpenSearch to the Linux foundation AND announced RTO in the same week or did I get it wrong!

[Romain Huet @romainhuet]

Super sleek AI automating all the project management grind, built on @OpenAI!

The UI is stunning, and watching the @height_app agent triage and edit in real time is awesome! ✨

[Emmett Shear @eshear]

This should be a wakeup call. Scary shit.

[Ian Cairns @cairns]

🧵 Everyone's looking for signal in all the AI noise... There’s no better way to learn than from teams who are already doing it!

So today we’re launching our new podcast to tell these stories. 🎙️

Episode 1 of Deployed features our friend, @jeffseibert, CEO & Co-founder of @digits. 🎉

Here's a quick preview: Why it's worth it to review lots of data by hand.

[Nico Tonozzi @ntonozzi]

@cairns Can we name ours 'hallucination'?

[Nico Tonozzi @ntonozzi]

@jobergum We need more muppet representation in the tokenizer pipeline.

[Nico Tonozzi @ntonozzi]

It seems likely that leading labs already use a process based reward model in the RLHF tuning steps, and it's not clear to me that using PRMs at inference time would be as effective as training.

[Nico Tonozzi @ntonozzi]

https://t.co/RAZ2r7sIjZ is super interesting (investigating if Q* really works!) It's not as trivial as Aidan argues in https://t.co/vQ91Dkgb70, because evaluating promising paths is obviously harder in the context of generating text than the context of chess.

[Nico Tonozzi @ntonozzi]

It's disappointing that they didn't see any improvements on the harder end of their evaluation dataset.

[Nico Tonozzi @ntonozzi]

@aidan_mclau @samdcbu What is q* without active search at inference time?

[Nico Tonozzi @ntonozzi]

I've been really impressed at how easy it is for our customers to get high quality evaluation suites once they can clearly see the completions flowing through their evaluations. There's no substitute for looking at the data!

[Ian Cairns @cairns]

Getting more teams to be successful with generative AI in production depends on making evals more practical and approachable.

We're fired up about early feedback: In multiple cases we’ve seen people go from 60% or 70% alignment to 100% in 2-3 iterations.

Curious? Read more on the details here.
https://t.co/zFV76HEVwX

[Nico Tonozzi @ntonozzi]

@ciphergoth @ModeledBehavior Yeah just bet on Harris and Trump, you'll come out way ahead

* in expectation

[Nico Tonozzi @ntonozzi]

GPT with advanced voice mode is incredible! The voices feel so much more alive, and identifiable with their name. I'd love to see something like Claude's artifacts integrated with the voice mode.

[Nico Tonozzi @ntonozzi]

Awesome to see smaller and cheaper models competing with the big guys on hallucination detection

[PatronusAI @PatronusAI]

2/ Lynx v1.1 uses the llama-3.1 architecture and is trained on data from real-world domains including finance and medicine. Lynx excels at reasoning in hard-to-detect hallucinations that include nuanced and challenging scenarios involving questions, documents and answers.

Check out our Hallucination Benchmark here: https://t.co/R9zE9heq6q
https://t.co/81RrWjDxKe

[Ian Cairns @cairns]

🚀 Thrilled to announce our new partnership with @MongoDB!

Together, we're empowering teams to test & tune RAG systems for production.

MongoDB Atlas combines the power of a high-performance operational database & vector databases into one platform for modern AI applications. @freeplay_ai makes it easy to build, test & optimize those applications to work at scale.

Dive in. 📖 https://t.co/YSPdIB8CaN