Piyush Gupta

AI, People, Simplicity, Products

[DAIR.AI @dair_ai]

The largest LLM-as-a-Judge reliability audit yet.

Researchers ran 21 judges from nine providers over roughly 541,000 judgments on MT-Bench, JudgeBench, and RewardBench.

Findings:

Validating a judge with exact-match agreement overstates its skill, because exact match does not correct for chance.

Switching to Cohen's kappa deflates agreement by 33 to 41 points on MT-Bench, and judge rankings move by up to 14 positions across benchmarks.

There is also a consistency paradox. Two production-deployed judges score above 0.95 test-retest reliability while carrying severe position bias above 0.10, so a judge can agree with itself every time and still be wrong in the same direction every time.

Paper: https://t.co/Jh8U1R2svQ

Learn to build effective AI agents in our academy: https://t.co/LRnpZN7L4c

[Karri Saarinen @karrisaarinen]

Craft is the new "growth hack".

Investing in craft, means you invest in quality of the experience, and that benefits in all of areas of the business.

People talk about you. People want your product. Sales is easier. You create customer champions. Retention is higher.

[Rabi Gupta @rabigupta_]

Super excited to present our new AI for sales tool at Llama Lounge.

[Piyush Gupta @gpiyush]

A super insightful description on some good work done by our team. If sales prospecting is something you’re familiar with then this is an essential perspective to gather.

[Revenoid @RevenoidAI]

60% response rate in prospecting? Yes, it's possible! Below is the iterated process that helped us achieve similar outcomes…
#Sales #BusinessGrowth

[Piyush Gupta @gpiyush]

Some concepts are so simple and yet drive the highest-order impact. Journey map mashed up with Storyboards being one. At @SigFigInsider we built similar journey maps and I saw the power firsthand in driving new ideas, focusing on the real issues, and aligning external partners.

[Brian Chesky @bchesky]

We created a blueprint of the Airbnb experience and used it to improve our service end-to-end

[Piyush Gupta @gpiyush]

So proud of this team and being a part of it. Serving the bottom of pyramid is just so satisfying.

[Algorand Foundation @AlgoFoundation]

☀️ @ekospeaks enables cash transactions, processing over $30B in transactions for nearly 50M customers through 400K micro-entrepreneur shops across India.

Eko is now tokenizing open-source services on Algorand to develop and deliver more innovative products on its platform.

[Sarah Constantino @saraheconstant]

Today we're marking the 10-year anniversary of @firstround Review. 🍾🤩

To celebrate, our editorial team has curated a list of 100 of the best bits of company-building advice we've heard in the last decade. (#4 is one of my personal faves!)

https://t.co/EstyxlwnD0

[Piyush Gupta @gpiyush]

Live streaming of some exceptional AI thinkers happening right now. Previous talk from @decisionleader was very intriguing. https://t.co/CrRIgk7RC0

[Piyush Gupta @gpiyush]

@bindureddy It’s worth experimenting to reduce the prompt refining effort. Ideally, the final outputs aren’t fully automated, it just takes away the taste that humans relish.

[Andrew Ng @AndrewYNg]

A little known part of the origin story of LLMs: A lot of early Natural Language Processing research was funded by US intelligence agencies that had an interest in machine translation (and also speech recognition). This funding helped develop benchmarks, grow communities and nurture researcher interest in translation research disproportionate to its commercial application, and led to new foundational technologies motivated by translation.

For example, the transformer paper was originally a paper on mapping from one language to another; now we use transformers to map from a prompt to a response. The most widely used tokenizer (BPE) was also popularized in a paper studying how to translate words not seen in the training set.

More on this piece of NLP history in my letter in The Batch! https://t.co/7GdjnrLBmv

[Piyush Gupta @gpiyush]

@AnthropicAI I've applied for the API access a few weeks ago for the research purpose and a personal project. I'd love to figure a way to accelerate the process.

[Piyush Gupta @gpiyush]

Wow! Having spent 6 months equivalent effort in petitions and reviewing it for so many friends, I can see what amazing use case this can be.

[Anthropic @AnthropicAI]

Our Social Good Shout-out went to Immigrant First: a project leveraging Claude’s long context window and extended outputs to generate complete petitions for immigrants.

Congratulations Benedict, Atal, Arafat and Bhaskar!

[Anthropic @AnthropicAI]

Our Social Good Shout-out went to Immigrant First: a project leveraging Claude’s long context window and extended outputs to generate complete petitions for immigrants.

Congratulations Benedict, Atal, Arafat and Bhaskar!

[Andrew Ng @AndrewYNg]

How can business leaders, not just coders, build AI business applications with LLMs? Our new short course, created with Microsoft, is out! In How Business Thinkers Can Start Building AI Plugins With Semantic Kernel, taught by @johnmaeda, you'll learn to use Semantic Kernel and build an AI planner that can automatically select the plugins it needs to create multi-step plans with sophisticated logic. Please sign up here: https://t.co/YdW5fu1UIj

[Sam Altman @sama]

seeing a lot of confusion about this, so for clarity:

openai never trains on anything ever submitted to the api or uses that data to improve our models in any way.

[Nishchay @agNishchay]

@BillGates heard you are in @peakbengaluru .. would love to take you out for a crispy masala dose and strong filter coffee at @VidyarthiBhavan 😄

[Piyush Gupta @gpiyush]

One of the most talented teams working on EVs in India. Personally a fan of this team. Great investment opportunity.

[V. Ramgopal Rao, Ph.D. @ramgopal_rao]

A team of young #iitdelhi grads building a tofally indigenous and the world's #safest next generation #electric two-wheeler in their award winning #startup @creatarateam. They need your encouragement and support. @PMOIndia @OfficeOfNG @nitin_gadkari

[Piyush Gupta @gpiyush]

Super sad state of affairs about the education system.

[V. Ramgopal Rao, Ph.D. @ramgopal_rao]

Children's education has really turned into a business. If you are parent of a 10th standard passout student, not a single day passes without receiving a spam call from some coaching institute. Schools seem to be leaking information about their enrolled students to them.

[Piyush Gupta @gpiyush]

Seeing my daughter play India’s National Anthem on piano at the talent show in her school and witnessing the audience standup and sing together in the middle of an internationally diverse community felt deeply joyful.

[Piyush Gupta @gpiyush]

@anupamvarghese The Indian Aunt's tactics worked finally :)