I will present our paper, “Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search,” at 2026 CODE@MIT, hosted by the MIT Initiative on the Digital Economy. Session N (AI Search, Discovery, and Publisher Economics), Saturday, November 14, 3:25 PM, Cambridge.
When an LLM answers directly on the results page, a no-click session can be a success (so-called “good abandonment”), and a click can still be a failure. We built Session Success Rate (SSR), an A/B test outcome metric grounded in human judgments of full sessions.
What we found:
(1) Click status alone misleads. 53.2% of abandoned queries were successful, and 6.5% of clicked queries were not.
(2) Online behavior signals can separate good from bad abandonment: viewport behavior (e.g., % of page visible), result-page interactions (e.g., scroll-down, mouse hover), and semantic reformulations based on embedding vectors.
(3) SSR agreed with Sessions per User in 7 of 8 conclusive experiment comparisons.
(4) Across about 4,000 production experiments, SSR was roughly 2.5x as sensitive as Sessions per User, which allowed smaller traffic allocations and more experiments under a fixed traffic budget.
Co-authors: Shavak Agrawal, David Chen, Carto Wong, Xin Du, Tiannan Wang, Georg Buscher (Microsoft), and Zheng Chen (Simplex AI).
Paper: https://t.co/hi1YSMEeR8
Conference: https://t.co/F0hjZe1ZLO
How do you define success for an AI answer that earns no click? Happy to compare notes in Cambridge.
#ABTesting #OnlineExperimentation #GenerativeAI
#AISearch #LLM #CausalInference #SearchSuccess #ABTesting #OnlineExperimentation #GenerativeAI
Hugging Face Build Small Hackathon:
I have participated in this Hackathon this weekend together with two other team members. Sharing Demo Video below. Please let us know if you have comments or feedback.
Demo Video: https://t.co/7bn0y5KM9c
App Desc:
https://t.co/7i7JZ3noMJ
Claude Code New Features vs. OpenClaw Capabilities
Claude Code now supports (1) Channels for pushing Telegram/Discord messages or webhooks into a running session, (2) scheduled tasks, (3) auto memory across sessions, and (4) remote control from web/mobile via /remote-control.
Markdown Converter for AI-native workflows:
https://t.co/tDkXnJGYEy
MarkItDown converts PDFs, Word docs, PowerPoints, Excel files, HTML, images, audio, and more into Markdown. In tools like GitHub Copilot CLI or Claude Code, this helps to ingest these artifacts as inputs.
GitHub Copilot X based on GPT4 is announced today. Powerful programming companion!
Key new features include:
1. More intuitive chat interface: Prompt-and-response conversations
2. Versatile coding tasks: test gen, code analysis, debug, add comments, explain
- 3. Voice chat
A dynamic week for Generative AI:
Stanford Alpaca
OpenAI GPT4
Anthropic Claude ChatBot
Google PaLM API
AdeptAI: raises $350M
Google: Add GenAI to Workspaces
PyTorch 2.0
Midjourney V5
Microsoft Office 365 Copilot
Baidu Earnie
Join me and @Manisha Arora for a webinar on Causal Inference, hosted by @PrepVector: Concepts, methods, and applications in real-world scenarios!
Date: March 15, 2023
Time: 8.30 PM EST/5.30 PM PST
Link to register: https://t.co/7S4lbIhRh9
Thrilled to announce the launch of the new Bing on both iOS and Android and the addition of voice access. 64% of searches occur on mobile, and this means you can take the new Bing with you wherever you go. Please keep the feedback coming! https://t.co/3v3ric50ka