@hamsabastani@METR_Evals Awesome. Especially love how this shifts the conversation from focusing on one curve to assumptions about what’s driving the progress and what the next step change could look like.
This is similar to what we find. Instead of easy vs hard questions, we use recombination of tags as a framework to define novel, niche and popular as question categories. We find that novel questions remained stable in volume and had the largest share. Interestingly, these novel questions were asked disproportionately by new users rather than SO users.
https://t.co/HnzpIjd5NN
There’s suddenly a lot of attention on Stack Overflow's collapse. However, the warning signs were visible as early as 2023. In our paper (data through 2018 - June 2023), we find that this decline was not uniform across questions.
Questions that were novel in the sense of recombining existing knowledge domains in new ways were stable, accounted for the largest share of what people asked over the years, and were still being answered. It was the routine questions that were impacted drastically by LLMs. See the hollowing out of the core of the community’s knowledge by May'23. Moreover, these novel questions were asked disproportionately by new users. This shift away from routine to novel questions and from existing to new users was an early warning sign. While novelty is valuable, it is not enough to sustain a knowledge community and now we might just have to rely on LLMs for the everyday technical knowledge! Paper (SSRN): https://t.co/HnzpIjd5NN
In this first episode of Research@ISB - ISB Faculty Milind Sohoni and Neha Sharma, PhD student at Kellogg School of Management, discuss their research paper titled: "List Now or Later? An Equilibrium Analysis of Advance-Booking Platforms."
Watch here: https://t.co/BQdGYORa0S
#ResearchAtISB #ISB