In fact, ML can be especially impactful in situations like this. The heuristics make excellent features for a linear model. The result is often good enough (or is a strong baseline). Keeping this in mind gives me a nice “playbook” for kicking off work on a new project.
For some classification problems, a first analysis usually uncovers several heuristics that would work ~50-75% of the time. My gut reaction to this is often: “Do we really need to use machine learning here?” After all, I don’t want to be the fool with a hammer looking for nails.
The majority of ML case studies floating around the internet are, unfortunately, fast food. I think this is a problem because we can’t share, learn from, and discuss our “recipes” as practitioners.
There are lots of recent papers in the ML literature that look at how to detect when we can’t make reliable predictions. I often see this described as detecting “out of distribution” samples. This is unusual to me, though. The same value can come from two different distributions.
When we talk about covariate shift, the support of the train and test distributions may be the same but the frequency of seeing a given input may have changed. This is important when we use low-capacity models, but maybe less so with the richer classes we use today.
@IAmSamFin Great write-up on this (unfortunately?) evergreen debate; thanks for sharing! I liked your point about avoiding the “who owns regression” question. The same tool can be used to accomplish different things.
Stats journals often have a separate “applications” track. Does something like this exist for machine learning? I’m looking for good write ups of the nitty gritty details behind successful ML applications.
The thread below is the kind of thing that we need more of in “applied ML literature”. In this case, they didn’t really need a model, but I would love to read more about this kind of clever detective work.
After 17 years, we finally “cracked” a $100M churn problem at PayPal. Zero fancy tech. Just a spreadsheet, some simple SQL, and a physicist named Ben. 👇🏼
We've heard lots about #MachineLearning in healthcare, but usually with few specifics. A new paper in @AnnalsofIM, featuring our very own @suchisaria & @pschulam, cuts through the buzz and discusses the real applications and real benefits of clinical #AI: https://t.co/0HRNAVcEoR