Ritesh Bansal

Products | Models | Finance | Tech | Counterfactuals | Incompleteness | Convexity | Falsification

[No context Hazz ⛩ @HazzimIO]

No comments.

[Zain Shah @zan2434]

Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see.

@eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)

[Probability and Statistics @probnstat]

"Probability does not exist."
This provocative quote is by the Italian statistician and probabilist Bruno de Finetti.
It encapsulates the core idea of Subjectivism or Bayesianism, a major school of thought in probability. De Finetti argued that probability is not an objective property of the world (like mass or temperature). Instead, it's a measure of an individual's degree of belief about an uncertain event, based on their knowledge and evidence.
Essentially, he meant that a "50% chance of rain" doesn't exist in the clouds themselves; it exists in the mind of the meteorologist who assigns that value based on the data they have. It's a powerful statement that reframes probability as a tool for reasoning under uncertainty, rather than a feature of physical reality.

[Ole Peters @ole_b_peters]

The gift that keeps on giving: my infamous coin toss.

One optimistic line: "economists call it the ergodicity problem."
Perhaps one day it will be true. So far, out of the 500,000 economists on earth, only a handful understand what ergodicity is and why it matters.

[Rothmus 🏴 @Rothmus]

🎯🎯🎯🎯🎯

[Dr Aseem Malhotra @DrAseemMalhotra]

BREAKING NEWS!

New systematic review of drug trials including statins in @BMJ_EBM reveals no reduction in heart attacks or death from lowering LDL.

Let’s continue to expose the great cholesterol con.

https://t.co/qsdfJEOgjl

https://t.co/INBBlH2jZZ

[Yann LeCun @ylecun]

It seems to me that before "urgently figuring out how to control AI systems much smarter than us" we need to have the beginning of a hint of a design for a system smarter than a house cat.

Such a sense of urgency reveals an extremely distorted view of reality.
No wonder the more based members of the organization seeked to marginalize the superalignment group.

It's as if someone had said in 1925 "we urgently need to figure out how to control aircrafts that can transport hundreds of passengers at near the speed of the sound over the oceans."
It would have been difficult to make long-haul passenger jets safe before the turbojet was invented and before any aircraft had crossed the atlantic non-stop.
Yet, we can now fly halfway around the world on twin-engine jets in complete safety.
It didn't require some sort of magical recipe for safety.
It took decades of careful engineering and iterative refinements.

The process will be similar for intelligent systems.
It will take years for them to get as smart as cats, and more years to get as smart as humans, let alone smarter (don't confuse the superhuman knowledge accumulation and retrieval abilities of current LLMs with actual intelligence).
It will take years for them to be deployed and fine-tuned for efficiency and safety as they are made smarter and smarter.

[Ritesh Bansal @riteshabansal]

@nntaleb Even a pretense of thought to causal credibility has been forsaken for catchy correlations.

[Shreyas Doshi @shreyas]

If you’re a Startup trying to compete with a Megacorp—the 800-pound Gorilla in the space—you need to understand the tax inherent to being a Gorilla

And then you need to make that tax work against the Gorilla—with your product's positioning & features

A thread on Gorilla taxes👇🏾

[Shreyas Doshi @shreyas]

Exotic Metrics: a disease that affects many product teams.

Exotic Metrics makes us feel smart, but if you really zoom out and are honest to yourself, it makes us act pretty stupid. It is highly contagious and chronic.

Let me explain, with a story.

--

Say you are working on a product that helps startups.

Maybe it helps them with their financial operations, maybe it helps them set up & manage bank accounts more easily, maybe it helps them hire high quality offshore engineering talent, or something else like that.

You now need to define a metric to measure your success / impact / progress towards your mission.

--

Say you come up with this metric:
Number of startups that are actively using our product.

Sounds reasonable? Sure, when it's just us two, it sounds eminently reasonable.

You now take this idea to a colleague, Bob.

This is where the trouble starts.

--

Bob, genuinely wanting to help, says:
This is a fine metric, but I am concerned it can be gamed.

You see, the definition of a "startup" can be very vague.

What if you / Sales just sign up a bunch of solo practitioners who are not really a startup.

You could inflate this number quite a lot, hit the OKR out of the park, and still miss the spirit of our mission.

--

Bob sounds smart. Bob sounds genuine. And you certainly don't want to come across as someone who'd game the system.

So you come up with a revised metric:
Number of venture-backed startups who are using our solution.

--

Sounds very reasonable. Only now, you will need to add some complexity to the computation of your metric. You will need to look up some external data sources for VC funding.

Oh, but there isn’t 1 great data source for this, so you do some Google research to identify 3-4 data sources that together should be fairly comprehensive.

Only problem: they publish data in different formats, frequency, accuracy. Sometimes data for the same company is contradictory.

--

Don’t worry though. Your data science counterpart, Dana, loves these kinds of data problems, so she gets to work on aggregating a bunch of external data sources for VC funding.

Alas, some of these data sources require licensing. Yet another problem to solve.

--

But not to worry. You’re a PM. PM stands for Problem-solving Machine, doesn’t it? No one actually knows, but it kinda sounds cool so we’ll go with it.

You look up the data licensing requirements, and then engage your Biz Dev counterpart to go figure out the best terms for being able to get access to this data.

--

You start to worry that this is starting to get a bit out of hand.

Scope is creeping.

But you reason: well, this is our North Star Metric. And you remember reading somewhere that a team without a great NSM is a like a ship without a rudder.

And you really don’t want your CEO to point out flaws in your metric at next month’s QBR.

--

So you give the green light to Biz Dev, while Dana has embarked on a 2 week project to normalize the data from these various sources. If it’s worth doing, it’s worth doing right - you reassure yourself.

You are scheduled to meet your manager and skip manager this Friday, so you can’t wait to give them an update on your beautiful metric.

--

Friday arrives. You present this metric as your team’s North Star Metric. You are confident that they will be impressed. Except, your skip manager, Beth, doesn’t seem quite convinced.

She says, very encouragingly:
This is a reasonable metric.

Although one issue I see is that we are not discriminating on the type of startups that we are helping.

You see, what we really want is for our customers – these startups – to succeed more than they would without our product.

So I think a real North Star Metric should reflect something about the quality / success of the startups we serve.

Your manager is quick to chime in:
I agree with Beth.

Perhaps our North Star Metric should be “startups that have been funded by Tier 1 VCs” since that can be a proxy for quality.

Or better still, how about just using total valuation? Since that is ultimately what signals how successful our customers are.

You now feel compelled to play along, so you say:
That makes sense.

Perhaps the right metric is “Total valuation of startups that use our product”.

You see nods of approval.

Towards the end of the conversation, your manager says:

I like this metric. For our next 1:1, I’d love to see what goals and targets you’d set for this metric over the next 3 months, 6 months, and 12 months.

You nod.

--

You walk away happy that you were able to get consensus, but nervous about how you’re actually going to measure this.

You debrief with your data science counterpart Dana and share this revised metric. Dana looks very excited about this additional complexity and reassures you that everything will be fine.

--

The big day is here. You do the warrior pose right in the bathroom stall just before your QBR and then walk into the conference room, beaming with confidence.

At the right time, you share the proposed North Star Metric for this important initiative. You make it a point to clarify to your CEO that you wanted a metric that is

1) not game-able
2) reflects impact on your customers
3) is a stable success measure

Your CEO’s eyes light up.

CEO says:
I like this metric. Too often teams propose metrics that don’t really reflect true customer impact.

I just have one question: since startup valuations follow the power law, does it make sense to narrow the metric such that it reflects the impact of the top startups we serve?

Beth is quick to chime in:
That’s a great point. As such we want all our customers to succeed wildly.

But I think a metric that focuses on the impact on our top 20 startups or top 10% of startups would further incentivize the team to build for our biggest customers.

Everyone nods in agreement.

You feel like you have to nod too.

So you do.

--

The group then discusses some targets for this metric. Your eng counterpart Eddie is quick to point out that the targets will seem small in the near term because your customers will take some time to grow their revenues & valuation. Everyone agrees.

Meeting ends, with all participants feeling that sense of smug satisfaction that is all too familiar in the corporate world: we are all smart, we are being smart, we are super-thoughtful about the metrics we set.

--

Of course, what follows after this should also be painfully obvious, because we’ve all seen some versions of this play out (even if we haven’t had the courage to accept it):

- This metric is very difficult to calculate

- So dozens of assumptions have to be made in its calculations

- Which in turn reduces the accuracy of the metric

- This metric is very difficult to move in the near-term

- So targets get set somewhat randomly

- Some quarters, you drastically outperform

- You rejoice. You’re doing it right!

- You make sure everyone knows that.

- Some quarters, you underperform

- So you ask for more resources and x-fn alignment at the next QBR

- You created a fancy dashboard to track the metric

- But now no one looks at that dashboard

- Because everyone knows there will be nothing new to see, nothing insightful to conclude, nothing you can actually act on

- You keep setting new quarterly targets for the metric, but deep down no one on the team actually has any confidence that the projects you’re prioritizing will help hit those targets

--

At some point, you say enough is enough

You go back to the drawing board with your new eng counterpart

And you jointly arrive at a better metric

Something that reflects proximate customer adoption, not approximate customer impact

Something that is a leading indicator, not a lagging indicator

And here it is:
Number of startups actively using our product.

--

You present this new proposal to your manager and your skip manager.

They agree that the old metric is the wrong one.

The next 30 minutes are spent poking holes in your proposed metric.

At some point, your manager says:

My main concern is that this metric can be gamed. We need a metric that represents the impact of our product.

How about:
% of startups with a >20% cost savings due to our product.

You see your skip manager nod.

Your gut knows this is not the right metric.

But your head involuntarily nods.

The chronic disease of Exotic Metrics strikes again.

And so the suffering repeats.

It is your destiny.

THE END.

[Ritesh Bansal @riteshabansal]

@nntaleb I find going across the Bronx to be perilous. The best option would be to ask on r/NYCbike/.

RideWithGPS gives the route below probably using their Heatmap but my risk/comfort probably does not comport with the population who bikes in the Bronx.

Danger of being data-driven :)

[Ritesh Bansal @riteshabansal]

@VManasvi @wileycerilli @threadreaderapp unroll

[Ritesh Bansal @riteshabansal]

@VManasvi @wileycerilli Slack was based on a successful internal tool called LineFeed at Glitch, a failed gaming startup. Butterfield & Co took Linefeed and refined/tested it over a 9 month period before launching it as Slack in 2013.

No overnight successes. Just wrong timescales.

4/4