luc

Builder, Craftsman, Founder at heart.

[Jason Fried @jasonfried]

Validation is a mirage.

“How do you validate if it’s going to work?”
“How do you know if people will buy it to not?”
“How do you validate product market fit?”
“How do you validate if a feature is worth building?”
“How do you validate a design?”

You can’t.
You can’t.
You can’t.
You can’t.
You can’t.

I mean you can, but not in spirit of the questions being asked.

What people are asking about is certainty ahead of time. But time doesn’t start when you start working on something, or when you have a piece of the whole ready. It starts when the whole thing hits the market.

How do you know if what you’re doing is right while you’re doing it? You can’t be. You can only have a hunch, a feeling, a belief. And if the only way to tell if you’ve completely missed the mark is to ask other people and wait for them to tell you, then you’re likely too far lost from the start. If you make products, you better have a sense of where you’re heading without having to ask for directions.

There’s really only one real way to get as close to certain as possible. That’s to build the actual thing and make it actually available for anyone to try, use, and buy. Real usage on real things on real days during the course of real work is the only way to validate anything. And even then, it’s barely validation since there are so many other variables at play. Timing, marketing, pricing, messaging, etc.

Truth is, you don’t know, you won’t know, you’ll never know until you know and reflect back on something real. And the best way to find out, is to believe in it, make it, and put it out there. You do your best, you promote it the best you can, you prepare yourself the best way you know how. And then you literally cross your fingers. I’m not kidding.

You can’t validate something that doesn’t exist. You can’t validate an idea. You can’t validate someone’s guess. You can’t validate an abstraction. You can’t validate a sketch, or a wireframe, or an MVP that isn’t the actual product.

When I hear MVP, I don’t think Minimum Viable Product. I think Minimum Viable Pie. The food kind.

A slice of pie is all you need to evaluate the whole pie. It’s homogenous. But that’s not how products work. Products are a collection of interwoven parts, one dependent on another, one leading to another, one integrating with another. You can’t take a slice a product, ask people how they like it, and deduce they’ll like the rest of the product once you’ve completed it. All you learn is that they like or don’t like the slice you gave them.

If you want to see if something works, make it. The whole thing. The simplest version of the whole thing – that’s what version 1.0 is supposed to be. But make that, put it out there, and learn. If you want answers, you have to ask the question, and the question is: Market, what do you think of this completed version 1.0 of our product?

Don’t mistake an impression of a piece of your product as a proxy for the whole truth. When you give someone a slice of something that isn’t homogenous, you’re asking them to guess. You can’t base certainty on that.

That said, there’s one common way to uncertainty: That’s to ask one more person their opinion. It’s easy to think the more opinions you have, the more certain you’ll be, but in practice it’s quite the opposite. If you ever want to be less sure of yourself, less confident in the outcome, just ask someone else what they think. It works every time.

[luc @lucdiallo]

raix, Obie Fernandez's mix-in gem, is 38 files of readable Ruby and a maze to trace. The modules find each other at runtime, through metaprogrammed dispatch, so my agent read every file and still couldn't say where a dispatch change lands. The seams aren't in the text.

The map carries the runtime edges, and relationship accuracy went 0.83 to 0.93. Ten points of pure wiring.

raix's most load-bearing line lives in no file you can open.

https://t.co/E3CzMefYtF

[luc @lucdiallo]

PS. The best benchmark embarrasses the person who built it. Mine has, twice. Tell me how to make it a third. That's the reply I most want.

[luc @lucdiallo]

1/ Genuine question. I want your answer more than your agreement.

How would you test whether an AI actually understands your codebase?

Not whether it writes code. Whether it knows what breaks if you change this.

[luc @lucdiallo]

4/ My attempt: "find every dependent before a teardown change," graded against a hand-built answer key.

I think it's flawed in interesting ways. I laid out the holes I already see.

Tear it apart: https://t.co/z64pIdvevA

[luc @lucdiallo]

I benchmarked my code map on the repo that needed it least. langchainrb is named so well my agent walked in without the map and still got the structure right, 0.96.

Sense's worst case, measured: citations stopped gesturing at the right directory and started naming the line, the session ran 16% faster, and the bill dropped from $1.29 to $1.01.

Every tool gets judged on its highlight reel. The floor is what you actually live with. This one held.

https://t.co/tfKMhaehFZ

[luc @lucdiallo]

3/ The tool that wins big on a monolith ties on a clean gem, and says so. Everyone thinks their code is the clean gem, right up until it isn't.

https://t.co/UTiUzCyjQ2

[luc @lucdiallo]

1/ RubyLLM is so clean an agent barely needs a map to navigate it.

[luc @lucdiallo]

2/ Tracing the provider/`Chat` boundary, the plain agent hit 0.76 alone. The map nudged it to 0.80. Small, colocated, every file named for what it does. When the layout already is the map, a second one adds little.

[Jorge Manrubia @jorgemanru]

Lexxy deserved a gorgeous home. Big kudos to @zoltanhosszu for making it happen!

https://t.co/xmfKYUwXgx

[luc @lucdiallo]

6/ There's no alert for the day your app outgrows what a model reads in one pass. The gap opens where you're not looking.

Which half is yours? Test it: https://t.co/Jn8kMe0LSl

[luc @lucdiallo]

1/ "Ruby is the most AI-friendly stack" is half true.

One half holds up on its own. The other half I measured on 13 real Ruby repos, and the board split clean in two.

[luc @lucdiallo]

5/ Surprise two. The famous repos were among the hardest. A model can recite memorized wiring as if it were current. One small gem hit the same wall: the model knew it well, at a version that's no longer the one you'd install.

[luc @lucdiallo]

Redmine, the Rails app that has quietly run Ruby's own bug tracker for years. An agent audits `Issue` and gets 10 of 13 dependents on its own. A map takes it from 0.67 to 0.78.

The map is worth exactly the slice of your codebase the model can't afford to read for itself. On a medium app, a sliver. On a monolith, most of it.

https://t.co/YgHohrIzLh

PS. Redmine sits right where the two halves of this series meet. The gap widens with size.

[luc @lucdiallo]

5/ The catch. The line between "grep is enough" and "you need the map" moves the day your repo outgrows what a model can read at once. Nothing tells you when it crosses.
https://t.co/rCLJ04Pfzf

[luc @lucdiallo]

1/ Lobsters is small enough that a strong agent reads it top to bottom and nearly nails the teardown alone. The map added one, maybe two dependents. That near-tie is the most honest number in the series.

[luc @lucdiallo]

4/ A benchmark that only shows you wins is selling, not measuring.