Pablius

FAANG, opinions my own, freedom. when uncertainty is high simplify, when uncertainty is low simplify.

[Yun-Ta Tsai @yunta_tsai]

When you are landing on a moon, you want the craft to be well engineered.

When you are on an airplane, you want it to be well engineered.

When you are in a car driving at high speed, you want it to be well engineered.

When you go across a bridge, you want it to be well engineered.

When you want to run a training job on thousands of GPUs, you want the clusters to be well engineered.

When you need something to be 99.99% reliable, you want it to be well engineered.

[Ihtesham Ali @ihtesham2005]

A mathematician who shared an office with Claude Shannon at Bell Labs gave one lecture in 1986 that explains why some people win Nobel Prizes and other equally smart people spend their whole lives doing forgettable work.

His name was Richard Hamming. He won the Turing Award. He invented error-correcting codes that made modern computing possible. And he spent 30 years at Bell Labs sitting in a cafeteria at lunch watching which scientists became legendary and which ones faded into nothing.

In March 1986, he walked into a Bellcore auditorium in front of 200 researchers and told them exactly what he had seen.

Here's the framework that has been quoted by every serious scientist for the last 40 years.

His opening line landed like a punch. He said most scientists he worked with at Bell Labs were just as smart as the Nobel Prize winners. Just as hardworking. Just as credentialed. And yet at the end of a 40-year career, one group had changed entire fields and the other group was forgotten by the time they retired.

He wanted to know what the difference actually was. And he said it wasn't luck. It wasn't IQ. It was a specific set of habits that almost nobody is willing to follow.

The first habit was the one that hurts the most to hear. He said most scientists deliberately avoid the most important problem in their field because the odds of failure are too high. They pick a safe adjacent problem, solve it cleanly, publish it, and move on. And because they never swing at the hard problem, they never hit it. He said if you do not work on an important problem, it is unlikely you will do important work. That is not a motivational line. That is a logical one.

The second habit was about doors. Literal doors. He noticed that the scientists at Bell Labs who kept their office doors closed got more done in the short term because they had no interruptions. But the scientists who kept their doors open got more done over a career. The open-door scientists were interrupted constantly. They also absorbed every new idea passing through the hallway. Ten years in, they were working on problems the closed-door scientists did not even know existed.

The third habit was inversion. When Bell Labs refused to give him the team of programmers he wanted, Hamming sat with the rejection for weeks. Then he flipped the question. Instead of asking for programmers to write the programs, he asked why machines could not write the programs themselves. That single inversion pushed him into the frontier of computer science. He said the pattern repeats everywhere. What looks like a defect, if you flip it correctly, becomes the exact thing that pushes you ahead of everyone else.

The fourth habit was the one that hit me the hardest. He said knowledge and productivity compound like interest. Someone who works 10 percent harder than you does not produce 10 percent more over a career. They produce twice as much. The gap doesn't add. It multiplies. And it compounds silently for years before anyone notices.

He finished the lecture with a line I have never been able to shake.

He said Pasteur's famous quote is right. Luck favors the prepared mind. But he meant it literally. You don't hope for luck. You engineer the conditions where luck can land on you. Open doors. Important problems. Inverted questions. Compounded hours. Those are not traits. Those are choices you make every single day.

The transcript has been sitting on the University of Virginia's computer science website for almost 30 years. The video is free on YouTube. Stripe Press reprinted the full lectures as a book in 2020 and Bret Victor wrote the foreword.

Hamming died in 1998. He gave his final lecture a few weeks before. He was 82.

The lecture that explains why some careers become legendary and others disappear is still free. Most people who could benefit from it will never open it.

[Omar Khattab @lateinteraction]

As promised, here's a recording of my 30-min keynote and the subsequent Q&A for the inaugural late interaction retrieval (LIR) workshop, cc @bclavie @antoine_chaffin.

The talk is admittedly advanced, as it's directed at an expert IR community. But hopefully still broadly useful!

[Pablius @pablius]

Happy Pi Day! Also, Apple pie should not have cinnamon.

[Pablius @pablius]

@VicVijayakumar @onetimefax Consider extending Opus a full time offer! He delivers fast 🤣

[Pablius @pablius]

Perhaps create a special "pro" landing page, and when they upload a file over a certain length (maybe 100+), just tell them that their file is considered for premium treatment and will be delivered with extra care... but it costs $.x / page. And then see how many convert? My friend Claude says we can ship it in an hour!

[Pablius @pablius]

@rakyll There is something about Opus that just goes beyond any prev model...

[Pablius @pablius]

@VicVijayakumar @onetimefax I wouldn’t say failed, I’d say unreachable to make it more clear that you tried. Also adding the times you tried if you can with a nice ��� next to them for impact

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Pablius @pablius]

@VicVijayakumar Builder this builder that…

[Vic 🌮 @VicVijayakumar]

A software engineer was on a weekend drive through rural countryside when he spotted a small, picturesque farm. Intrigued, he pulled over and met an older farmer tending to a modest vegetable garden.

"Beautiful place you have here," said the engineer, admiring the neat rows of vegetables, small fruit orchard, and chickens pecking contentedly in the yard.

"Thank you," replied the farmer. "Been working this land for 10 years now."

The engineer noticed the farm’s small size. "Is this all the land you work? You could scale up significantly with the right technology."

"This is enough," the farmer said with a smile. "Provides for me and my family, with extra to sell at our roadside stand and the farmers' market on Saturdays."

Curious, the engineer asked, “what’s your daily routine like?”

The farmer leaned on his hoe. "I wake with the sun, tend to my animals and crops for a few hours. I take the kids to school, then I lift weights, watch some tv, help my wife with her Shopify store. Afternoons, I might fix something around the property, read a book, take a nap, lunch with my wife. Evenings, I get the kids from school and we make dinner and eat together. Weekends, we go chat with folks at the market and swim in the pond. Simple but satisfying."

The engineer's eyes lit up. "Look I’ve built 3 successful SaaS startups and I see enormous potential here! You should develop a farm to table app that connects farmers directly to customers. Start with an MVP, validate product-market fit, then scale! Once we have traction metrics, I could draft a pitch deck and apply to YCombinator's next cohort."

The farmer looked confused. "That sounds like a lot of meetings and screens. I prefer being outdoors."

"That's just the beginning!" the engineer continued excitedly. "By Series B, we'd have expanded to multiple regions, acquired smaller competitors, and maybe even launched a proprietary line of smart farming equipment. We could hit unicorn status in 5-7 years and IPO shortly after!"

"And why would I want to do all that?" asked the farmer.

The engineer smiled confidently. "That's the dream! My Notion dashboard has my entire exit strategy planned—once I build a successful startup and have a liquidity event, I'll cash out, achieve financial independence, marry a San Francisco 6, and buy a small farm just like yours."

"So after all this app building and fund raising, what would you do with your life?" the farmer asked.

"Freedom! I'd finally escape the tech hamster wheel, buy a modest piece of land away from the city, grow heirloom vegetables, maybe raise some chickens, live according to the seasons rather than sprint deadlines. I'd delete PagerDuty, read actual paper books, send only handwritten letters via Onetime Fax to everyone, and reconnect with the natural world. That's the life I've been coding my way toward this whole time."

The farmer gestured to his fields, the chickens, and the small pond reflecting the afternoon sun. "Isn't that exactly what I'm doing now—minus the pitch decks, burn rates, and investor updates?"

The software engineer fell silent, noticing for the first time how his watch had been buzzing with Slack notifications the entire conversation.

[Guillermo Rauch @rauchg]

Some engineering principles I live by:
✓ Make it work, make it right, make it fast
✓ Progressive disclosure of complexity
✓ Minimize the number of concepts & modes
✓ Most 'flukes' aren't… your tech just sucks
✓ Feedback must be given to users instantly
✓ Maximize user exposure hours
✓ Demo your software frequently to fresh eyes
✓ Sweat every word of product copy you render
✓ You're never done working on performance
✓ You're never done. Software ages like milk, not wine
✓ Visualizing traces of time is the best way to optimize it
✓ Ship frequently and strive to build in public
✓ Errors must have globally unique codes & hyperlinks
✓ Red is not enough to signal "error" (8% of men have red-green color blindness)

[Pablius @pablius]

@VicVijayakumar Clearly no store, just an API to purchase! The possibilities are endless!

[DHH @dhh]

When Ruby on Rails was launched over twenty years ago, I was a twenty-some young programmer convinced that anyone who gave my stack a try would accept its universal superiority for solving The Web Problem. So I pursued the path of the crusade, attempting to convert the unenlightened masses by the edge of a pointed argument.

And for a long time, I thought that's what had worked. That this was why Ruby on Rails took off, became one of the most popular full-stack web frameworks of all time, inspired countless clones, and created hundreds of billions in enterprise value for companies built on it. But I was wrong. It wasn't the crusade that did it.

Since those early days, I've talked to thousands of programmers who adopted Ruby on Rails back then, and do you know what virtually every one of them cite? That original 15-minute blog video. Which didn't contain a single comparison to other named solutions or specifically pointed arguments against alternatives. It just showed what you could do with Ruby on Rails, and the A/B comparison automatically ran inside the mind of every programmer who was exposed to that.

That's what did it. Showing something great, and letting those who weren't happy with their current situation become inspired to check it out. Because those are the only people who is able to convert to your cause anyway. I've never seen someone who was head'over'heels in love with, say, functional programming be won over by arguments for objected-oriented programming.

You simply can't dunk someone into submission, and it's usually counterproductive if you try. But you can absolutely attract people who aren't happy with their current circumstances to give an alternative a chance, if you simply show them how it works, and allow them to conclude by themselves how it would make their programming life better.

What I've also come to realize is that programmers come in many different intellectual shapes and sizes. Some of those shapes will click with functional programming, and that'll be their path to passion. Others will click with vanilla JavaScript, and be relieved to give up the build pipelines. Others still will find their spirit in Go. This is great. Seriously. The fact that working for the web allows for such diverse ecosystem choices is an incredible feature, not a bug.

I found my life's work and passion in Ruby. I have friends who've found theirs in Python or Elixir or PHP or Go or even JavaScript. That's wonderful! And that's really all I want for you. I want you to be happy. I want you to find just that right language that opens your mind to the beautiful game of coding in your most compatible mode of conception, as Ruby did for me.

This is not the same as just saying "everything has trade-offs, use what works best". That to me is a bit of a cop out. There is no universal set of trade-offs that'll make something objectively "work best". Half the programming conundrum lies in connecting to an enduring source of motivation. I wouldn't be a happy camper if I had to spend my days programming Rust (but I LOVE so many of the tools coming out of that community by people who DO enjoy just that).

It also doesn't mean we should give up on technical discussions of advantages or disadvantages, but I think those are generally more effective when performed in the style of "here's what I like, why I like it, so look at my code, my outcomes, and see if it tickles your fancy too".

Programming is a beautiful game. I would give up all the fancy cars I have in a heartbeat, if I was made to choose between them and programming. The intellectual stimulation, the occasional high from hitting The Zone, is such a concrete illustration of Coco Chanel's "the best things in life are free, the second best things are very expensive". Programming is one of those "best things" that is virtually free to everyone in the Western world (and increasingly so everywhere else too).

So let's play that beautiful game to the best of our ability, in the position that flatters our conceptual capacities the most, and create some wonderful code.

[Colin Percival @cperciva]

LLVM/clang nerd sniping, FreeBSD edition: We have a few files in our tree which build differently for the i386 target depending on whether the build host is i386 or amd64. In a very small number of places, the register allocator picks differently.

Anyone want to take a look?

[dax @thdxr]

i only want to work people who are willing to tolerate a ton of up front pain to avoid ongoing pain

this is extremely rare

[Massimo @Rainmaker1973]

#Today in 1995, Microsoft released Windows 95.

[📹 PC USER 486]

[OneSimpleApi @OneSimpleApi]

We released https://t.co/Jq9c4nGXf9 for API Caching. Now you can use OSA to cache webpages or API responses, ensuring faster load times and reduced server loads!

[Glenn at TheFlow @TheFlowAgency]

✨🚀 Big Launch Day Today 🚀✨

Ready to massively level-up your NoCode game? It's a bit of a 🧵 but it's worth it...

🖐🏻Hands up if you identify with 1+ of these:

❓You're a dedicated @webflow'er
❓You're confident with visual web design
❓You've built some neat projects but...