Sahil Thaker

♥️ Machines, Math, Sports, Music. Co-Founder @hyperline_xyz Past: @Glean, @Facebook News Feed AI, Video AI, @Uber marketplace optimization

[Max Levchin @mlevchin]

IMO, just as today we think of codegen AI as a 10x productivity multiplier for CS degrees (or equivalent), in a year or two we will be thinking of CS degrees as 10x productivity multipliers over codegen AI.

[Jamie Voynow @voynow]

I took 100g of creatine and now i’m doing LLM inference in my head

[Gautam Kedia @thegautam]

TL;DR: We built a transformer-based payments foundation model. It works.

For years, Stripe has been using machine learning models trained on discrete features (BIN, zip, payment method, etc.) to improve our products for users. And these feature-by-feature efforts have worked well: +15% conversion, -30% fraud.

But these models have limitations. We have to select (and therefore constrain) the features considered by the model. And each model requires task-specific training: for authorization, for fraud, for disputes, and so on.
Given the learning power of generalized transformer architectures, we wondered whether an LLM-style approach could work here. It wasn’t obvious that it would—payments is like language in some ways (structural patterns similar to syntax and semantics, temporally sequential) and extremely unlike language in others (fewer distinct ‘tokens’, contextual sparsity, fewer organizing principles akin to grammatical rules).

So we built a payments foundation model—a self-supervised network that learns dense, general-purpose vectors for every transaction, much like a language model embeds words. Trained on tens of billions of transactions, it distills each charge’s key signals into a single, versatile embedding.

You can think of the result as a vast distribution of payments in a high-dimensional vector space. The location of each embedding captures rich data, including how different elements relate to each other. Payments that share similarities naturally cluster together: transactions from the same card issuer are positioned closer together, those from the same bank even closer, and those sharing the same email address are nearly identical.

These rich embeddings make it significantly easier to spot nuanced, adversarial patterns of transactions; and to build more accurate classifiers based on both the features of an individual payment and its relationship to other payments in the sequence.

Take card-testing. Over the past couple of years traditional ML approaches (engineering new features, labeling emerging attack patterns, rapidly retraining our models) have reduced card testing for users on Stripe by 80%. But the most sophisticated card testers hide novel attack patterns in the volumes of the largest companies, so they’re hard to spot with these methods.

We built a classifier that ingests sequences of embeddings from the foundation model, and predicts if the traffic slice is under an attack. It leverages transformer architecture to detect subtle patterns across transaction sequences. And it does this all in real time so we can block attacks before they hit businesses.

This approach improved our detection rate for card-testing attacks on large users from 59% to 97% overnight.

This has an instant impact for our large users. But the real power of the foundation model is that these same embeddings can be applied across other tasks, like disputes or authorizations.

Perhaps even more fundamentally, it suggests that payments have semantic meaning. Just like words in a sentence, transactions possess complex sequential dependencies and latent feature interactions that simply can’t be captured by manual feature engineering.

Turns out attention was all payments needed!

[Sahil Thaker @_sahilt]

@martin_casado to me, books serve as storytellling tool. its a film that captures state of mind of the author. not the most efficient way to learn if i'm motivated

[Balaji @balajis]

FROM MAGA TO CHINA
Here are four things MAGA is getting wrong, and why it's handing over the world to China.

(1) First, MAGA correctly understands that America’s economic position is in decline but thinks this is due to economic competition itself, rather than lack of competitiveness.

(2) Second, MAGA also understands that the US has wasted trillions abroad in foreign wars, but thinks the problem is global leadership itself rather than poor leadership.

(3) Third, MAGA knows that their Blue American enemies have allies abroad, but has incorrectly overreacted to this by treating every non-Red-American as an enemy.

(4) Fourth, MAGA sees the billions of dollars flowing from the US to foreign recipients, but isn't grasping that the US can only print those dollars in the first place so long as it's the hub of a global empire.

When you put these together you can both understand MAGA's actions and understand why they will not lead to the intended result.

Basically: MAGA is hyperfocused on cutting off any apparent flow of funds from Red Americans to Blue Americans and non-Americans. And they only have ~500 days in power. So they're trying to quickly shut off imports, close down institutions, and exit all wars.

OK.

Except the reason the imports exist in the first place is because US products aren't competitive relative to Chinese products (or Fed printing). The reason those institutions exist is because the US set them up to run the world. And the reason those wars are happening is not because of American leadership per se, but because of the absence of good leadership.

If you shut all of that down at once — if you abandon global competition and global leadership — you shut down American Empire, and with it the ability to print money. And then everyone in that empire has a very bad time.

[Sahil Thaker @_sahilt]

one way to think about ai: its humanity's collective brainpower

over time i think we'll get experts and fragments to this central brain, but directionally its "collective intelligence"

[Derya Unutmaz, MD @DeryaTR_]

I just asked OpenAI Deep Research to analyze some fresh & highly complex unpublished experimental data that we couldn’t decipher, related to an immune cell. When I read its report, I was absolutely stunned!😳 It solved the issue in the most profound way possible🤯Speechless🥹

[Sahil Thaker @_sahilt]

@DeryaTR_ humanity's collective brainpower

[Sahil Thaker @_sahilt]

@eddybuild curious: how do you eval a branch at any depth? whats the objective func here?

[Sahil Thaker @_sahilt]

Came across this neat tool with viz to dig into llm internals: https://t.co/uN2uNmpavs

[Sahil Thaker @_sahilt]

@waitin4agi_ or is it just beginning

[Sahil Thaker @_sahilt]

@tszzl @markchen90 what did you reply

[Sahil Thaker @_sahilt]

Great to see meaningful usecases of reasoning models

[Derya Unutmaz, MD @DeryaTR_]

Something amazing happened with the @deepseek_ai model! DeepSeek R1 just generated a highly innovative idea for a cancer project I’m working on😮

This is the first time an LLM, other than o1 models, has produced such creative ideas! I’m now using it side by side with o1-pro!

[Sahil Thaker @_sahilt]

@pelaseyed which compute service do you use for deepseek?

[Andrej Karpathy @karpathy]

The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days.

Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design.

It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends).

Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time.

Thank you for the story @DBahdanau !

[Sahil Thaker @_sahilt]

Deepmind chess paper showed that deterministic software can be mapped to neural nets
https://t.co/PmX2F9gNYl

This allows us to start from existing knowledge, but design the system to self-improve it through experience

[Shital Shah @sytelus]

Now the time for that bigger insight:

Stockfish engine is littered with all kind of complexities built with traditional computational primitives of conditionals, branching, data movements and so on.

What DeepMind paper points to is that all of these can potentially be distilled down to mere matrix multiplications!

There is something so utterly bewildering here. I think we are on the precipice of entire computer science being rewritten from its origin of Turing machines.

[Sahil Thaker @_sahilt]

@sytelus What do you think is the benefit of converting traditional/ heuristic software to NNs? Perhaps its that nns can continue to learn from usage and improve vs heuristics cant?

[Crémieux @cremieuxrecueil]

The benefits of caloric restriction for primate lifespans (including our own!) are probably overrated.

We have two rhesus monkey studies, and they do not support big benefits🧵

First, take a look at this diagram:

[Sahil Thaker @_sahilt]

@martin_casado earlier, write c and you heart pounds: will it compile. later, write python and heart pounds: will it run. now, prompt an ai and heart pounds: will it do the right thing next time

[Nivi @nivi]

What’s the difference between reason, intelligence and understanding?

Reason is all methods of criticism that improve ideas.

Intelligence is how fast we create ideas that solve problems.

Understanding is the ability to address criticisms of an idea.

Rationality is seeking to improve ideas and using the best ones.

Judgment is anticipating the long-term consequences of actions.

Conjecture is the unexplained process of creating ideas.

Creativity is making ideas that are not predetermined by the evolution of a system.

Science is ideas that can be tested through repeatable experiments.

Knowledge is ideas that are useful.

Did I miss anything?

[Deedy @deedydas]

“Experience is what you get when you didn't get what you wanted.”

“When you're screwing up and nobody's saying anything to you anymore, that means they gave up.”

Video: https://t.co/lrpPCJ9tXv
Book: https://t.co/sAbpTmZMwn

2/4