Andrew Sheinberg

ML efficiency @princetonCS

[Andrew Sheinberg @AndrewSheinberg]

@oneill_c How you are deriving 2400 requests? To saturate compute throughput where hardware theo FLOPS/bytes is ~300, and for expert matrix of K=7168, N=2048, we need the M dimension (tokens per exp) to be >300; ideally 384. For 8 act/256 tot =>(384*256/8)=12,288 tokens per batch

[Andrew Sheinberg @AndrewSheinberg]

@harold_matmul Thank you for sharing such detailed report. Did you guys experiment what throughput curve looked like as you increased gbsz? I understand that for learning efficiency that it might not be effective, but I am curious.

[Andrew Sheinberg @AndrewSheinberg]

@ar0cket1 @reinerpope Yes, though on the machines you referenced host memory capacity is underprovisioned (nearly equal to hbm). Interconnect to host is about 1/20 to 1/10 BW vs. hbm. If 10x host capacity then scheduling could occur at 10x lower freq but enable more massive models

[Andrew Sheinberg @AndrewSheinberg]

@ar0cket1 @reinerpope Depends on model size / sparsity. If your existing model size / sparsity is able to saturate throughput on these racks => you could have larger / sparser model if you use secondary memory (and accept latency hit)

[Andrew Sheinberg @AndrewSheinberg]

@reinerpope @ar0cket1 Or, for latency insensitive regimes, you could have scheduling interval >> HBM capacity / HBM bw by using secondary memory. For many (model, seqlen, hbm capacity) => max batch size regimes a lot of FLOPS are wasted due insufficient batch size.

[MathMatize Memes @MathMatize]

Everything is linear algebra

[Andrew Sheinberg @AndrewSheinberg]

@karpathy I’d recommend annotating and profiling to find bottlenecks. I went through this same process of replicating a ResNet from scratch. Hardware vendors provide tooling. On Nvidia systems you can use NVTX for annotations and Nsight Systems to nicely visualize timeline

[10-K Diver @10kdiver]

1/

Get a cup of coffee.

In this thread, I'll walk you through "Gambler's Ruin".

This is a classic exercise in probability theory.

But going beyond the math, this exercise can teach us a lot about life, business, and investing.

[MIT CSAIL @MIT_CSAIL]

9 Distance Measures in data science with algorithms.

via: @gp_pulipaka

[Alta Ski Area @AltaSkiArea]

LCC Sunsets 😍

[Michael Kearns @mkearnsupenn]

@Tim_Roughgarden Might be Angluin’s algo for learning DFAs from queries

[MIT CSAIL @MIT_CSAIL]

Machine learning systems, explained.

comic v/@xkcd

[Andrew Sheinberg @AndrewSheinberg]

@FormulaDeltaOne it depends what you are trying to model. if too much noise and no true distribution to be learned => linear model better. more fine grained pattern where truer distribution is likely to exist => fancier learning model can be used to better estimate function

[Prof. Feynman @ProfFeynman]

When your "education" limits your imagination it's called indoctrination. Those who cannot think for themselves are truly lost.
Education should be a rewarding experience which allows you to think, imagine, question, doubt and solve problems. 🧠

[Andrew Sheinberg @AndrewSheinberg]

weird how most of life is smooth, and that the extremely few moments of discontinuity make all the difference (choosing school x vs. y, getting the job or not, marrying the girl, starting hobby etc.). When discontinuity occurs it is hard at first; but adapt over time - “smooth”

[Andrej Karpathy @karpathy]

Currently products brag about being "smart". Like my coffee cup warmer that had me download an app, sign up for an account and ask for location permissions before it would warm my coffee. A future where products brag about being "dumb" must be coming and can't come soon enough.

[Andrew Sheinberg @AndrewSheinberg]

forgot how beautiful generalized stokes was 😍

[10-K Diver @10kdiver]

1/

Get a cup of coffee.

In this thread, we'll analyze the "Devil's Card Game".

This is a super useful thought exercise. It can teach us several key concepts in economics, probability, betting, hedging, investor/market psychology, etc.

[Robot James 🤖🏖 @therobotjames]

You need to know:

1. why people are willing or forced to trade at prices that are disadvantageous to them
2. why, in a highly competitive market, you get to be the party taking the other side of those trades.