CaliRoll

Born as a designer and die as a cryptographer | Triple-Major STEM Enthusiast | research @calblockchain

[CaliRoll @digital_carol]

Models are biased for career advice!
Same prompt: continue quant or try sales for entrepreneurs in 20s to optimize breadth and depth?
@grok 3: technical skills 100%
Claude 3.7: be well-rounded
Gemini 2.0: be greedy and do both

[CaliRoll @digital_carol]

Plz read it

[vitalik.eth @VitalikButerin]

Against choosing your political allegiances based on who is "pro-crypto"

https://t.co/rEbRjZYtFB

[Andrej Karpathy @karpathy]

📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
https://t.co/QTUdu8b0qh

The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
https://t.co/BOzkxQ8at2

Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo

[CaliRoll @digital_carol]

🌙@nocturne_xyz

0x490bca744a2045029d9e1622bf8304e08a3f0c2f875e3d1207185bd474f444544744694e0c050909efac59f734cb31655eae30abfd0b51d971c58328b4767db0

0x70258363f971dd280f7ada8dd5d4ac93e0ff50a14f05b029cb72ae4adb2ba4bcd53cb1c5a3a8ed9324c84c0d1008fb8be81b27ada636320a50d9cf1ebc779281

[CaliRoll @digital_carol]

@BMANLead @ABCDELabs Cool stuff! What time is the Stanford campus zk event? Any links?

[CaliRoll @digital_carol]

@nocturne_xyz 🙋🏻

[Pratyush Ranjan Tiwari @PratyushRT]

And it's here! In this work, we establish new results on the Fiat-Shamir security of several protocols widely used in practice: FRI, batched FRI, and most Plonk-like zk SNARKs that use low-degree proximity testing as a subroutine. Why should you care?
1/3

https://t.co/RSoDYmlwag

[CaliRoll @digital_carol]

👀

[ABCDE @ABCDELabs]

📢 😍Exciting news! We are thrilled to announce the official kick-off of the ABCDE ZK Hacker Camp Part I: ZK Curriculum Registration!

🎯Brace yourself for mind-blowing sessions, hands-on workshops, and incredible networking opportunities.

REGISTER NOW: https://t.co/TNxvSGT9FH

[CaliRoll @digital_carol]

@zama @EthCC @mortendahlcs Amazing looking forward to it!

[CaliRoll @digital_carol]

@maggielove_ @She__Fi Here 🙋🏻

[CaliRoll @digital_carol]

@nocturne_xyz Awesome work buildooor!!

[CaliRoll @digital_carol]

Great to see privacy as a feature for the Ethereum ecosystem coming!! Amazing team and looking forward to the testnet

[Nocturne @nocturne_xyz]

1/ For crypto to reach its next stage of growth, it must be used less for speculation and become more like money. We believe privacy is a necessary condition for this to occur.

@nocturne_xyz we’re excited to introduce a new primitive to the Ethereum ecosystem—private accounts 🧵

[CaliRoll @digital_carol]

LFG

[Blockchain at Berkeley @CalBlockchain]

Check out what some of our members did at LionHacks last weekend! @digital_carol @timg51237 @vihanbh @brendan_j_w @imatomster

[CaliRoll @digital_carol]

@Aptos_Network Is there a deadline for application? Trying to collab with more New York based artists to work on a joint project

[CaliRoll @digital_carol]

@dberenzon Very good analogy lol. And each proof writing library assemble a different mechanism so more driver license the merrier

[CaliRoll @digital_carol]

@defi_dua 🙋🏻

[CaliRoll @digital_carol]

@defi_dua I envy thy spring break

[CaliRoll @digital_carol]

@dberenzon I’m waiting for it! That’d need user behavior research - no outsiders understand what goes in the brain when writing circuit 🤫

But sooner later once we have enough circuits, copilot will take it over

[CaliRoll @digital_carol]

@defi_dua The ecosystem for AI infra is just so much more developed than blockchains. So you have to learn farming before cooking, just cuz ingredients cannot be assumed