Oliver

[NASA @NASA]

Hello, Moon. It’s great to be back.

Here’s a taste of what the Artemis II astronauts photographed during their flight around the Moon. Check out more photos from the mission: https://t.co/rzM1P0QbOl

[Elon Musk @elonmusk]

NASA Artemis passing close to the Moon

[Oliver @oliverjlee]

@tim_cook Foldable iPad air?

[Andrej Karpathy @karpathy]

📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
https://t.co/QTUdu8b0qh

The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
https://t.co/BOzkxQ8at2

Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo

[Carl Carrie (@🏠) @carlcarrie]

The Inelastic Market Hypothesis: A Microstructural Interpretation

Market Impact, Order Flow and Latent Liquidity Theory
#TCA Paper by Jean-Philippe Bouchaud

https://t.co/FAW6raJkkx

[Oliver @oliverjlee]

Elon's Twitter Play: Valuation Question or Political Rorschach Test? https://t.co/GUucMRJZJQ via @YouTube

[Carl Carrie (@🏠) @carlcarrie]

QF-lib (QuarkFin) is a financial Python library built at CERN by their Pension team and has features for back testing, portfolio construction, time series analysis, risk monitoring and financial charting

About: https://t.co/dwBATVM8bY

Python GitHub:
https://t.co/lCX4dyzM5S

[Carl Carrie (@🏠) @carlcarrie]

Markov Transition Fields (MTF) time-series to image transformation

Article:
https://t.co/BfxHUwuBur

Jupyter Python Notebook:
https://t.co/HpbvSy8r0r

[Carl Carrie (@🏠) @carlcarrie]

Optimal deep learning based trading using convolutional transformers - paper

https://t.co/p9wsm48tZA

[Carl Carrie (@🏠) @carlcarrie]

Quantum Computing Algorithm developed by IBM and Barclays applied to Trade Settlement Optimization

#QUBO

https://t.co/tiR1WLrUJj

[Carl Carrie (@🏠) @carlcarrie]

Similarity measures in machine learning - 17 types
https://t.co/nFn55xzbfV

[Carl Carrie (@🏠) @carlcarrie]

#AI / Code Generation using Transformers

https://t.co/Om1HtjXYg9

More on Transformers
https://t.co/Pvt4mtKimw

[Carl Carrie (@🏠) @carlcarrie]

Swap curve fitting using TensorFlow Quant Finance (TFF)

https://t.co/Mja3NcY5FC

https://t.co/LTYPUEeEDO

[Carl Carrie (@🏠) @carlcarrie]

Black Box model risks inclusive of adversarial attacks - paper

https://t.co/Bw3qbcgUvk

[Compound248 💰 @compound248]

Dear Media,

What’s happening with RobinHood?

A quick primer.

This is a “plumbing” issue. It is esoteric, even for those on Wall Street.

A very long thread on how the toilet is clogged.🚽🧻🪠

Read on
👇👇👇💎💎💎🚀🚀🚀👇👇👇

[Carl Carrie (@🏠) @carlcarrie]

QLib - by Microsoft Research
Investment Platform based on #AI principles

Paper:
https://t.co/iB3WWP9eyk

GitHub:
https://t.co/ezpuzlJfiw

[Carl Carrie (@🏠) @carlcarrie]

Handcalcs - auto rendering of Python formulas in Jupyter

https://t.co/Jcq9S2Iyl8

[Carl Carrie (@🏠) @carlcarrie]

Simple trend detection for stocks: Trendnet

Python code:
https://t.co/IktpILBvkL

[Carl Carrie (@🏠) @carlcarrie]

High-Frequency Realized estimators of volatility - paper

https://t.co/QgxSHSm0u4

[Marcos López de Prado @lopezdeprado]

A closed-form solution for optimal mean-reverting strategies https://t.co/DegU4eeLgQ