Marcel

Software Engineer - mastodon: [email protected]

[Mitchell Hashimoto @mitchellh]

A bunch of alternate icon choices are coming in Ghostty 1.1.1. 😍 You already have the ability to customize the colors of the default Ghostty icon, but these artist-drawn variants are super cool and will help make your Ghostty style more you.

[maharshi @maharshii]

sometimes it’s okay to give up a bit of productivity to ensure that your problem solving skills remain intact.

[Mitchell Hashimoto @mitchellh]

https://t.co/39Xj39wheA 👻

[teej dv 🔭 @teej_dv]

Merry Christmas 🎅

[Mitchell Hashimoto @mitchellh]

100%. Ready to go. The release will probably be on the 26th though so I can take time to enjoy the holidays with my family without stressing about inevitable launch day issues. For those interested: thanks for hanging in there, not long now. Happy holidays y'all. ❤️

[Marcel @hmrc87]

@_aufgegleist @justusbersten @martiinhier :D

[Andrej Karpathy @karpathy]

📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
https://t.co/QTUdu8b0qh

The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
https://t.co/BOzkxQ8at2

Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo

[Marcel @hmrc87]

@pocketcasts will we be able to sync our progress between Apple Watch and iPhone? I want to continue an episodes on my watch or iPhone when starting it on the other device

[ZOLLHOF @zollhofHQ]

2/2 HACK|BAY days done and with that, our 8th hosted #hackathon in a row ✔️ We are still super thrilled and excited over the past two days.

Congrats to the winning teams and a big thank you to
our partners, supporters and about 330 national and international attendees.

[Marcel @hmrc87]

@NobleOne13454 @waybackintimes @Enezator Helicopters can land without power- pilots train this quite a bit. It’s called autorotation https://t.co/1i5l6kxek3

[NuPhy @nuphystudio]

🎁New Keycaps Giveaway!⁠
⁠
To enter:⁠
1. Follow @nuphystudio;⁠
2. Like the post and repost it;⁠
⁠
Winner: 1
Prize: 1 of these 2 keycaps⁠
Date: April 18 - April 25

[ThePrimeagen @ThePrimeagen]

@unclebobmartin @mrclbschff I think we could make this happen

[NuPhy @nuphystudio]

⁠⏳️⁠Launching SOON⁠
⁠
Keycaps: Space Engineer🌌⁠
Compatibility: Air Series⌨️

#mechanicalkeyboard #keycaps #wirelesskeyboard

[Marcel @hmrc87]

Writing tests also helps understanding the code and changes the way we think about it.

[ThePrimeagen @ThePrimeagen]

“The fundamental problem of software testing… is that software has to handle many situations that the developer has never thought of or will never anticipate. This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too. This makes conventional testing great for catching regressions, but really terrible at catching all the “unknown unknowns” that life, the universe, and your endlessly creative users will throw at you.” - Will Wilson

[Marcel @hmrc87]

@TheJackForge @ThePrimeagen But will he be able to deliver as good when the Netflix influence is gone?

[Marcel @hmrc87]

@nuphystudio Mint @NoFriends_Alone @NoFriendOnTwit

[NuPhy @nuphystudio]

🎁Mint, Raspberry, and Lemon. Which switch option tickles your ear?😉

To enter the giveaway:
1. Follow @nuphystudio;
2. Like and repost;
3. Tag 2 friends in the comment.

Winners: 1
Prize: 1 × Mint/Raspberry/Lemon switches
Date: Jan. 5 - Jan. 12

[Holger Sturm @holgersturm82]

#Deepfake Markus Söder - Atemlos (tapfer bleiben)