haewon

[Alex Lieberman @businessbarista]

This is one of the best breakdowns on the fundamentals of LLMs I've ever read.

Anytime someone asks me for resources to climb the steep AI learning curve, I always provide the same list.

1) @3blue1brown's neural network videos
2) @karpathy's zero to hero playlist
3) @dwarkesh_sp's whiteboard explainers

Now @_raghavdixit_'s "Vectors are all you need" and future articles in the explainer series are getting added to the list.

[haewon @hwnprc]

my daily note goal

[Umut Güner @umuttgunerr]

Ford ve Porsche için tasarımlar yapan kıdemli tasarımcı Ege Argüden'in günlük notlarını tuttuğu defterleri dünyada gündem oldu.

[haewon @hwnprc]

Attended an onchain OS session by OKX.
my goal: turn 10 usdt into 300 by the end of this month

#hwpinseoul

[haewon @hwnprc]

Day 5. LLM from scratch

why do n-gram language models fail?
• curse of dimensionality: possible n-grams grow exponentially, making data sparse.
• zero-count problem: unseen n-grams get zero probability, so perplexity become infinite.
• no semantic meaning: words are treated as integer IDs, without understanding similarity or context.

[haewon @hwnprc]

how can one develop a taste in research?

#hwpinseoul

[haewon @hwnprc]

@EXM7777 @eptwts @madiyarzm

[Vlad Feinberg @FeinbergVlad]

How to land a job at a frontier lab

https://t.co/oHIqLgBMbC

[haewon @hwnprc]

<A neural probabilistic language model>

[haewon @hwnprc]

Day 4. LLM from scratch

• n-grams fail on unseen word combination (probability 0 -> infinite perplexity). neural model fixes this using word embeddings, where similar words cluster tgtr.
• this generalization is a double-edge sword (hallucination, repetition, and bias)

[haewon @hwnprc]

Day 3. LLM from scratch

• intro of neural language model
• core concepts: curse of dimensionality, word embeddings, neural networks, data preparation, CPU parallelization

[haewon @hwnprc]

@benln would love to join!

[haewon @hwnprc]

Day 2. llm from scratch

• built n-gram model from scratch, trained on a Korean novel.
• starting from tmrw, moving to neural language models.

[haewon @hwnprc]

Day 1. LLM from scratch

• Implemented character-level N-gram models in C.
• trained 6 gram on 442k korean words vs. 30k
-> accuracy went from 45% to 94%
• empirically observed the scaling law: data volume matters more than model complexity for coverage