Suryansh Sinha

20

[Suryansh Sinha @sspub2]

Day 31 of LLMs from scratch

Covered today:

→ the basic idea behind GPTQ, SmoothQuant,and AWQ
→ some math around scaling and zero-point
→ floating point formats and why they matter

[Suryansh Sinha @sspub2]

For the next few days, I’ll be focusing on learning model quantization.

[Suryansh Sinha @sspub2]

Day 30 of LLMs from scratch

Pretrained my medical SLM and ran inference on it.

→ 15M parameter model, 70M tokens, 20K steps
→ Val loss 3.40, perplexity 30 (almost no overfitting this time)
→ Model generates coherent medical text from scratch

[Suryansh Sinha @sspub2]

Day 29 of LLMs from scratch

Trained my medical SLM,

Training loss dropped from ~10 down to 0.15, but validation loss stayed at 8.5 with perplexity of 5389.

The model had memorized the training data, I guess 11M tokens were not enough for a 92M parameter model to generalize🤡

[Suryansh Sinha @sspub2]

I ran pretraining for 10,000 steps,

Now I'm starting over the right way:

→ Downloading 500K+ PubMed abstracts (~78M tokens, 7× more data)
→ Reducing model to ~15M parameters to match the dataset size
→ Retraining for 20,000 steps

[Suryansh Sinha @sspub2]

Day 29 of LLMs from scratch

Trained my medical SLM,

Training loss dropped from ~10 down to 0.15, but validation loss stayed at 8.5 with perplexity of 5389.

The model had memorized the training data, I guess 11M tokens were not enough for a 92M parameter model to generalize🤡

[Suryansh Sinha @sspub2]

Day 28 of LLMs from scratch

Updates on medicalSLM:

-> Wrote the main training loop
-> Training has officially started

Current estimate:
10,000 steps × “please don’t crash” = a small eternity

Realistically, ~20–40 hours… assuming everything runs smoothly

[Suryansh Sinha @sspub2]

Day 28 of LLMs from scratch

Updates on medicalSLM:

-> Wrote the main training loop
-> Training has officially started

Current estimate:
10,000 steps × “please don’t crash” = a small eternity

Realistically, ~20–40 hours… assuming everything runs smoothly

[Suryansh Sinha @sspub2]

Day 27 of LLMs from scratch

Update:
-> Defined learning rate, training steps, and optimizer (AdamW)
-> Working on two projects side by side - one on RL, one on LangGraph

[Suryansh Sinha @sspub2]

Day 27 of LLMs from scratch

Update:
-> Defined learning rate, training steps, and optimizer (AdamW)
-> Working on two projects side by side - one on RL, one on LangGraph

[Suryansh Sinha @sspub2]

Day 26 of LLMs from scratch

Update:

-> Defined the training configuration for the SLM.
-> Started revising Langgraph, revisited some topics.

[Suryansh Sinha @sspub2]

The most honest thing an LLM can say is "I don't know" and we spent years training that instinct out of them

[Suryansh Sinha @sspub2]

Day 26 of LLMs from scratch

Update:

-> Defined the training configuration for the SLM.
-> Started revising Langgraph, revisited some topics.

[Suryansh Sinha @sspub2]

Day 25 of LLMs from scratch

Built the transformer architecture from scratch in PyTorch.

→ 92M parameter model
→ TransformerBlock with residual connections and gradient checkpointing
→ Implemented RMSNorm, RoPE, Grouped Query Attention, SwiGLU FFN

Next: pretraining loop

[Suryansh Sinha @sspub2]

Day 25 of LLMs from scratch

Built the transformer architecture from scratch in PyTorch.

→ 92M parameter model
→ TransformerBlock with residual connections and gradient checkpointing
→ Implemented RMSNorm, RoPE, Grouped Query Attention, SwiGLU FFN

Next: pretraining loop

[Suryansh Sinha @sspub2]

Day 24 of LLMs from scratch

Today I tried understanding LLaMA's architecture as it basically covers almost everything I am trying to achieve through my medical SLM.

Things like KV Cache, Rotary Positional Embeddings, RMS Norm, Grouped Query Attention, SwiGLU, etc.

[Suryansh Sinha @sspub2]

Day 24 of LLMs from scratch

Today I tried understanding LLaMA's architecture as it basically covers almost everything I am trying to achieve through my medical SLM.

Things like KV Cache, Rotary Positional Embeddings, RMS Norm, Grouped Query Attention, SwiGLU, etc.

[Suryansh Sinha @sspub2]

Day 23 of LLMs from scratch

→ Implemented RMSNorm from scratch
→ Understood rotary positional embeddings (RoPE) and implemented them
→ Understood why GQA cuts KV cache 4× compared to multi-head attention

Before I proceed, I need to learn more about RoPE, SwiGLU and RMSNorm

[Suryansh Sinha @sspub2]

Day 23 of LLMs from scratch

→ Implemented RMSNorm from scratch
→ Understood rotary positional embeddings (RoPE) and implemented them
→ Understood why GQA cuts KV cache 4× compared to multi-head attention

Before I proceed, I need to learn more about RoPE, SwiGLU and RMSNorm

[Suryansh Sinha @sspub2]

Day 22 of LLMs from scratch

Things I learned while building my medical SLM:

→ Why transformers need positional encoding
→ Grouped Query Attention (implemented 8 query heads share 2 KV heads)
→ RMSNorm vs LayerNorm

Next: writing RoPE + Grouped Query Attention block

[Suryansh Sinha @sspub2]

Day 22 of LLMs from scratch

Things I learned while building my medical SLM:

→ Why transformers need positional encoding
→ Grouped Query Attention (implemented 8 query heads share 2 KV heads)
→ RMSNorm vs LayerNorm

Next: writing RoPE + Grouped Query Attention block

[Suryansh Sinha @sspub2]

Day 21 of LLMs from scratch

Building a medical small language model from scratch.

→ 70K+ PubMed abstracts from NCBI
→ Custom BPE tokenizer on medical text
→ Full data pipeline, binary training files ready

Next thing now is to write the transformer block in PyTorch.

[Suryansh Sinha @sspub2]

Day 21 of LLMs from scratch

Building a medical small language model from scratch.

→ 70K+ PubMed abstracts from NCBI
→ Custom BPE tokenizer on medical text
→ Full data pipeline, binary training files ready

Next thing now is to write the transformer block in PyTorch.

[Suryansh Sinha @sspub2]

Day 20 of LLMs from scratch

I will be following these two playlists from now on

→ Starting Reasoning LLMs from Scratch
→ Following Transformers for Vision

Focusing on both reasoning + architecture to build deeper

[Suryansh Sinha @sspub2]

Day 19 of LLMs from scratch #100DaysOfCode

Started learning about fine-tuning

Also started with vision transformer architecture and vlms

[Suryansh Sinha @sspub2]

Day 19 of LLMs from scratch #100DaysOfCode

Started learning about fine-tuning

Also started with vision transformer architecture and vlms

[Suryansh Sinha @sspub2]

Day 18 of LLMs from scratch #100DaysOfCode

Learnt some techniques which are used to control the randomness and creativity.

-> Temperature
-> Top-k sampling

[Suryansh Sinha @sspub2]

Day 18 of LLMs from scratch #100DaysOfCode

Learnt some techniques which are used to control the randomness and creativity.

-> Temperature
-> Top-k sampling

[Suryansh Sinha @sspub2]

Day 17 of LLMs from scratch #100DaysOfCode

Learned about how does loss function works in llms
->cross entropy loss and perplexity

[Suryansh Sinha @sspub2]

Day 17 of LLMs from scratch #100DaysOfCode

Learned about how does loss function works in llms
->cross entropy loss and perplexity

[Suryansh Sinha @sspub2]

Day 16 of LLMs from scratch #100DaysOfCode

Topics Revised:
-> Layer Normalization
-> Gelu
-> Shortcut connections

[Suryansh Sinha @sspub2]

Day 16 of LLMs from scratch #100DaysOfCode

Topics Revised:
-> Layer Normalization
-> Gelu
-> Shortcut connections

[Suryansh Sinha @sspub2]

Day 15 of LLMs from scratch #100DaysOfCode

Revised implementing a simplified attention mechanism (with trainable weights).

[Suryansh Sinha @sspub2]

Day 15 of LLMs from scratch #100DaysOfCode

Revised implementing a simplified attention mechanism (with trainable weights).

[Suryansh Sinha @sspub2]

Day 14 of LLMs from scratch #100DaysOfCode

Revised implementing a simplified attention mechanism (without trainable weights).