shivam

writing:

[shivam @sshvmk]

nowhere to go and back from dead after letting everyone down.

[Tom Yeh @ProfTomYeh]

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 https://t.co/2yRBcyz4xy

The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples.

Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced.

• Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors.
• Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left.

The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H.

• Single-head: seq × seq = 40² = 1600 dot products
• Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×)

But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal:

• Single-head: seq² × (H × head_dim) = 40² × 36 = 57600
• Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600

Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

[Elliot Arledge @elliotarledge]

Looks like the perfect paper to throw at Qwen3.6 27B for a local setup.

[shivam @sshvmk]

Its is not really looking good nifty is down 870 pts from todays high

#nifty50

[shivam @sshvmk]

Crazy things are happening in market today

[प्रथम✴︎ @Indraketu1]

Sarasvatī, best of mothers, best of rivers, best of goddesses, we are, as it were, of no repute; grant us, mother, distinction.

~Rig Veda

[shivam @sshvmk]

never be short on God

[shivam @sshvmk]

I seek more
I seek the beyond

[shivam @sshvmk]

the brain cannot store everything. everyday experiences fade, high-salience events persist. it solves this by active memory selection.

what persists, how is it decided ?
> it is implemented by the hippocampus, the core structure for episodic memory (events ordered in time)
> our ability to remember personal experiences as sequences of events in temporal order, like remembering what happened this morning or on your first day at a new job.

hippocampus operates in two modes:
> 1. online (wake) mode
• continuously records experience
• encodes sequences, spatial context, and temporal order
• builds a cognitive map of the environment and events
> 2. offline (sleep) mode
• sensory input suppressed
• replays selected experience traces
• this replay is accompanied by a distinctive pattern of brain activity called a sharp wave ripple (SWRs)

SWRs are brief, high-amplitude hippocampal population events generated by:
- large excitatory input (sharp wave)
- rapid inhibitory feedback (high-frequency ripple)

this excitation - inhibition interplay creates narrow temporal windows for neural firing.
and the result:
> competition between memory representations
> synapses strengthened during wake bias the competition.
> strongly encoded experiences are more likely to reactivate.
> weakly encoded patterns are suppressed.

memory selection emerges naturally from circuit dynamics.
replays are temporally compressed: Seconds of experience → ~100 ms during SWRs

and this compression is critical

during sleep, the neocortex enters a receptive state. precise timing allows hippocampal activity to drive cortical synaptic plasticity. this process is known as systems consolidation

puzzle:
many experiences occur during the day. but consolidation can only happen during sleep. the solution is discovered in recent research
> hippocampus also generates awake SWRs during brief pauses or immobility.
> these awake replays:
• occur immediately after important events
• are too few for consolidation
• occur when cortex is not receptive
> their function is memory tagging, not storage.
> think : neural bookmarks

evidence comes from experiments recording ~400 hippocampal neurons in mice learning a figure-8 alternation task. to decode population activity, researchers used dimensionality reduction (UMAP) to uncover a low-dimensional neural manifold.
result ?
> neural trajectories mirrored the maze geometry
> learning corresponded to systematic movement through this manifold
> awake SWRs projected onto this manifold revealed:
• replays of the just-completed successful trajectory
• trial-specific and temporally compressed

sleep SWRs later replayed the same tagged trajectories, repeatedly.

mechanistic hypothesis : awake SWRs induce local hippocampal synaptic plasticity, biasing network dynamics so tagged patterns are preferentially reactivated during sleep.

FINAL MODEL :
two-stage memory selection system
• wake → select & tag via awake SWRs
• sleep → consolidate via repeated SWRs → cortex

memory is not storage. it is competitive neural selection constrained by circuit dynamics and sleep-dependent plasticity.

[shivam @sshvmk]

https://t.co/M4BBAGQJcf

[shivam @sshvmk]

i have always grossly ignored how important and essential it is to keep moving and keep moving *fast*. i need to be constantly on my feet. working, thinking, going to new places and meeting new people. nothing inspires more than exploration. move more, do more, fuck up more i think thats the key and thats what next year will be about

[shivam @sshvmk]

https://t.co/M4BBAGQJcf

[shivam @sshvmk]

evergrowing youtube garage thread

[shivam @sshvmk]

https://t.co/7FZWzgDFCG

[shivam @sshvmk]

Camus

[shivam @sshvmk]

i love binge watching @veritasium

[shivam @sshvmk]

Everyone please count all your joys today and everyday moving forward

[Alcuin Press @AlcuinPress]

Dostoevsky, Christmas coded

[shivam @sshvmk]

have the courage to believe in the impossible

[shivam @sshvmk]

having a mid life crisis in mid 20s was not on my 2025 card

[shivam @sshvmk]

me core

[shivam @sshvmk]

everything breathes