shivamm

Software Engineer

[adi @theoldmnk]

just scaled a new brand to $2.4M ARR in the last few weeks

everything everyone told you about building consumer brands is a lie

if you're young, ambitious and can grind 16hrs a day for the next 6 months of your life

i promise you i can show you how to achieve these outcomes

DM me if you alr know me or AMA on the comments

[rishabh @rishabhguptajs]

i spent the last 10 days trying to build a language model that never runs out of memory, on a MacBook Air.

it works. it's faster, it uses less energy, and it writes better tool calls than a transformer of the same size. but last night it failed the one test I cared about most.

here's the full story, with real numbers.

the goal

transformers have a dirty secret: the longer they read, the heavier they get. every word they've seen sits in memory (the "kv cache"), and every new word has to look back at all of them. a robot or an agent that runs for hours can't live like that.

so i set out to build a small model where:

• memory stays the same size no matter how long it runs • cost per word stays the same at word 10 and at word 10 million
• quality matches or beats a transformer of the same size

everything was trained from scratch on one 16 gb m4 macbook air, using apple's mlx: 11 languages (including hindi, chinese, arabic and swahili), plus tool-call data. each model was ~14m parameters and saw 49m tokens, with each run taking ~2 hours. every number below comes from a logged run.

what i built

a hybrid: 9 gated deltanet layers (a fixed-size recurrent memory, from existing research) plus 3 layers of my own idea, which i call priority memory.

priority memory is a small bank of exact "slots": 64 per head. every word gets a learned importance score. the bank always holds the highest-priority words seen so far, and the rest get evicted. it's like a notebook with a fixed number of pages where you only keep what matters.

compared against:
• a tuned transformer (same size, same data, same everything)
• the same model with my layer removed (pure deltanet), to check whether my idea actually does anything

—

round 1: efficiency (big win)

memory needed to keep reading:
• transformer: 13 mb at 512 tokens → 805 mb at 32k tokens (and still growing)
• mine: 1.07 mb. at every length. forever.

total ram at 32k tokens: 2.5 gb vs 176 mb.

speed generating text:
• transformer: 585 tokens/sec at 512 → 124 at 8k → 30 at 32k
• mine: ~326 tokens/sec at every length

the transformer is faster on short text. but at 8k tokens mine is 2.6x faster, and at 32k it's 11x faster.

energy (measured with powermetrics, above idle):

• generating: 23.6 → 17.9 mj per token (−24%)
• reading: 0.83 → 0.32 mj per token (−61%)

—

round 2: quality (small win)

bits per byte on held-out text, lower is better:
• transformer: 1.642
• deltanet only: 1.636
• mine: 1.619

mine won on all 11 languages, plus tool-call text (−4%).

tool calls, fully correct json with the right name and arguments: transformer 38%, deltanet only 37%, mine 44%

—

a lesson in between: short runs lie

before the full runs, i did quick 12m-token tests to tune learning rates. in those, the deltanet-only model beat mine. i honestly thought my layer was useless.

then i ran the full 49m tokens. deltanet-only started faster, but mine kept improving, overtook it around step 3,500, and finished ahead on every language.

if i had trusted the short runs, i would have thrown away the idea lol.

—

round 3: memory for facts (where it gets interesting)

the test: plant a fact ("the secret code for x is 48213"), add filler text, then ask for it. the score is how many bits more the model prefers the true code over a fake one.

at distance 0:
• transformer: 10.6
• mine: 6.3
• deltanet only: 1.6

so my layer gives ~4x the recall of pure deltanet. that's the job it was designed for. the transformer was still better, but every model collapsed beyond 512 tokens, which is the length they were trained on.

fair enough: 512 tokens is too short for a long-term memory to matter. so i ran the real test.

—

round 4: the long-text test (the failure)

all three models got 12m more tokens of training on 2,048-token text. a quarter of it was recall practice: facts planted early, questions much later. the wording and names were deliberately different from the test, so they couldn't just memorise phrasing.

my expectation: the slot memory finally gets a job, learns what to keep, and pulls ahead.

recall afterwards (before → after):

transformer:
• 0 tokens away: 10.6 → 15.9
• 512 away: 1.5 → 9.9
• 1,024 away: 0.2 → 4.6
• 1,800 away: 0.03 → 1.7

mine:
• 0 away: 6.3 → 3.6
• 512 away: 1.5 → 1.2
• 1,024 away: 0.9 → 0.5

deltanet only:
• 0 away: 1.6 → 1.0

the transformer learned to remember across its whole window. both fixed-memory models got worse.

text quality ended up tied (1.617 mine vs 1.619 transformer). tool calls: still mine, 42% vs 38%. and nobody recalled anything beyond 4k tokens (beyond the training length).

—

why i think it failed

my best guess: my memory decides what to throw away, but nothing in training ever tells it "you threw away something you needed later". the keep/evict decision is a hard cut-off, so no learning signal flows through it. it literally cannot learn from its own forgetting.

two smaller suspects:
• 3 of my 4 memory heads have a fixed bias toward recent words, so only one head can hold a fact for 1,000+ tokens
• ~1,500 recall examples is probably too little practice

a transformer doesn't have this problem. it never forgets anything inside its window, so it just has to learn where to look. a fixed-memory model has to learn what to keep, and that is a much harder lesson.

—

what i take away from this

constant memory is the easy part. you get the efficiency for free, by construction.

recall has to be earned, and the hard part isn't the memory structure. it's the training signal that teaches the model what's worth keeping.

judge architectures on full runs and on the task they're built for, not on short runs or average loss.

you can do real architecture research on a laptop.

caveats, because they matter: tiny models (14m), one seed each, single machine. this is a lab notebook, not a paper.

if you work on recurrent models, state-space models, or long-context memory, i'd love to hear how you'd attack the "learning what to keep" problem.

[shivamm @codecShivam]

Posting my minimal setup so you post yours and I can take some inspiration 👀🙌🏻

[utkarsh @twtutkarsh]

the web2app funnel where users pay on the web and open the iOS app already logged in, even before it's installed

acquire on web and retain them on mobile with 0 friction.

https://t.co/sfgJzHcO9w

[shivamm @codecShivam]

@k_sauraj https://t.co/K6zqXX40wz

[Sauraj @k_sauraj]

hey , I’m Sauraj.
21, final year CSE | DevOps engineer.

- In class 11th: worked at PPUI and CipherOS (2021)
- In starting of class 12th: built an Android dump automation workflow that got 500+ dumps
- In the end of class 12th: built a JEE CLI counselling tool used by 3.5K+ users, with 0 marketing
- 2023: worst year of my life. Grinded for JEE and still ended up at a tier-3 college.
- 1st & 2nd year: explored as many domains as I could and became president of one club.
- In the end of 2nd year: started interning in DevOps, the domain I ended up loving the most.
- In my pre-final year: got a PPO at an L2 role.

one thing i’ve realised through all these years: i fkinn love tech and want to contribute to it as much as i can.

So here i am. will share projects, learnings, and try to help folks from tier-3 colleges :P

[rishabh @rishabhguptajs]

i gave claude opus 5 random photos and asked it to doodle on them. no image gen, every wobbly line is javascript on a canvas ✏️ it draws them stroke by stroke, then they come alive. even the music is code 🎶

check below 🔽

[shivamm @codecShivam]

so this is four hooks and one mcp server, all public api. nothing patched, no network, no telemetry. a bun binary and a json file per session

https://t.co/0tY650Ui9a

[shivamm @codecShivam]

two minutes into a claude code task, you already know the next things you want. interrupt it, or sit there and try to remember
so i built a third option. a queue next to claude. type it now, edit it, reorder it, park it for tomorrow. it lands when the turn ends

[shivamm @codecShivam]

the other way to get this is patching claude code's binary. that is a tos gray area and it breaks every time claude updates

[shivamm @codecShivam]

@beingivish

[shivamm @codecShivam]

mf!

[shivamm @codecShivam]

No way 😭

[shivamm @codecShivam]

@BlueCircle0 then Pasta bar veneto

[shivamm @codecShivam]

@thejasnars Can I join pls??

[shivamm @codecShivam]

@BlueCircle0 @rai_shiv_ @buildwithsid

[rishabh @rishabhguptajs]

🧵built a local visual explorer for pgvector embeddings.

paste your postgres connection string → pick a table → see your embeddings as a 2D scatter plot in seconds. no cloud. no auth. no bs. runs entirely on your machine

https://t.co/hNOkNDyTXN

[Anthropic @AnthropicAI]

Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.

It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
https://t.co/NQ7IfEtYk7