rahulxf

| gsoc'24 @submitty | lfx @ cncf | oss @

[rahulxf @rahulxf_]

tl;dr: Always keep a recorded demo. 😅

Spoke at Open Source Summit India last week.

This was my first-ever talk at a public conference.

The live demo didn't go as planned, but having a recorded backup saved me.

[rahulxf @rahulxf_]

my lightning talk at OSS India just got upgraded to a full 40 min session 🎉

if you’re attending, add it to your schedule :)

https://t.co/dJmt4j44b4

[rahulxf @rahulxf_]

@__spongeboi tag @nvidia may be they send you gift someday 😂

[rahulxf @rahulxf_]

@__spongeboi 😂

[Richard Seroter @rseroter]

Go is the best language for AI-assisted software engineering. Why?

It's a complete system with built-in eng tools, is designed for readability, offers static typing, has a secure software supply chain, invests in backwards compatibility, compiles fast, and runs anywhere.

[Richard Seroter @rseroter]

"At Netflix, I lead the Go language guild. We've been seen increasing reports of users finding their AI agents writing better Go code than other languages, and increasing reports of projects favouring Go over other languages." https://t.co/8ES4q05YPA

[Jesús Espino @jespinog]

Starting the GopherCon UK! 🎉🎉

[OpenSSF @openssf]

Join us for the upcoming OpenSSF Tech Talk to learn how industry leaders are navigating regulatory milestones and strengthening software supply chain security.

📅 August 20, 1 PM ET / 7 PM CET

Read & register: https://t.co/OD6BfG80X0

[Go @golang]

AI is your newest teammate—a hyper-productive contributor that requires strong guardrails to succeed. Read more on why this partnership makes your choice of programming language more important than ever, and why Go’s early focus on collaboration and reliability makes it an ideal choice for AI-assisted software engineering.
https://t.co/4mffRzeoEF

[Lovepreet Singh @SinghDevHub]

Shopify recently wrote about a surprisingly interesting problem:

How do you reserve inventory for thousands of customers at the same time without overselling?

The obvious SQL solution looks simple.

Imagine a product has:

quantity = 100

When someone checks out:

BEGIN;

SELECT quantity
FROM inventory
WHERE product_id = 123
FOR UPDATE;
UPDATE inventory
SET quantity = quantity - 1;
COMMIT;
Correct?

Yes.
Scalable?

Not really.

The problem is that every checkout for the same popular product tries to lock the same database row.

If 10,000 people are buying the same shoe:

Customer 1 locks:

product=shoe, quantity=100

Customer 2 waits.

Customer 3 waits.

Customer 4 waits.

And so on.

You might have a huge MySQL cluster, but for this particular product your effective concurrency is still close to 1.

You created a HOT ROW.

Shopify solved this by changing how they represented inventory reservations.

Instead of thinking:

quantity = 5

Think:

[unit]
[unit]
[unit]
[unit]
[unit]

Each row represents permission to reserve one item.

Not necessarily a physical serialized item.

Just a reservation token.

Now suppose two customers arrive.
Customer A wants 2 units.
It runs something like:
SELECT id
FROM reservation_units
WHERE product_id = 123
LIMIT 2
FOR UPDATE SKIP LOCKED;
It locks:

unit 1
unit 2

At the same time Customer B runs the same query.

Normally it might wait for those locked rows.

But SKIP LOCKED means:

"That row is busy? Ignore it and find another one."

So Customer B gets:

unit 3
unit 4
Now both transactions can run concurrently.
Instead of:
1000 customers
↓
ONE ROW 🔒
you have:

customer A → unit 1 🔒
customer B → unit 2 🔒
customer C → unit 3 🔒
customer D → unit 4 🔒

The contention gets distributed.
This works because inventory units are interchangeable.

The customer doesn't care which reservation token they receive.

They just need one available unit.

This is very similar to a worker queue.

Workers often do:

SELECT job
FROM jobs
WHERE status = 'READY'
FOR UPDATE SKIP LOCKED;

Worker A takes job 1.
Worker B skips job 1 and takes job 2.

Worker C takes job 3.

Shopify essentially applied the same pattern to inventory.

But there is another problem.
Imagine a warehouse has 500,000 units.
Would you create 500,000 reservation rows?

That would be wasteful.

So Shopify keeps a bounded pool of reservation units.
Think of it like a buffer.
Real inventory:
50,000 units

Reservation pool:

[ ][ ][ ][ ][ ] ... around a limited number of tokens

Customers consume tokens from this pool.

A replenisher fills the pool again from the actual inventory ledger.

So the reservation table is basically a fast working set.

The real inventory ledger remains the source of truth.

Another interesting problem appears when the pool becomes empty.
Suppose real inventory is still available, but all reservation tokens were consumed during a traffic spike.
You don't want 500 checkout requests to simultaneously notice this and all start refilling the pool.
That creates a thundering herd.

So one transaction becomes responsible for replenishing the pool while others wait/retry.

Fast path:

grab an available token.

Rare slow path:

replenish tokens.

Shopify also ran into some very database-specific problems.

One was index locking.

Initially their lookup went through a secondary index and then the clustered primary key.

In InnoDB, locks are closely related to index records.

So the shape of your indexes can directly affect how many locks your query touches.

They changed the primary key to align better with the reservation lookup.

Something conceptually like:

(shop_id, item_id, inventory_group, token_id)

Now reservation tokens belonging to the same item are physically close in the B+ tree.

Important lesson:

Indexes don't only affect query speed.

They can affect locking and concurrency too.

Then they hit gap locks.

Under MySQL's REPEATABLE READ isolation level, a locking query may also lock ranges between index records.

Imagine:

10 ---- 20 ---- 30

MySQL can lock not only 10, 20 and 30, but also gaps such as:

(10,20)

This prevents another transaction from inserting rows into that range.

Useful for preventing phantom reads.

Bad if your algorithm wants to dynamically insert reservation tokens into an empty pool.

Shopify moved these transactions to READ COMMITTED, where this kind of gap locking is reduced for their access pattern.

This is a good reminder:

Higher isolation isn't automatically better.

The correct isolation level depends on the invariant your algorithm actually needs.

They also encountered classic deadlocks.

Imagine:

Transaction A:
locks reservation table
then inventory table

Transaction B:
locks inventory table
then reservation table

Now:

A waits for B.

B waits for A.

Deadlock.

The fix is the same rule we use with mutexes in Java or Go:

Always acquire locks in the same order.

A → B → C

Never:

Flow 1: A → B
Flow 2: B → A

Another optimization was reducing database round trips.

A checkout might contain:

shoes
laptop
shirt
keyboard

Instead of sending one SQL query per line item, they batched reservation queries together.

Because even fast SQL gets expensive when you repeatedly pay for:

network round trip
query execution
result transfer
connection usage

Then came one of the more interesting production lessons.

At some point the system stopped scaling.

The obvious assumption was:

"MySQL must be overloaded."

But CPU wasn't maxed.

Queries weren't particularly slow.
The real bottleneck was:
DATABASE CONNECTIONS.

This matters a lot.
Imagine a service has a pool of 100 DB connections.
A query takes only 5ms.

But your application keeps the transaction open for 100ms while doing other work.

Your connection isn't occupied for 5ms.

It's occupied for 100ms.

At scale:

connection hold time
can matter more than
query execution time.

Shopify added observability to understand which business operations were holding database connections.

Instead of just asking:

"Which SQL query is slow?"

they could ask:
"Which application workflow is consuming our connection capacity?"

That helped them find existing checkout paths that were already using the database inefficiently.

The inventory project didn't create every scaling problem.

It simply pushed the system hard enough to expose them.
The biggest lesson for me from the whole design is this:
Sometimes the solution to database contention isn't Redis, Kafka, distributed locks or another service.
Sometimes the solution is changing the data model.

Shopify transformed:

one hot counter

quantity = 1000
into:
1000 independent reservation opportunities
[token]
[token]
[token]
[token]

Then used:

SELECT ... FOR UPDATE SKIP LOCKED

to let concurrent transactions grab different tokens.

One global lock became many independent locks.

And because the reservation state lives with the inventory state in MySQL, they can also use normal ACID transactions instead of coordinating Redis + MySQL writes.

The pattern is worth remembering:

Hot counter
↓
Turn capacity into tokens
↓
Let transactions claim different tokens
↓
Use SKIP LOCKED
↓
Keep the token pool bounded
↓
Replenish when needed
↓
Keep lock ordering consistent
↓
Measure connection usage, not just query latency

A very nice example of solving a distributed-systems problem by understanding the database deeply instead of immediately adding more distributed systems like kafka, redis or queues

[rahulxf @rahulxf_]

@iximiuz Thank you, @iximiuz

[rahulxf @rahulxf_]

@gumroad GitHub

[Ellie Huxtable @ellie_huxtable]

kubernetes agent sandbox actually seems pretty good

[Akash 🇬🇧/🇮🇳 @thewritingdev]

Jamie has done a great job with this resource on DNS.

Very valuable and extremely informative!

[rahulxf @rahulxf_]

@AtharvaXDevs find someone for oss summit japan (24 aug is last date)

[rahulxf @rahulxf_]

they get out of credit so forgot to send me rejection mail 😭

[Atharva @AtharvaXDevs]

one more rejection :p

[rahulxf @rahulxf_]

happy life i am noob 😛, lets get back to work

[rahulxf @rahulxf_]

@AtharvaXDevs naa

[Ilya Grigorik @igrigorik]

Shipped AgentBoard 0.9, huge upgrade.

👉 https://t.co/KHHIDRFMCx

You can now mount a local workspace that will auto load your AGENTS.md, IDENTITY.md, etc. Better yet, it unlocks read+write support for MEMORY.md — the agent can save and retrieve context across threads, and everything lives as plain markdown on your device.

h/t @tobi for the workspace nudge, huge unlock.

p.s. bonus: did you know that millions of Shopify storefronts ship 🪄 WebMCP tools for catalog + cart? Makes a *huge* difference in speed and quality of agents assisting the user on the storefront.

[rahulxf @rahulxf_]

https://t.co/dUCTa8A2yq

[rahulxf @rahulxf_]

@AtharvaXDevs hnn , m nhi aunga i have other stuff to do in that timeline

[rahulxf @rahulxf_]

@AtharvaXDevs