Sudeep🚢

Software Engineer, working to quit the rat race and build a one-person business

[Abhishek Singh @0xlelouch_]

Tech story time !!

One year ago, we had one of those production bugs that was almost impossible to notice.

We had a Kafka consumer processing events from another service.

99.9% of the time everything worked fine.

Consume message -> process it -> update DB -> move on.

But once in a while, processing would fail.

Maybe the downstream service timed out.

Maybe some payload had unexpected data (some edge cases here that weren't handled).

Sometimes the DB connection dropped for a few seconds.

Nothing unusual in distributed systems.

The problem was our retry logic.

We retried the message a few times, and after that... we basically gave up.

No DLQ.

The consumer kept moving forward, so from the outside the system looked completely healthy.

CPU fine.
Kafka lag looked fine.
Error rate almost zero.
No major alerts.

But that tiny percentage of messages were effectively lost from the business workflow.

And because it happened so rarely, nobody noticed immediately.

Weeks later someone found a mismatch between two systems.

We started tracing IDs manually and eventually found events that had entered Kafka but never completed processing.

That was the scary part actually.

The system was not "down" per se.
But it was silently wrong.

We fixed it by introducing a proper retry + DLQ flow.

Normal topic
-> consumer
-> temporary failure
-> retry topic with backoff
-> retry again
-> after max attempts -> DLQ

But just adding a DLQ was not enough.

We also added the original payload, error reason, retry count, timestamp and correlation ID so someone could actually debug the failed message.

Then alerts for DLQ growth.

Then a small replay tool so after fixing the underlying bug, we could safely send failed events back through the pipeline.

We also had another bug which led us to make consumers idempotent because replaying messages without idempotency can create an entirely new problem.

Duplicate payments, duplicate rows, duplicate notifications... whatever your consumer happens to do.

That incident really taught me something important about event-driven systems:

Message loss does not always look like message loss.

Most of the times Kafka is healthy, consumers are healthy, dashboards are green...

But your business data is slowly becoming inconsistent.

A DLQ is not just a queue where "bad messages go".

It is a safety net for failures you did not predict.

And in distributed systems, there will always be failures you did not predict.

[Sudeep🚢 @SudeepTek]

@egr_investor I dont understand the reason behind all the gymnastics. I know many friends who bought a house 10yrs ago and they have aggressively paid off their mortgage. That's such a relief. No amount of rent vs buy gives you that kind of peace.

[Sudeep🚢 @SudeepTek]

@egr_investor IMHO 401K is not meant for aggressive growth. Atleast for me. I would rather preserve wealth and grow the money at 10% or whatever which is still significant.

[Sudeep🚢 @SudeepTek]

@theliverdoc I treat supplementation as a medicine. Take it only when you know you are deficient. All the "its proven to do this or that" is horseshit. There is too much variability.

[Sudeep🚢 @SudeepTek]

@seandsweeney At NYU I can audit a course.

[Sudeep🚢 @SudeepTek]

@GuruAnaerobic Guru, do you have any guide or workout program that you recommend?

[Sudeep🚢 @SudeepTek]

@theliverdoc @BeerBicepsGuy I enjoy long-form conversations and when @theliverdoc is talking, you better listen. Cant wait to watch/listen to the conversation!

[Sudeep🚢 @SudeepTek]

@jschultzf3 In other words, sell CSPs on whats going to go up!

[Sudeep🚢 @SudeepTek]

@theliverdoc @JarAppHQ @Akash_singla99 Exactly right, it beats the purpose. Similarly when people suggest vegetarians to eat paneer to meet their protein goals, they forget that paneer comes with a lot more fat. It beats the purpose.

[Sudeep🚢 @SudeepTek]

@Dickies_Mom @dickiebush @Dickies_Mom thanks for putting things in perspective. @dickiebush take a note 😜

[Sudeep🚢 @SudeepTek]

Man up only when it suits you.

[ZUBY: @ZubyMusic]

Every society expects men to sacrifice (up to and including their own lives) for things bigger than themselves - God, family, country, respect, honour, duty, etc.

But as these things as systematically eroded, men rationally become less willing to jump into the fire.

Modern society expects young men to keep running into the firing line, metaphorically and literally, without ever explaining why they should, nor rewarding them adequately for doing so.

"Man up" just doesn't cut it anymore.

[Sudeep🚢 @SudeepTek]

@theliverdoc This tweet is going to put lot of companies out of business.

[Sudeep🚢 @SudeepTek]

@Meta_Trav Holy shit. This guy gets it.
This has been my experience.

[Nicolas Cole 🚢👻 @Nicolascole77]

Be forever in beta.

There's no final version of you.

You're always growing and changing.

[TheLiverDoc™ @theliverdoc]

I am a Hepatologist.

Arguably, I care for the sickest group of patients in medical practice – end stage liver disease.

I see death almost every day in my ICU.

Some are natural deaths, leading to multiple organ failure triggered by a failing liver.

Some are decided. The family decided to pull the plug. They decide it was time to let go.

And every time, I see the heart rate drop, the blood pressure fall and the electrocardiogram go flat.

And I wonder. What goes on inside a dying brain?
Was it a small black hole, that gets bigger and bigger and then engulfs the field in darkness as the heart shuts down taking the brain away with it?

Or was it just all light at the end of the tunnel and then darkness, once the destination had reached?

A group of doctors did something strange. Something unimaginable.

And it led to them witnessing something melancholic, but beautiful.

They went inside an ICU. Saw that four patients were unconscious.

They were not clinically dead, yet. And not brain-dead. But were comatose.
They were kept alive by a machine that pumped oxygen into their lungs.
Three had brain injury due to cardiac arrest. One had haemorrhaged into the brain.

All were unresponsive for a very long time and sadly, the family decided to pull the plug on them.
To let them go away, peacefully.

But before they did that, the loving family did something bittersweet. They signed a document which was made by the group of doctors, allowing them to study their beloveds’ dying brain.

The group was led by Dr Jimo Borjigin of the Department of Molecular and Integrative Physiology, University of Michigan School of Medicine, Ann Arbor at Michigan.

The doctors fitted machines to the heart and brain of the dying patients and then withdrew their life support by removing the breathing tube.

Immediately after the tube was removed, the brain activity did not show anything different.
But as the brain got closer to death, about 300 seconds before death, a power surge. And it occurred in the regions of the brain which was associated with conscious experience. Like the dying was suddenly yearning to live.

And make no mistake, these were not “some bursts” of electrical activity that happens when the brain cells withered away without oxygen. This was a special kind of activity.

These power surges increased connectivity in a region in the brain called posterior cortical “hot zone” – an area crucial for conscious perception – when it came to just before dying, the brain cells were having a cross-talk, not dying away into oblivion.

They were passing final messages to each other within the regions of the brain, while the physical body lay, limp and motionless.

And the doctors were heartbroken – because the final electrical pattern of surge that the dying brain was showing was the similar to those seen in dreaming humans.

All the patients in the study died.
But before they died, they taught us something.
That even in death, our minds are resilient.
That we clutch at only everything that is good.

What were they dreaming of?
A husband, about his loving wife?
A mother, her children?
Children, their parents?
Or maybe a scene from a movie that changed you?
Or a song under a starry sky?
Or a pet who loved you unconditionally?
We will never know.

But know this, even in death, the mind conquers – with dreams and love, not nightmares and hate.

Remember, nothing perfect lasts forever, except in our memories.

Make good memories while living, so that we can re-live them in death.

Go hug you family, take your children for an ice cream. Walk your dog into the sunset.

Sources:
➡️"Surprising brain activity moments before death" by Dr. F Perry Wilson @fperrywilson via https://t.co/g2NmWlMrAs

➡️Research Article: Surge of neurophysiological coupling and connectivity of gamma oscillations in the dying human brain from @PNASNews https://t.co/YqvZIlDaxN

[Sam McGregor - Social Skills Coach @SpeechWrx]

Relationships: @taylorburrowes
Reality check: @robwood
Encouragement: @jwginn
Heavy Iron: @MorganAgran
Thoughts: @Coach_Schultz
Incredible: @abharris03

Whose done the same for you?

[Sudeep🚢 @SudeepTek]

Love this. Get used to the fear and do the thing you are supposed to do.

[Nat Eliason @nateliason]

You can handle much more than you think.

A few years ago, I got a call from a mortgage lender.

They wanted to confirm that a customer of theirs was an employee of mine, that the employee was currently employed, and that they were indeed earning the salary they said they were.

I confirmed and gave them the information they needed, then hung up and went back to work.

A few minutes later, it hit me. Someone’s house now depended on me not screwing up.

It really, really stressed me out. For a few hours. Then I started to get used to it. And eventually, I felt completely normal going to bed and waking up every day with employees who had mortgages.

But if you had gone back to me in college trying to figure this blogging thing out, and said, “Hey Nat, starting tomorrow, twenty people’s livelihoods will depend on you continuing to write good articles each week,” I would have freaked out. That is a terrifying change to be confronted with.

And in some ways, that terror is justified. I wouldn’t have had time to adjust to that reality, so imagining stepping into it is terrifying. But actually living that reality wasn’t scary at all.

Five years ago I would have looked at someone with two kids under two and thought “Wow that looks so insanely stressful, so hard, how do they do that?”

And now I have two kids under two, and sure, it is stressful and hard, but it’s also normal now. You get used to it. And just like you rarely go from aspiring entrepreneur to having twenty employees with mortgages overnight, you rarely go from thinking about having kids to having two overnight.

Big changes take time. And during that time, you adjust.

I say all this because some people shy away from going after big goals or life changes because they see people who are in those states and get scared.

You’ve probably heard the “hedonic adaptation” version of this, where you quickly get used to having a fancy car or nice clothes, and they stop making you feel good anymore.

But hedonic adaptation also works for things that seem hard or stressful.

You get used to worrying about your kids, or needing to run payroll, or getting a few emails each week from people who think you’re an idiot who can’t write.

Don’t avoid going after big goals because the life that comes with it scares you.

Start the business. Publish the articles. Have the kids. Yes, it will be hard, stressful, and a big adjustment, and you can’t imagine what life will look like once you have all those new obligations and worries and fears on you. But you’ll get used to them.

You can handle much more than you think.

[Sudeep🚢 @SudeepTek]

@mcgillmd921 How about adding a stoic quote on the wristband?

[Dickie Bush @dickiebush]

My framework for communicating ideas:

1. Start with an analogy
2. Give a "compare & contrast"
3. Finish with a personal example

It's like looking at a diamond.

Sure, you could look at it straight on.

But you'll see its true beauty from multiple angles.

Just like this tweet.

[Sudeep🚢 @SudeepTek]

Everyone's time is limited. Make conscious decisions and do not be afraid to.....live a little.