Alon Shrestha

cloud architect | distributed systems & scaling | building on aws | logging my tech stories

[Alon Shrestha @alonshrestha]

over the years, i've learned a lot from people sharing their experiences online.

today, i'm sharing mine.

from a confused computer science student to principal platform engineer, this is the story of how i got into cloud and where i'm headed next.

my journey into the cloud.

link in comments

[Alon Shrestha @alonshrestha]

@Incitelabz that's a good approach. all the best with your journey!

[Alon Shrestha @alonshrestha]

i started working with aws during my internship in 2020, and i've been with the same platform ever since.

over the years, i kept hearing that learning multiple cloud platforms is important for better career opportunities. there's some truth to that, but i chose to go deep into aws first.

i also used to underestimate aws certifications.

that changed in 2024 when i started preparing for one seriously. it helped me connect concepts i was already using at work and filled a lot of knowledge gaps.

today i have 5 aws certifications. after my first one, i got addicted to get more.. haha..

by the end of this year, i still have 2 more to go. hopefully, i can find the time to finish them. after that, i think i'll take a break from certifications for a while.

this photo is of the notes i made while preparing for my first aws certification in 2024. surprisingly, i'm still using those same notes today while preparing for my next one.

[Alon Shrestha @alonshrestha]

@bishow___pandey thank you! really appreciate it.

[Alon Shrestha @alonshrestha]

first client onboarded. will remember their name.

[Alon Shrestha @alonshrestha]

july 17. today marks one of the biggest milestones of my career.

i've been waiting for this day since last year, when i started this project.

today, we officially released version 2 of a product that has been serving our company for over a decade. it's the platform that securely transfers phi data files between our clients and our data analytics teams.

the old system served us well, but years of technical limitations made it difficult to add new features or improve the overall workflow.

over the past year, i rewrote and re-architected the platform using modern technologies, making it more scalable, secure, and maintainable. along the way, we introduced significant automation and parallelism into the file processing pipeline to make operations smoother, faster, and more reliable.

after countless discussions, design decisions, trade-offs, late nights, and a lot of hard work, the product is finally live today.

but this isn't the finish line.

the real challenge starts now.

next comes onboarding new clients and, even harder, migrating existing clients to the new platform. i already know what's ahead: technical issues, communication delays, unexpected edge cases, and plenty of lessons along the way.

and i'm ready for it.

i'll be sharing a detailed blog soon about the architecture.

[Alon Shrestha @alonshrestha]

@bishow___pandey 🍻

[Alon Shrestha @alonshrestha]

just tried recording myself explaining aws security groups being stateful and nacls being stateless in my native language, nepali.

भनेजस्तो सजिलो हुँदैन, guys.

it feels fun and cringe to watch and listen yourself.

but one thing is clear: i need to keep working on this.

feedback much appreciated.

[Alon Shrestha @alonshrestha]

we’re having a difficult time, and things aren’t going well for us.

[Alon Shrestha @alonshrestha]

hear my story. this might save you a lot of pain.

if you're not monitoring aws api calls, you're probably paying for bugs you don't know exist.

i discovered this by accident.

i was debugging an issue and searching through cloudtrail logs.

while checking kms access, i noticed something that made me stop.

an iam role from a recently released application had called the same kms key 50 times in 29 seconds.

that's about 1.7 calls per second.

over a month, that's roughly 4.5 million api calls.

the application wasn't anywhere close to that scale.

the surprising part was that nobody owned this problem.

the cloud team provisions resources and grants access. after that, applications can call aws apis as often as they want. there are usually no guardrails around usage patterns.

that day, we decided to monitor api calls too.

we first built some in-house automation using cloudtrail and athena. it worked, but collecting logs, running queries, and analyzing everything created operational overhead.

later, we started streaming cloudtrail events to cloudwatch logs, added metric filters, and configured alarms.

it costs a little more, but it has saved us far more.

we now use the same approach across multiple aws services, not just kms.

it's helped developers catch bad loops and retry logic, and unnecessary api calls before they became expensive.

when we spoke with the team, the root cause wasn't aws.

it was ai-generated code combined with weak system design knowledge.

[Alon Shrestha @alonshrestha]

@pushpikaa_ great minds think alike ✌️

[Alon Shrestha @alonshrestha]

july 17. today marks one of the biggest milestones of my career.

i've been waiting for this day since last year, when i started this project.

today, we officially released version 2 of a product that has been serving our company for over a decade. it's the platform that securely transfers phi data files between our clients and our data analytics teams.

the old system served us well, but years of technical limitations made it difficult to add new features or improve the overall workflow.

over the past year, i rewrote and re-architected the platform using modern technologies, making it more scalable, secure, and maintainable. along the way, we introduced significant automation and parallelism into the file processing pipeline to make operations smoother, faster, and more reliable.

after countless discussions, design decisions, trade-offs, late nights, and a lot of hard work, the product is finally live today.

but this isn't the finish line.

the real challenge starts now.

next comes onboarding new clients and, even harder, migrating existing clients to the new platform. i already know what's ahead: technical issues, communication delays, unexpected edge cases, and plenty of lessons along the way.

and i'm ready for it.

i'll be sharing a detailed blog soon about the architecture.

[Alon Shrestha @alonshrestha]

Recently involved in a new project focused on file transactions using SFTP.

The architecture is nearly complete, with an emphasis on high availability and fault tolerance.

#Cloud #AWS #architect

[Alon Shrestha @alonshrestha]

@pushpikaa_ thank you

[Alon Shrestha @alonshrestha]

companies like aws can still set a low operational bar.

yesterday we had a major outage because of a whitespace in an aws tag.

aws docs say that tag keys can't have whitespace when instance tags in instance metadata is enabled. but this was never enforced for us before.

our emr cluster config had been working for a long time. yesterday it suddenly started failing in some regions while the exact same config worked fine in others.

after talking to aws support, they told us this validation was enforced recently.

the weird part is i still can't find any announcement or release notes about this change. i only found the rule in the docs, and that page has been around for a long time.

because of this, we missed our daily emr processing window and couldn't deliver to our client on time.

changing the tag wasn't as easy as it sounds. that tag was used by multiple systems, so we had to find every dependency, update everything, and test it. it took us hours.

what made it even more frustrating was that one region which was failing earlier started working later without us changing anything.

so was the rollout incomplete?
did they roll it back?
or again the ai issue?

[Alon Shrestha @alonshrestha]

nepali tech companies that serve the public through their apps (banking, trading, entertainment, etc.) should build a culture of transparency.

i've seen some isps already follow this practice, and it's a step in the right direction.

just like many leading tech companies such as google, meta, and aws, they should publish a postmortem after every major system failure, explaining what happened, how customers were affected, and similar stuff.

customers deserve transparency because they're paying for the service. and next sharing postmortems builds trust, improves accountability, and ultimately leads to better products.

[Alon Shrestha @alonshrestha]

@bishow___pandey i wish your dream comes true.

[Alon Shrestha @alonshrestha]

if everyone can decrypt, encryption is just expensive encoding.

[Alon Shrestha @alonshrestha]

@webdevcody so? you mean code volume equal success. i'm more worried if you used ai to build this content.

[Alon Shrestha @alonshrestha]

just hopped off a great call with an aws ssm specialist about patch management.

our journey with ssm has been a big win. we are 100% compliant and rely nearly full on it to manage of our resources. while linux patching works beautifully and completely blew me away, windows has been a totally different story with painful limits.

we dug deep into those hurdles today. moving forward, we are teaming up with aws for sometime to rewrite our windows patching process, crush zero-day vulnerabilities, 3rd party application patching, tackle workspace patching, and build better automation.

[Alon Shrestha @alonshrestha]

wallets (esewa/khalti), banking apps, or ntc treat a major bug as a feature.

they must hear this a lot since i personally complained twice. people make mistakes doing topups instead of transfers, and it is non-reversible. accidentally recharging 5k leaves users stuck with money that is difficult to spend. topups must have a 1000rs threshold or below. the logic to fix this is simple from either side, but the system clearly looks like profit first.

[Alon Shrestha @alonshrestha]

this can be a lesson for many. never start the first day of the week with a fixed plan of what you will work on. instead come in with an open mind. you are not the only one bringing new ideas or priorities. your seniors and your team will too.

you might get into the flow of one project and then get assigned something else. switching context can be frustrating. this could be solved with better project management, but somehow we in the infra team have never really believed in project managers.