Won the 3rd prize in an internal hackathon at office :)
Check out my latest side gig product and let me know your thoughts
Full demo video available at Youtube :
https://t.co/5gZrkIZg4t
Lost $440M in 45 mins
The hidden cost of hiring
tier 3 software engineering team
and why HFT firms avoid it?
It took Walter R
17 years to build
a top HFT firm
in Wall Street
but in August 2012
it was brought down
to its knees
by a System Administrator
The story...
To understand the story better
remember these key components
of the HFT System
1. OMS- receives and manages trade orders
2. SMARS - the algorithms engine
3. EV - execution venues like NSE, BSE, NYSE
Flow of execution:
1. Trader places order to OMS
2. OMS forwards to SMARS
3. SMARS does the calculation and strategy
4. EV processes the request of long or short
Simple..
Back to the story,
The day before incident Joe ( Sysadmin )
was tasked to deploy
a new piece of code (RLP)
to SMARS
SMARS a distributed system
with 7 servers
Joe deployed RLP
on first 6 servers
but missed the 7th
The legacy code
had a feature flag
that toggles
a system called
Power Peg
Power Peg dissects
a large order to smaller
orders and takes position
in seconds
interestingly,
PP was deprecated
6 years ago by the firm
but as of today
some genius decided
RLP will use
the same feature flag
used by Power Peg
Moving on..
Night before incident,
Joe was alone in office
Joe tests few servers
they worked as expected
Joe activates feature flag
for all servers
and logs off
Next morning
the market opens,
First 6 SMARS server
worked as expected
the 7th server
receives ~181 orders
and boom π₯
Within 45 mins
Power Peg places
long position worth $3.5B
and short position of $1.3B
All hell breaks loose
Knight capital will now have to clear
$3.5B in payments in 3 days
which they don't have..
Thomas Joyce the then CEO
went crazy and a spine chilling
incident grasps the entire
Wall Street
-------
So what went wrong?
Joe ( Sys Admin ) was the tail
of the problem
a mere executionist
The root of the problem
was somewhere else
Big problems:
1. Deprecated code still lies in active prod code
2. No automated deployment process
3. No second technical reviewer for Joe
4. No regression testing suites
We often hear "If it works don't touch it"
This is the worst advice for software engineers
Every action that the software does
should be:
1. Tested
2. Documented
3. Reviewed
Else we end up in the Knight Capital fiasco
Goldman Sachs
had to settle the dues
of $440M
Knight Capital was eventually
sold within 2 years
Bad software engineering practices
and failed software architects
brought down a good company in a flash
---
Key takeaways:
1. Software Engineering is an orchestra of DevOps, Development, QA, Processes etc.
2. Processes and Systems makes software predictable and less erroneous.
3. Great coding standards is a culture that leadership has to imbibe from day 0
-
Enjoyed the content?
Like Comment Retweet to educate others
I keep writing stories on Software Architecture, Career and Life
β‘οΈ Follow @sagar_codes for more
If you like technical content,
you'd love my weekly letters
Subscribe to my Substack today β€οΈ
(Link in bio)
Spread the love for open source with #Hacktoberfest, a month-long celebration of open-source projects, their maintainers, and the entire community of contributors. https://t.co/uCyIIFY6hP
Any company would be ultra lucky to have an asset like @jasleen101010
Please feel free to reach out to her if there are any openings in your company for data based roles!
set the scene: henry is 35. ted is old and grey.
"do you ever regret it, dad? coming back?"
"do i regret coming back to see you grow up and become this brilliant man? hell no. do i ever think about what couldve been if i stayed? every day, son. every day."
I'm excited to share that my winning project is now available at :Β
https://t.co/FLbph3hAI8.
You can find more information about my submission in the readme. Feel free to explore and discover the details of my solution!
Thrilled to announce that I emerged as the grand winner of the Bengaluru Open Mobility Challenge '23! π₯β¨
I had an amazing time participating in this extraordinary event hosted by @nammayatri and @ONDC_Official.
Thank you, Team @nammayatri and @ONDC_Official, for organising this remarkable hackathon. It has been a privilege to be part of this journey. Excited for more challenges ahead!