Evren Önem

Builds and operates billion-QPS distributed systems. Senior Director @MongoDB. Ex AWS DynamoDB. Core infra, reliability and eng management. Views are my own.

[Evren Önem @eonem]

@abilgegunduz @ebrulithm Yapılan iş detaylarını sürekli kayda almak, belli terfi hedefi koyup müdürünüze "bana şu tarihte terfi alabilmem için destek ol" mesajını vermek, geribildirim sağlayabilecek olanlardan periyodik geribildirim isteyip adreslemek, vs gibi.

[Evren Önem @eonem]

@abilgegunduz @ebrulithm Burda sizin bilmediğiniz, terfiyi etkileyecek başka detaylar olabilir. Fakat önemli olan, Bilge'nin de dediği gibi, hayatın adil olmamasına enerji harcamayıp, terfi için şansınızı arttıracak ve sizin kontrolünüzde olan şeylere odaklanmak.

[Evren Önem @eonem]

@GergelyOrosz Whereas you can very likely survive through a lot of mistakes if your software is used for running surveys for a thousand customers, or for running influencer campaigns for 10 companies.

[Evren Önem @eonem]

@GergelyOrosz The level of tolerance to such mistakes heavily depends on how easily your software can erode customer trust. If you're running core infra for millions of customers and experience repeated reliability issues, your business can easily be toast.

[Evren Önem @eonem]

@GergelyOrosz It's also now super easy to fake this (vs before), so I don't think it produces a useful hiring signal at all.

[Evren Önem @eonem]

“The best executives are very tough.”

Being firm but polite while keeping people accountable is absolutely possible. It's the hard way though. Easier way is to act tough & irritated to create a culture of fear. When someone is praising "being tough", they're praising being lazy.

[David Senra @davidsenra]

Revolut CEO Nik Storonsky says the best employees are like “self-guided missiles.”

“You don’t even need to tell them what the goal is. They self-identify the goal, and then they reach it.”

“Average people, you tell them the goal, and then you need to spend weekly iterations with them. So you spend management time. That’s what I don’t want.”

“The best executives are very tough.”

“Likable is never the goal for executives. The goal is always to achieve the goal with the minimal number of resources spent.”

[Evren Önem @eonem]

New Executive interview question that makes Big Tech CEOs tremble dropped.

[Keith Edwards @keithedwards]

this is so bleak

[Evren Önem @eonem]

@fatihguzeldev Sen yukarda benden daha güzel anlatmışsin Fatih kardeşim

[Evren Önem @eonem]

@fatihguzeldev ♥️

[Evren Önem @eonem]

A lesson on the value of redundancy for improved uptime: Either your bank should have a multi-zone deployment, or you should have two banks.

[Evan Luthra @EvanLuthra]

🚨 Google lost power in London this morning and Revolut, a bank with 80 million customers, went down with it.

I've spent the morning reading what Revolut users went through. It's wild.

People stranded at petrol stations, unable to pay for fuel or a taxi to the airport.

One traveller couldn't get out of a train station in Japan.

A 71-year-old woman stood at a supermarket till for an hour. Her kid had to drive 30km to bring her cash.

Card payments, transfers, ATM withdrawals, currency exchange, share trading. All down.

Revolut's statement said "all our customers' money and data are safe."

Safe. You just can't touch it.

Google's explanation so far is a "power event" in one zone of its London cloud region. Its advice to affected customers was to move their workloads to another zone or region "if feasible."

Must be nice. Revolut customers couldn't even move £10.

In July the UK government put Google Cloud, AWS, Microsoft and Oracle under direct Bank of England oversight, because one outage at any of them could shake the entire financial system.

12 days ago the Bank of England ran a drill with the UK's 38 biggest banks and market operators. The scenario they practised was a global cloud outage.

And exactly one year ago today, AWS went down and took Lloyds and Halifax with it.

Same date. Different cloud. Lloyds was having problems again this morning.

Revolut spent five years earning a UK banking licence.

Bitcoin has been running for almost 18 years. It doesn't have a London region.

[Evren Önem @eonem]

@fatihguzeldev Database'leri sevmemin en büyük nedeni, bir yerden sonra essential complexity'nin öyle bir noktaya gelmesi ki, çok iyi bir muhendisin bile olan bir şeyi geriye doğru anlamaya çalıştığında çok zorlanıyor olması.

[Evren Önem @eonem]

@abilgegunduz I always thought PhD studies would be so much more fruitful a bit later in life, after having explored a variety of topics professionally and developed an organic curiosity to go deeper (or, deepest than everyone else) in one of those. Congrats!

[Evren Önem @eonem]

A special-purpose database optimized for a specific workload can indeed do better than a general multi-purpose one. The hard part is to operate this database with a similar reliability posture.

Should review the cost of operating this and the uptime figures in a year's time.

[Aravind Srinivas @AravSrinivas]

We built a replacement for AWS DynamoDB, a key-value database for fast web content fetches. This was done with two engineers and hundreds of persistent Computer agents over two months. Migrating to our in-house database will save us up to a hundred million dollars yearly.

[Evren Önem @eonem]

@alioksuzdev Hiçbir zaman kaldırma bence. Kaldırmanın da başka dezavantajları olabilir. Senin durum zaten açsa açsa pozitif önyargıya yol açar.

[Evren Önem @eonem]

@VirSanghavi13 Two issues: 1/ You can't really mimic the prod load cause it's variable and the "bad combination" may only happen probabilistically (so it may not during testing). 2/ For many companies such full fledge infra per PR would be too costly. Impressive for a 16 yo though.

[Evren Önem @eonem]

@edleonklinger And who runs the Cloud?

[Evren Önem @eonem]

@oguzbor @MongoDB Fakat veriler çok büyüyünce de relational database'ler daha kolay sıkıntı yasiyor. Bu yüzden erişim pattern'lerini iyi anlayarak, verinin ne kadar büyüyeceğine dair iyi bir fikir sahibi olarak, latency toleransımızın miktarını önceden bilerek, doğru database'i seçmek bize düşüyor

[Evren Önem @eonem]

Today I started as a Senior Director of Engineering at @MongoDB. I'll own Observability Engineering. Ping me if you have pain points and/or ideas!

[Evren Önem @eonem]

@oguzbor @MongoDB NoSQL database'leri kullanmak relational'a göre biraz daha farklı bir yaklaşım gerektiriyor. Her tablo için erişim pattern'lerini iyi öngöremezsek sonrasinda kolayca darboğazlar oluşuyor. Relational'larda bu kısım daha esnek, bu yüzden daha çok tercih ediliyorlar. +++

[Evren Önem @eonem]

Full incident write up: https://t.co/XniTk91udb

[Evren Önem @eonem]

Highlights from the Aug 25 Buildkite incident:
- Deployment caused an EKS pod to scale up.
- Infra couldn't scale up in time. More scale up requests piled on. New pods put extra pressure on internal DNS.
- Discovery slowed down by 780X and went down with a huge blast radius.