Vlad F

CTO @github

[Vlad F @Vlad_GitHub]

@nk_developer1 Glad it is working well for you. Most of my day is spent on places where something didn’t work, so always nice to hear about positive experiences.

[Vlad F @Vlad_GitHub]

@sepisoad Not close :)

[Vlad F @Vlad_GitHub]

On August 17, GitHub experienced a significant outage that disrupted developers and organizations around the world. If you were trying to ship software that day, we let you down.

I posted in March and April about the steps we’re taking to make GitHub more reliable. The work is still underway. Here is an update:

https://t.co/9KaQy7O4CG

[Vlad F @Vlad_GitHub]

@matchaman11 We definitely don’t suspend for number of commits. The are a number of account takeover attacks and if we see account in early stages of exploitation we block it. DM me the details

[Vlad F @Vlad_GitHub]

@fxd0h00 @fxd0h Can you DM your details and I will help with support. It sounds like account takeover block

[Vlad F @Vlad_GitHub]

@sawlani_coder The circuit breaker flipped in the Copilot Token Service but the challenge was how to bring all the clients back into healthy state. This required returning a different error code for traffic to a site and then gradually opening up to keep circuit breaker from flipping again.

[Vlad F @Vlad_GitHub]

It is a problem for very large monorepos (ie single shard). Scaling static content is a solved problem (this is how all of us get to watch Netflix). Scaling reads of a repo which absorbs significant write volume and provides read after write guarantees is a harder problem as write volume increases. That said we are on track to roll out linear scaling:

[Vlad F @Vlad_GitHub]

@paulhudsonme A number of assumptions here are not true, but the most critical one is that growth is not coming from paid users. The load from paid users is both growing faster and is much larger share of total load.

[Vlad F @Vlad_GitHub]

@filipe_eh @fxd0h00 Let me get them opened. Generally that will go through @kdaigle and his team but happy to connect

[Vlad F @Vlad_GitHub]

@TheSurenk Thank you

[Vlad F @Vlad_GitHub]

@CantelopePeel Hi (maybe Will or Liam)

I am going to assume good intent behind your message and help with your support issue. I know how frustrating that can be - please DM me with details.

[Vlad F @Vlad_GitHub]

@codeshaunted We use a GitHub Enterprise Server mirror and the deploy process is tested every Sunday. When we needed it we found some knowledge and documentation gaps (since only the folks who tested it used it regularly). But it worked great once used correctly and that was key

[Vlad F @Vlad_GitHub]

@Robby_Seventeen You are right on the conclusion. The sidecar CPU metric was actually collected and hooked up the alarm. However, alarm wasn’t at the right priority and the signal wasn’t hooked up to the autoscaling.

[Vlad F @Vlad_GitHub]

@shahin43 @X I think everyone at GitHub believes into importance of open source (many joined from various projects). One thing we are trying to do is fit in more features for maintainers despite all the scaling work.

[Vlad F @Vlad_GitHub]

@AlexSieroshtan We have a series of technical blogs about the migration and the rebuilding coming. Let me see if it fits somewhere there.

[Vlad F @Vlad_GitHub]

2. Why and how was it done now ?
The why is answered in the sentences above the one posted - the load outpaced the power available in custom data centers. The how is a lot of really hard work by the team. It was necessary to sequence to the work to get and break it apart. We are doing a blog on that soon.

[Vlad F @Vlad_GitHub]

@mehulmpt There are two parts to your question:

1. Why wasn’t it done before ?

GitHub load prior to agentic age grew with number of developers and new features released. So its own custom data centers were sufficient and the migration was always going to be technically challenging.

[Vlad F @Vlad_GitHub]

The status page is accurate - 7 hours 47 minutes is maximum duration. Different services were affected for different periods of time. So if you scroll over Copilot you will see the amount of time it was affected. The incident link is in the post but here it is as well - https://t.co/fQL8nTQU1z

[Vlad F @Vlad_GitHub]

@ryannadel @alysson_rx This is the incident - https://t.co/S9kEW7nfQu

[Vlad F @Vlad_GitHub]

@chrisschouk We are actively rebuilding systems as we go. You are right in that systems designed for human rate of change need be re-architected. Below is one of the examples: