Daniel Schierbeck

Senior Principal Engineer at Zendesk

@[email protected]

[Daniel Schierbeck @dasch]

@ditheringfm I’m getting a 403 error when downloading today’s episode, tried deleting and redownloading

[Daniel Schierbeck @dasch]

Anyone in my network with *production* experience with #Istio circuit breakers?

[Daniel Schierbeck @dasch]

I am also @dasch on that other site that I am not allowed to refer to!

[Jason Smale @jwswj]

We've focused more on reliability through the last couple of years than ever before at @zendesk. I've just shared the 8 principles that have helped guide us internally

https://t.co/WTqwW0R8tA

[Daniel Schierbeck @dasch]

@ciaranarcher @rails @heroku 👹

[Daniel Schierbeck @dasch]

@benthompson assume you’ve seen this already? Interested in hearing your take. https://t.co/PDza4FwBMS

[Daniel Schierbeck @dasch]

@eileencodes @github Congrats!

[Daniel Schierbeck @dasch]

My colleague Luke did a great writeup comparing Apache Pulsar and Kafka: https://t.co/FbdFps3XsV #apachekafka #apachepulsar

[Daniel Schierbeck @dasch]

@glenathan @beanieboi @yann_ck @t_crayford @nicolefv In my experience, going to smaller but more numerous services comes with its own problems. specifically, figuring out whether stuff works correctly gets an order of magnitude more difficult. Deployment complexity is a function of organization size and amount of functionality.

[Daniel Schierbeck @dasch]

@marcdel @beanieboi @yann_ck @t_crayford I agree, and we use this heavily. Some stuff can’t use feature flags though, like library upgrades. Inherently risky.

[Daniel Schierbeck @dasch]

@glenathan @beanieboi @yann_ck @t_crayford That’s quite a blanket statement 😉 with hundreds of thousands of customers, the opportunity cost is pretty big. And there’s stuff that just takes time. We have ten or so production environments spread over the globe.

[Daniel Schierbeck @dasch]

@glenathan @beanieboi @yann_ck @t_crayford That’s great in theory, but not always achievable or the best use of resources for large orgs with older code bases.

[Daniel Schierbeck @dasch]

@beanieboi @yann_ck @t_crayford What happens when a bad change gets deployed to production but other changes get enqueued before that’s detected? Dev env doesn’t catch everything.

[Daniel Schierbeck @dasch]

@beanieboi @yann_ck @t_crayford … the “enqueueing” of changes to deploy. Merging to e.g. `production` is basically just enqueuing a deploy+merge to master – and you need automated rebasing to get rid of tainted deploys from the queue.

[Daniel Schierbeck @dasch]

@beanieboi @yann_ck @t_crayford I think most orgs use deploy from master, which they also use for development. Also: do you have a separate merge from production to master? That _would_ solve most of the problem, but it’s also just what I’m describing – managing the merges to the dev branch separately from…

[Daniel Schierbeck @dasch]

@beanieboi @yann_ck @t_crayford But even then, you now have all the merges in between plus the fix as a single deployable unit - you can’t rebase, because the main branch is also used for dev. With 40+ devs working on a sprawling and old code base, this happens a lot.

[Daniel Schierbeck @dasch]

@t_crayford if that becomes 30-45 minutes then the whole thing changes…

[Daniel Schierbeck @dasch]

@t_crayford How long does a single deploy take - including soak time before the next deploy?