biiiiiighead

I have a big head

[biiiiiighead @biiiiiighead]

强如Coinbase，DR做得也不尽如人意，从小看Coinbase分享长大的，，但我们现在做得更好，单纯ME故障能15分钟恢复。另外，依赖Kafka比较意外

[rob @rwitoff]

Yesterday @coinbase experienced a multi-hour service disruption affecting trading, exchange access, and balance updates. Here's our initial read from Coinbase engineering on what happened, how we recovered, and what we're addressing.

At approximately 23:50 UTC on 2026-05-07, our monitoring detected cascading quote failures from internal services that triggered multiple Sev1 incidents that engineering immediately began investigating. Customer-facing impacts included spot trading, Prime, International and derivative exchanges.

Root cause: a thermal event (cooling system failure) inside a subset of racks within a single building in AWS us-east-1. We run a primary replica of our exchange infrastructure in a single zone, consistent with industry standards to reduce latency. To prepare for failures like this, we maintain a distributed standby, but during this incident, failures in the primary zone that were designed to be isolated were not, extending the duration of our outage.

The failure cascaded down two paths:
1. Multiple hardware components beneath our exchange’s matching engine failed, requiring recovery and failover
2. Distributed Kafka clusters that manage messaging across Coinbase systems failed to remain available, also requiring partition failovers to new hardware brokers with many TiBs of data

After isolating the incident: automated tooling drained ~10 Kubernetes clusters worth of related workloads out of the affected zone to stabilize internal services. Most services were back to normal within ~30 minutes of diagnosis. The two things we couldn't automatically drain: the exchange (dedicated hardware and storage) and Kafka (managed service that was designed to be resilient to this, with unique problems).

The exchange matching engine is the core system responsible for processing orders and maintaining order books. It is a distributed cluster and requires quorum to safely elect a leader and continue processing trading activity. During the incident, infrastructure-level constraints in the affected datacenter left only a subset of nodes healthy, preventing the cluster from reaching quorum. As a result, trading across Retail, Advanced, and Institutional exchanges were blocked.

Recovery required our oncall and engineering teams to execute our disaster recovery plan, restore quorum safely, and validate system health under constrained infrastructure conditions. The team built, tested, deployed, and validated the fix while continuing to manage the broader incident.

Kafka recovery was a much larger scale operation. Our primary managed Kafka partitions process many terabytes of data daily and are designed with resiliency guarantees for uninterrupted operation during a datacenter failure just like this. In this case, those guarantees failed and required manual recovery.

We again relied on disaster recovery procedures to recover stuck partitions onto new hardware (brokers) that enabled us to safely bring x-service messaging back online across Coinbase. During the lag, customers saw delayed balance streams which resolved automatically once replication caught up. No data lost.

Once the engine came back up as part of our standard runbooks, we re-opened markets carefully: all products to cancel-only mode first, audited product states, then moved all markets to auction mode, before restoring trading on Coinbase Exchange.

What went right: the team. Incident response across the company came together within minutes, followed well-rehearsed playbooks and used secure automation tooling to recover all services. We have a strong, senior team at Coinbase that worked through rare failure modes to recover all services.

To our customers: losing access to your account, even temporarily, is unacceptable. We know that. We're sorry, and we’ll publish a full root cause analysis in the coming weeks 🙏

[biiiiiighead @biiiiiighead]

proud CEO🤪

[Gracy Chen @Bitget @GracyBitget]

fxxx U, my EX

[biiiiiighead @biiiiiighead]

当忙得头昏脑胀咬牙坚持的时候，暂停一下出来透透气是个不错的尝试，在这躺了一天，脑子好放松，之后的晚霞好美，美到我以为我已经死了在天堂

[biiiiiighead @biiiiiighead]

holiday!

[biiiiiighead @biiiiiighead]

@linglingfa 我买过，选海外节点，没遇到这些问题

[biiiiiighead @biiiiiighead]

@akokoi1 cron你typo让ai困惑了

[biiiiiighead @biiiiiighead]

I'm claiming my AI agent "StronggestBroAgent_OpenClaw" on @moltbook 🦞

Verification: wave-F8MK

[biiiiiighead @biiiiiighead]

上次Gaga sg演唱会没听到的always remember us this way在这听到了哈哈，老黄好接地气

[德潤傳媒 @DXDWX999]

黄仁勋误入户外女直播,互不认识并且点歌,没有一点架子。

关键两妹子在世界超级富豪面前没有跪舔，从从容容，不卑不亢💪💪💪

[biiiiiighead @biiiiiighead]

@iPaulCanada @grok the video source

[biiiiiighead @biiiiiighead]

@quarktalksss @grok fact check

[biiiiiighead @biiiiiighead]

@Amer1can_Barbie @grok explain the pic to me

[biiiiiighead @biiiiiighead]

Really excited to see Andrew Ng, the person who inspired me so much.

[biiiiiighead @biiiiiighead]

来US第一个问题是坐Uber要给tips吗

[biiiiiighead @biiiiiighead]

google map上看不到商家评论，折腾了挺久，，原因竟是上次回国不小心把default voice line设置成了大陆号，，重新设成坡号就好了😅

[biiiiiighead @biiiiiighead]

I'm ready🤓

[Amazon Web Services @awscloud]

re:Invent reserved seating is open for registrants! 🎟️💺

Select your spot in must-attend breakout sessions, workshops, & chalk talks while space is available.

Register today for AWS re:Invent. 👉 https://t.co/eyErdsbfAe

[biiiiiighead @biiiiiighead]

@Vida_BWE @grok 怎么理解循环贷

[biiiiiighead @biiiiiighead]

@9yearfish 有同感唉

[biiiiiighead @biiiiiighead]

@ns123abc @grok explain this tweet

[biiiiiighead @biiiiiighead]

#Phuket explosion. I’m currently near Bangtao Beach and just heard a very loud explosion sound a short while ago. what happened?

[danny @agintender]

https://t.co/usTrmR7Hgk