This issue is now intermittent so may continue to impact fulfilment for the next few hours however our queues are now successfully draining. We're continuing to monitor closely.
We're experiencing an extended fulfilment outage this morning due to an issue with an upstream transit vendor. This is currently out of our direct control but we're monitoring closely. The API remains in service.
We're aware of some markup changes affecting around 8% of Google fulfilment today. Our team are working on the necessary extraction updates which will be deployed within the next 6-12 hours.
We have ongoing fulfilment delays on all queues this morning due to the extended AWS outage affecting services within us-east-1. We're closely monitoring the situation.
Compose managed to restore access around 00:30 UTC and we've been monitoring since this time. We've had a few periods of higher than normal latency but overall things appear to now be stable.
Unfortunately IBM Compose have still not resolved the issue within their AWS US-East-1 cluster and one of our core DBs remains inaccessible. You can subscribe to the incident directly for immediate updates by visiting: https://t.co/X8GYkymx94
From Compose: "We are continuing to work on a fix for this issue. Progress is being made. We'll post more updates as we have them. Thanks for your patience."
From Compose: "We've been able to stabilise internal network traffic and are now looking at gradually bringing things back back online." Their ETA on full cluster recovery is "hours" but we hope this is a worst-case estimate.
Our database host (IBM Compose) have confirmed a wide-spread outage within their AWS cluster and are working to bring all services back online. Unfortunately we're reliant on their engineers at this stage but will post updates as we receive them.