Identified: Fire is currently experiencing elevated latency & error rates due to issues with Discord's API. Discord has posted about the issue on their own status page, https://t.co/WPBDxM9dVV https://t.co/mGw4kd4j4t
Monitoring: Discord has implemented a fix and both error rates & latency have returned closer to expected levels. I will continue monitoring for some time to ensure things are fully resolved https://t.co/mGw4kd4j4t
Investigating: It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost
It's un https://t.co/pHmMarPHuF
Monitoring: Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning https://t.co/pHmMarPHuF
Completed: Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days https://t.co/oooquhIvcS
Planned: During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like e https://t.co/oooquhIvcS
In progress: Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventu https://t.co/oooquhIvcS
Completed: Maintenance has completed successfully.... almost.
During the upgrades, Fire's postgres installation got completely removed along with all the data. Fortunately, I have backups, but only for the main bot. Fire Beta ha https://t.co/pAf6bSfY8B
Planned: During this time, several upgrades (both hardware & software) will be performed on Fire's VPS which will result in some lengthy downtime of ALL SERVICES.
This downtime is unfortunate but necessary to ensure stability, s https://t.co/pAf6bSfY8B