@TheRyex We have remote backups but using backups means losing all data between the time of the backup and the time of the problem. I want to avoid that.
A forced reboot in the physical hardware has caused an abrupt OS shutdown.
The abrupt OS shutdown has caused corruption which we identified as limited to two key database tables. They are now unreadable and potentially unrecoverable by our database.
(sorry. No ETA)
5 ways were tried to do a full data recovery but none worked.
I'm now doing an almost full data recovery. Some data will be lost but I don't know how much. However, I know it's not more than 24h and I'm expecting just some minutes of data loss. We'll see how it goes.
However, databases have multiple tricks under their sleeve. But making use of them requires meticulous manual action to make sure the final state is coherent.
Worst case, we recover from last night's backup which is now safely and redundantly stored.
Wish me luck.
Hey guys. Good news. The last part went well. I mean, went *really* well.
For the ones who figured out, the website is back up and has been for around 10 minutes. Hopefully it stays.
As usual, scream and do noise if it goes back down. Hopefully I can rest now.
Almost done.
However, BakaBT systems are being unable to communicate with each other right now.
We can start them but they can't reach each other.
After a break to cool down, we'll tackle this again to make it work.
ETA: Monday (but can be earlier)
Just a friendly reminder that all BakaBT staff is doing voluntary unpaid work. All income we get is used to keep the server alive and thriving.
If things look like they are going slow, is because they are going slow and done whenever possible.
@ArminMakovec Your account would take 1 month to get pruned after your warning. We will also turn off pruning (but not the pruning warning) for a few days after the server is up.