The site is now back up and we are working on the underlying problem. Apologies to any who were in the middle of uploading earlier - if you have any half-done submissions they can be completed via Edit Files and then Edit Details to publish.
Well, that was a surprise ending to the evening for our system administrators! In short, the database disk filled up, very quickly. This seems to be due to a replication failure - there is no data loss, but if the disk is full the server can't generate submission lists etc.
What we are doing now is moving the backlog of replication data to another disk. This should allow us to restore service while we debug the problem, which is likely to be a conflict of data with the database replica we use to take backups and determine recommendations on +fav.
The intervention has not started yet as there is a queue at our provider, so the maintenance window has been extended. The actual period of downtime should not be. Posting is still possible, but we suggest avoiding large PNG uploads that may be interrupted.
Inkbunny will be undergoing maintenance beginning at 12:00 GMT on Friday 29 to release a significant software update, including enhancements to privacy, comment display, spam handling, and performance.
Expected downtime is three hours. Any further updates will be posted here.🐇
We've updated the code and finished our database changes. Staff are now testing the new features. Please bear with us while we check that we didn't break anything unexpected... 🎶🧀🎶
A few members may have seen blank journal or PM lists earlier today - this was due to an error introduced by a security update to our database software triggered by non-English characters, that is now resolved. 🐘
Database geeks: see https://t.co/2kzlfWcfwG for the gory details.
We will rebuild the replica and restore recommendations for newer works after the weekend, when we have the time and people aren't quite so active on the site. Of course, IB never sleeps - but midday UTC is the slowest, so we schedule most of our work then.
As you may have noticed, we are having some unplanned downtime. 😼
We've identified at least part of the cause and work is ongoing to bring back your regularly scheduled bunny experience. Please bear with us! 🐇
Since we were busy last night: this is essentially what ended up happening. In fact, the replica slot dropped itself, but not fast enough to avoid the issue.
On the plus side we now have the latest version of our Linux distribution *and* our database software on the main server.
@jeffmarks1996 Hopefully nothing noticeable except possibly slightly faster performance - but we will see. Updates are necessary sooner or later as software falls out of support.