UK Server 9 news and announcements. http://forum.ezpzhosting.co.uk Sat, 19 Nov 2011 18:23:45 +0000 5 http://forum.ezpzhosting.co.uk/index.php/topic/780-srv9-network-issue/
We're in contact with the DC and onsite techs are working on the issue.

18:39: This appears to be a failed network card. Onsite techs are replacing this now.

19:36: The server is back online]]> Sat, 19 Nov 2011 18:23:45 +0000 http://forum.ezpzhosting.co.uk/index.php/topic/780-srv9-network-issue/ http://forum.ezpzhosting.co.uk/index.php/topic/709-filesystem-check/
Currently 19% complete

11:09: Now 30% complete

11:26: Now 57% complete

11:38: 79% complete

11:54: The automatic scan failed. We're running a manual check now which will take a little longer to run. We'll continue to post updates.

11:58: The manual fsck has completed in record time. The server is now booting. It may still need manual intervention to repair some services but we're getting closer.

12:19: Services are starting to come back up. We're just working on MySQL.
Note the RAID is initializing which will affect performance and may take a few hours.

12:51: MySQL databases are being repaired. This is around 50% complete.

14:10: The repair has finished but we're still seeing MySQL errors. We're still working on bringing any remaining sites up.

15:38: Databases are currently being restored from the last valid backup (23rd). This will take around 2 hours to complete (estimated).

17:25: The estimations R1Soft was giving us were way off (unsurprisingly). It looks like it may take around another 3 hours.

All sites should now be back online. We're testing a number of sites at random to make sure everything is working as it should.

MySQL may respond slowly for an hour while it rebuilds the cache.

A full RFO is to follow.

RFO:

At around 10am this morning our monitors alerted us that some services had started to fail. We immediately logged in to the server and found errors indicating a filesystem problem in the logs.

The server was booted into recovery mode and an initial fsck (filesystem check) was performed. This took just over an hour to complete but failed to fully repair the error so a manual check was run, taking 4 minutes, which did repair the remaining errors.

Once the server was booted it quickly became clear a number of databases were missing.

We initiated a restore of 23GB worth of databases using R1Soft. Unfortunately due to other issues with the backup server the last complete backup we had was from the 23rd.

The restore ran much slower than we expected and took 6 hours to complete.

We've had problems with R1Soft in the past, including slow restores and failed backups. We've now added a secondary backup system and it's currently performing the initial backup seed.

The future of R1Soft at EZPZ hasn't yet been determined and we're going to test alternative solutions to ensure data can quickly be restored if data is corrupt, accounts deleted or a server fails.]]> Mon, 26 Sep 2011 09:59:55 +0000 http://forum.ezpzhosting.co.uk/index.php/topic/709-filesystem-check/ http://forum.ezpzhosting.co.uk/index.php/topic/613-server-9-announcements/ We will be posting any issues relating to Server 9 in this forum as soon as they happen. Fri, 05 Aug 2011 13:39:43 +0000 http://forum.ezpzhosting.co.uk/index.php/topic/613-server-9-announcements/