Blitz, guild and archwar down
Blitz, guild and archwar down
Just as a heads up for those wondering: the server hosting blitz, guild and archwar started experiencing some rather erratic behaviour at around 21:13 game time. As a result those three game servers are unavailable until further notice. We are working to determine the problem and solve it as soon as possible. Further updates will be posted as we learn more. Thanks for understanding.
Head Designer
Developer
Former Head Admin
IRC Team
Developer
Former Head Admin
IRC Team
Re: Blitz, guild and archwar down
To give an update, however vague... We have so far been unable to fix or even accurately determine the cause of the problem. Best guess is bad hardware, not unlikely the storage, but the server is in such a state that it is very difficult to troubleshoot; most commands simply fail with various unusual errors. The way it looks, we'll need the hosting staff at the data centre where the server is located to physically access it, and only then can we really hope for any progress. Unfortunately, in good accordance with Murphy's Law, the server chose to act up in the late evening at the local time of the data centre, and we cannot expect much to happen until morning, European time. This could mean some time after 08:00 game time, but due to the degree of uncertainty in the situation, it could also take longer.
Since we're already looking at a longer downtime, we will need to consider an appropriate way to bring the game servers back online once the technical issues have been dealt with. This could for example mean that the return of the servers could be intentionally delayed further in order to give more people a better chance to be online when they do. A rollback is less likely to be relevant, since the server went down at the same time for everyone and has consequently remained unreachable ever since. Nevertheless, we will post details about this as well once the situation has been further addressed.
Since we're already looking at a longer downtime, we will need to consider an appropriate way to bring the game servers back online once the technical issues have been dealt with. This could for example mean that the return of the servers could be intentionally delayed further in order to give more people a better chance to be online when they do. A rollback is less likely to be relevant, since the server went down at the same time for everyone and has consequently remained unreachable ever since. Nevertheless, we will post details about this as well once the situation has been further addressed.
Head Designer
Developer
Former Head Admin
IRC Team
Developer
Former Head Admin
IRC Team
Re: Blitz, guild and archwar down
We have concluded that the problem with the physical server running blitz, guild and archwar stems from fatal disk failure. Most of the day has been spent trying to recover data from the bad drive, with so far moderate success. Unfortunately we uncovered at the same time that our redundancy system has been inoperative for some time, which means that the worst case - and increasingly likely - scenario is complete data loss for the sets in progress. We have one or two more cards to play before we resign to that conclusion, which will be explored as quickly as time permits. Meanwhile we have already started work on getting the game servers back to a working state, and hope to be able to announce a return date in the near future, be it with continuation of the sets in progress, or with a clean slate for everyone.
We regret the inconvenience, but wish to remind our players that arch, beta and lightning servers are all operating normally. Additionally, solo server is scheduled to start a new set this Friday.
We regret the inconvenience, but wish to remind our players that arch, beta and lightning servers are all operating normally. Additionally, solo server is scheduled to start a new set this Friday.
Last edited by Laanders on Apr 10th 2013 15:48, edited 1 time in total.
Reason: Removed archwar from list of available servers.
Reason: Removed archwar from list of available servers.
Head Designer
Developer
Former Head Admin
IRC Team
Developer
Former Head Admin
IRC Team
Re: Blitz, guild and archwar down
Unfortunately we have been ultimately unable to recover any data relevant to the sets that were in progress when the server went down Tuesday evening game time. We therefore have no way of continuing those sets, meaning once the game servers are back up they will each start a new set. We are sorry.
Head Designer
Developer
Former Head Admin
IRC Team
Developer
Former Head Admin
IRC Team
Re: Blitz, guild and archwar down
Just updating what progress has been made.
All game servers have been rebuilt and will be ready for operation in the next few days. We apologize for the delay.
All game servers have been rebuilt and will be ready for operation in the next few days. We apologize for the delay.
Re: Blitz, guild and archwar down
Blitz, guild and archwar have now all been fully configured and given starting dates for their new sets, all within the next couple of days (Monday-Tuesday). In addition, solo server has been migrated to the same hardware as the other three and will also begin a new set on Tuesday.
As always when things have been configured from scratch and otherwise extensively rearranged, there is a somewhat elevated risk of bugs. Please report observations of such in the General Bugs forum, and we will do our best to fix them as quickly as possible.
As always when things have been configured from scratch and otherwise extensively rearranged, there is a somewhat elevated risk of bugs. Please report observations of such in the General Bugs forum, and we will do our best to fix them as quickly as possible.
Head Designer
Developer
Former Head Admin
IRC Team
Developer
Former Head Admin
IRC Team