Blitz, guild and archwar down

General announcements for the community to read.
Post Reply
User avatar
Laanders
Head Designer, Dev etc
Posts: 8460
31Rank Halls 3Guild Halls
Location: Malmö, Sweden

Blitz, guild and archwar down

Post by Laanders » Apr 09th 2013 20:00

Just as a heads up for those wondering: the server hosting blitz, guild and archwar started experiencing some rather erratic behaviour at around 21:13 game time. As a result those three game servers are unavailable until further notice. We are working to determine the problem and solve it as soon as possible. Further updates will be posted as we learn more. Thanks for understanding.
Head Designer
Developer

Former Head Admin
IRC Team

User avatar
Laanders
Head Designer, Dev etc
Posts: 8460
31Rank Halls 3Guild Halls
Location: Malmö, Sweden

Re: Blitz, guild and archwar down

Post by Laanders » Apr 09th 2013 22:51

To give an update, however vague... We have so far been unable to fix or even accurately determine the cause of the problem. Best guess is bad hardware, not unlikely the storage, but the server is in such a state that it is very difficult to troubleshoot; most commands simply fail with various unusual errors. The way it looks, we'll need the hosting staff at the data centre where the server is located to physically access it, and only then can we really hope for any progress. Unfortunately, in good accordance with Murphy's Law, the server chose to act up in the late evening at the local time of the data centre, and we cannot expect much to happen until morning, European time. This could mean some time after 08:00 game time, but due to the degree of uncertainty in the situation, it could also take longer.

Since we're already looking at a longer downtime, we will need to consider an appropriate way to bring the game servers back online once the technical issues have been dealt with. This could for example mean that the return of the servers could be intentionally delayed further in order to give more people a better chance to be online when they do. A rollback is less likely to be relevant, since the server went down at the same time for everyone and has consequently remained unreachable ever since. Nevertheless, we will post details about this as well once the situation has been further addressed.
Head Designer
Developer

Former Head Admin
IRC Team

User avatar
Laanders
Head Designer, Dev etc
Posts: 8460
31Rank Halls 3Guild Halls
Location: Malmö, Sweden

Re: Blitz, guild and archwar down

Post by Laanders » Apr 10th 2013 15:46

We have concluded that the problem with the physical server running blitz, guild and archwar stems from fatal disk failure. Most of the day has been spent trying to recover data from the bad drive, with so far moderate success. Unfortunately we uncovered at the same time that our redundancy system has been inoperative for some time, which means that the worst case - and increasingly likely - scenario is complete data loss for the sets in progress. We have one or two more cards to play before we resign to that conclusion, which will be explored as quickly as time permits. Meanwhile we have already started work on getting the game servers back to a working state, and hope to be able to announce a return date in the near future, be it with continuation of the sets in progress, or with a clean slate for everyone.

We regret the inconvenience, but wish to remind our players that arch, beta and lightning servers are all operating normally. Additionally, solo server is scheduled to start a new set this Friday.
Last edited by Laanders on Apr 10th 2013 15:48, edited 1 time in total.
Reason: Removed archwar from list of available servers.
Head Designer
Developer

Former Head Admin
IRC Team

User avatar
Laanders
Head Designer, Dev etc
Posts: 8460
31Rank Halls 3Guild Halls
Location: Malmö, Sweden

Re: Blitz, guild and archwar down

Post by Laanders » Apr 11th 2013 07:06

Unfortunately we have been ultimately unable to recover any data relevant to the sets that were in progress when the server went down Tuesday evening game time. We therefore have no way of continuing those sets, meaning once the game servers are back up they will each start a new set. We are sorry.
Head Designer
Developer

Former Head Admin
IRC Team

User avatar
Hypello
Head Developer
Posts: 3503
1Rank Halls 2Guild Halls 1Arma Halls
Location: Australia

Re: Blitz, guild and archwar down

Post by Hypello » Apr 13th 2013 10:13

Just updating what progress has been made.

All game servers have been rebuilt and will be ready for operation in the next few days. We apologize for the delay.
Image

User avatar
Laanders
Head Designer, Dev etc
Posts: 8460
31Rank Halls 3Guild Halls
Location: Malmö, Sweden

Re: Blitz, guild and archwar down

Post by Laanders » Apr 14th 2013 03:39

Blitz, guild and archwar have now all been fully configured and given starting dates for their new sets, all within the next couple of days (Monday-Tuesday). In addition, solo server has been migrated to the same hardware as the other three and will also begin a new set on Tuesday.

As always when things have been configured from scratch and otherwise extensively rearranged, there is a somewhat elevated risk of bugs. Please report observations of such in the General Bugs forum, and we will do our best to fix them as quickly as possible.
Head Designer
Developer

Former Head Admin
IRC Team

Post Reply