Power outage in our provider's data center


#1

Currently, our hosting provider is experiencing a power outage at one of their DCs…three of our servers and 6 game worlds are affected, as it seems. These are:

  1. Gatow
  2. Kaitak
  3. Tempelhof
  4. Fornebu
  5. Devau
  6. Otto

We will monitor the situation and let you know once the problem has been resolved.


#2

Status Update

Two game worlds are back online:

  • Tempelhof
  • Fornebu (with slight backlog)

The recovery of the other four game worlds will take a bit longer: Due to the immediate loss of power, our databases were corrupted:

  • It looks like for Otto, Gatow and Kaitak we were able to recover data dumps.
  • We will try and move Otto to a free slot on another server to see whether the dump is functional. If so, Otto will be back online shortly.
  • We plan to use the opportunity to move Gatow and Kaitak to new hardware in the process, but that still depends on whether the provider can deliver the required hardware quickly enough.
  • For Devau we were unable to recover an intact data dump and we're investigating other options at the moment.

Thank you very much for your patience!


#3

Moving Otto to our newest piece of hardware proved effective: Back online and zero backlog after just a few minutes.

Also, the dump we recovered looked good except for one minor data corruption that I was able to fix manually. We'll monitor it though and should you encounter any issues, please let us know.

We'll keep you posted about the status of Gatow, Kaitak and Devau.


#4

Quick action and clear communication.

Good job !

Jan


#5

Yes - thanks for this, Martin and Team!


#6

Good Job AS Team. Hope they get this resolved as i remember not long after meigs released the server had similar problem and the server was corrupted completely and lost my airline.

Hope Everyone Airlines Will Be Ok :).   


#7

Status Update

Since waiting for new hardware will most likely take too long, we are currently investigating an alternative option. Thanks once again for your patience. We hope to have all game worlds back online as soon as possible.


#8

Status Update

You might have noticed that Devau is back online. We experimented with a VM that was available on short notice, but so far we are unhappy with the performance. We will leave it running for ow until we've decided how to proceed with the remaining game worlds.

Note that we had to restore Devau from a backup we created last night, hence the long 12h backlog it is currently trying to cope with.


#9

Status Update

For now all game worlds are back online. Obviously, Kaitak, Gatow and Devau still are working off a considerable backlog. Note that demand distribution and the simulation update run in parallel, with the demand distribution typically falling behind due to its high resource requirements. That should resolve itself eventually, but you might see booking behavior that's unusual compared to normal operation.

A word on Devau: As said, the database of Devau was destroyed when the hosting server went down. We were able to restore it from a backup, but this backup was several hours old, so some data is missing. Also, the "relation statistics" data unfortunately is not part of our backups because it is many gigabytes in size. We apologize for the inconvenience caused by all this!

Last but not least: We still plan to move at least Kaitak, Gatow and Devau to new hardware soon. We will inform you about this move separately.