Note that we can only collect these detailed performance data from a handful of game worlds because we're measuring the execution time of _every_ writing transaction. So Devau is the blue line, high and rather erratic values. Red is Stapleton, with extremely erratic values up until around 10 pm last night. And yellow is Kaitak which kind of serves as the "baseline" here...that's how the performance should look.
So we're having a little problem: Stapleton was our most problematic game world performance-wise up until last night. Now the game world with the most problematic recorded values is Devau, so we're switching our focus to there.
We know from last week-end and from Stapleton's current performance that - in general - our setup should be able to handle the load we produce. We've done quite a bit of brainstorming about what might have caused the relapse on Monday and our best guess right now is that it has to do with our transport statistics. Or more precisely: Their deletion at the beginning of the week.
We are using Devau and its neighbor game world Gatow for trials into this direction now. As you've probably noticed, this has some unfortunate effects like downtimes and slow response times.
We are very sorry for this, but I feel we are very close to finding the issue once and for all. Please stick with us until we have figured it out! Thank you very much!
For the moment, both game worlds are back online. Both with tremendous backlog of course. We're continuing to fiddle with configuration parameters, so further disturbances can occur at any time.
Since yesterday's downtime took longer than planned, it also took Devau to pick up its backlog until some time very early this morning. As such we're only seeing comparable figures now and it looks to me like our changes done yesterday didn't do all that much. In any case they didn't solve the problem completely.
So the investigation continues. What's become apparent is that we still don't have enough data available about many aspects of our systems. We need to correlate this data though to figure out what's wrong. So I'll probably be spending even more time instrumenting our servers and software to get to the core of the issue.
As you might have noticed, all game worlds delivered good performance over the week-end again. And just like last week they slowed down around 2 o'clock at night today. So I am about 90% sure that is has something to do with the deletion of our transport statistics which are kept for 30 weeks to render graphs and reports. Once a new week starts, the 31st week is discarded. Somewhere in this process something must be wrong. Keep your fingers crossed that this theory is the right one and that we will find a good solution to it.
I'm feeling quite positive about this, sounds good and comprehensible. Let us know if you need any help of the players, like posting stats or whatever.
Respective changes have been implemented and are currently undergoing basic internal testing.
If everything goes according to plan I intend to deploy the changes to Devau and Gatow tomorrow which can - due to the nature of the changes - cause longer downtimes once again. I'm very sorry for this, but unfortunately I see no other way of doing it.
hm..... game worlds (aspern) do not work properly from Monday to Friday, not only on Mondays. Only in the early morning and on weekends you might have some good chances that the server is working almost properly.
In addition to low performance transport statistics seem to show wrong data at the moment.
Sorry guys, I have been paptient during the past months but now the situation is getting annoying. The game is far away from being fun.
I still have credits for the next six weeks and I am really hoping for improvement during that time.
I really hope this is worth it. If it has the desired effect we will definitely do the upgrade of the other game worlds a.) one by one and b.) over night.
Is Devau likely to be off all day today then? A little infuriating to say the least! This has gone on for weeks now and unless i'm mistaken we've been offered no compensation for it.
Is Devau likely to be off all day today then? A little infuriating to say the least! This has gone on for weeks now and unless i'm mistaken we've been offered no compensation for it.
Because, as always in the past, we offer and hand out compensations once we have fixed the issue at hand.
Whats more important is getting the problem solved once and for all. It really has been tedious trying to operate the airlines, especially when i operate multiple ones.
The copying process was so ridiculously slow that I have decided to cancel it and attempt to restart the game worlds without the statistics data for the moment. This will of course take a while as well to allow the database to roll back changes made over the past 6 hours (seriously, I can't fucking believe this). Once that worked I'll copy the statistics-data manually. The top priority right now is to get the game worlds back online though.
Thank you very much for keeping us informed. It´s really appalling to see that your work went fruitless despite your efforts. Let´s see and please keep us updated.
Both Devau and Gatow are back online for now. As said, I've left the statistics table more or less empty. I just manually copied in the data for this and the two/three previous weeks. Both worlds habe tremendous backlog and inserting the remaining statistics data puts a lot of load onto the database. As such I'll probably wait until tomorrow before I do this.
Devau came back online around 19:44 CET and has picked up around 2.5h of backlog since. Gatow only came online now. I'm monitoring both for the remainder of the evening.
