Chaos Monday: Various problems (extending to Tuesday)

Server Status
21

Croydon is back online, as is Idlewild, both with several hours of backlog though.

The situation of the other game worlds has somewhat stabilized, but we still see occasional, spontaneous shutdowns.

The failed hard drive of the Idlewild/Croydon machine failed to recover, so we requested the data center to replace the component. Once they get to it, the server and the game worlds running on it will be down once more, hopefully only for a short period of time though.

No real news from the Stapleton front: It still appears to run smooth like butter while Aspern's figures are not really that great, at least not permanently.

22

Meigs has run into problems again and so has Kaitak. I hope this day is over soon.

23

Murphys Laws. Nevermind - keep your head high. ;)

24

Status Update

I haven't had much time to actually think about the causes of the issues today, since there were so many fires to extinguish. One clue that only came to my mind now is that only game worlds are affected that have not had their statistics converted to the new format yet. I tried to avoid doing this as this causes quite a bit of disturbance until all the data is transferred. 

Nevertheless, this is the only thing the affected game worlds have in common, so I'll give it a try on the server that hosts Nicosia and Pearls first.

25

Status Update

Will try the same thing on other game worlds. Nicosia and Pearls have to rebuild their statistics (the most important weeks to render "Facts and Figures" have already been completed) but all the other issues seem to have disappeared for now.

First up are Fornebu and Tempelhof. Croydon and Idlewild are candidates, too.

26

Thanks for the night shift.  :)  I hope, you won't have to get up early tomorrow.

27

Status Update

Fornebu and Tempelhof have received the updated, I will wait with other game worlds until tomorrow. 

I'm going to bed...have been sitting in front of this thing for well over 15 hours now. Chances are that something will go wrong tonight. If so, I will take care of it first thing in the morning. 

On a side-note: I'm also letting Stapleton complete its statistics transfer. Shouldn't pose much of a problem with a new setup to get back to the full set of 25 weeks.

28

Sorry das ich nerve aber es wäre schön wenn hier jemand mal im deutschen etwas zusammenfasst. Bin des englischen nicht so mächtig und google translate spuckt nur komisches aus.

29

Pearls message today:

An unexpected error occured!

Please see stacktrace below:

org.hibernate.exception.SQLGrammarException: could not extract ResultSet

org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:82)

org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)

org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)

org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)

org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:88)

org.hibernate.loader.Loader.getResultSet(Loader.java:2062)

org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859)

org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838)

org.hibernate.loader.Loader.doQuery(Loader.java:906)

org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348)

org.hibernate.loader.Loader.doList(Loader.java:2548)

org.hibernate.loader.Loader.doList(Loader.java:2534)

org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2364)

org.hibernate.loader.Loader.list(Loader.java:2359)

org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:124)

org.hibernate.internal.SessionImpl.list(SessionImpl.java:1587)

org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)

de.airlinesim.data.NewsData.loadLatestArticles(NewsData.java:67)

de.airlinesim.model.report.portal.PortalIndexReport.(PortalIndexReport.java:209)

de.airlinesim.operations.portal.PortalIndexOperation.execute(PortalIndexOperation.java:47)

ch.rhj.business.filters.ExecutionFilter.filter(ExecutionFilter.java:48)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

de.airlinesim.PremiumOperationFilter.filter(PremiumOperationFilter.java:66)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

ch.rhj.business.filters.AuthorizationFilter.filter(AuthorizationFilter.java:63)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

de.airlinesim.MaintenanceModeFilter.filter(MaintenanceModeFilter.java:39)

de.airlinesim.operation.OperationRunnerService.executeReadOnly(OperationRunnerService.java:44)

de.airlinesim.operation.OperationRunnerService$$FastClassByCGLIB$$9ac9cd4d.invoke()

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)

org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634)

de.airlinesim.operation.OperationRunnerService$$EnhancerByCGLIB$$3abb7e2d.executeReadOnly()

de.airlinesim.OperationExecutor.execute(OperationExecutor.java:30)

de.airlinesim.web.actions.AirlineSimAction.reading(AirlineSimAction.java:142)

de.airlinesim.web.actions.portal.PortalIndexAction.execute(PortalIndexAction.java:28)

sun.reflect.GeneratedMethodAccessor1304.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

java.lang.reflect.Method.invoke(Method.java:597)

com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:404)

com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:267)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:229)

de.airlinesim.web.interceptors.ExceptionInterceptor.intercept(ExceptionInterceptor.java:46)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.BusinessContextInterceptor.intercept(BusinessContextInterceptor.java:147)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.LanguageResourcesInterceptor.intercept(LanguageResourcesInterceptor.java:107)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.NoCacheInterceptor.intercept(NoCacheInterceptor.java:72)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)

org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:184)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50)

org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504)

org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:422)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:149)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:40)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.InstallModeFilter.doFilter(InstallModeFilter.java:48)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.FirstContactFilter.doFilter(FirstContactFilter.java:72)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:167)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:56)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

java.lang.Thread.run(Thread.java:662)

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown table engine 'InnoDB'

sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown Source)

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

java.lang.reflect.Constructor.newInstance(Constructor.java:513)

com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

com.mysql.jdbc.Util.getInstance(Util.java:386)

com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)

com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)

com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)

com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)

com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)

com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)

com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:116)

org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79)

org.hibernate.loader.Loader.getResultSet(Loader.java:2062)

org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1859)

org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1838)

org.hibernate.loader.Loader.doQuery(Loader.java:906)

org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348)

org.hibernate.loader.Loader.doList(Loader.java:2548)

org.hibernate.loader.Loader.doList(Loader.java:2534)

org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2364)

org.hibernate.loader.Loader.list(Loader.java:2359)

org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:124)

org.hibernate.internal.SessionImpl.list(SessionImpl.java:1587)

org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)

de.airlinesim.data.NewsData.loadLatestArticles(NewsData.java:67)

de.airlinesim.model.report.portal.PortalIndexReport.(PortalIndexReport.java:209)

de.airlinesim.operations.portal.PortalIndexOperation.execute(PortalIndexOperation.java:47)

ch.rhj.business.filters.ExecutionFilter.filter(ExecutionFilter.java:48)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

de.airlinesim.PremiumOperationFilter.filter(PremiumOperationFilter.java:66)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

ch.rhj.business.filters.AuthorizationFilter.filter(AuthorizationFilter.java:63)

ch.rhj.business.BusinessFilter.filter(BusinessFilter.java:43)

de.airlinesim.MaintenanceModeFilter.filter(MaintenanceModeFilter.java:39)

de.airlinesim.operation.OperationRunnerService.executeReadOnly(OperationRunnerService.java:44)

de.airlinesim.operation.OperationRunnerService$$FastClassByCGLIB$$9ac9cd4d.invoke()

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)

org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634)

de.airlinesim.operation.OperationRunnerService$$EnhancerByCGLIB$$3abb7e2d.executeReadOnly()

de.airlinesim.OperationExecutor.execute(OperationExecutor.java:30)

de.airlinesim.web.actions.AirlineSimAction.reading(AirlineSimAction.java:142)

de.airlinesim.web.actions.portal.PortalIndexAction.execute(PortalIndexAction.java:28)

sun.reflect.GeneratedMethodAccessor1304.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

java.lang.reflect.Method.invoke(Method.java:597)

com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:404)

com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:267)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:229)

de.airlinesim.web.interceptors.ExceptionInterceptor.intercept(ExceptionInterceptor.java:46)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.BusinessContextInterceptor.intercept(BusinessContextInterceptor.java:147)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.LanguageResourcesInterceptor.intercept(LanguageResourcesInterceptor.java:107)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

de.airlinesim.web.interceptors.NoCacheInterceptor.intercept(NoCacheInterceptor.java:72)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)

org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:184)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121)

com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50)

org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504)

org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:422)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:149)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:40)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.InstallModeFilter.doFilter(InstallModeFilter.java:48)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.FirstContactFilter.doFilter(FirstContactFilter.java:72)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:167)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

de.airlinesim.web.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:56)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

java.lang.Thread.run(Thread.java:662)

30

Hello, 

server kaitak yesterday evening worked. But his morning not:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.2.16 (Debian) Server at kaitak.airlinesim.aero Port 80
31

Hello, 

server kaitak yesterday evening worked. But his morning not:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.2.16 (Debian) Server at kaitak.airlinesim.aero Port 80

same problem wth croydon again

32

"Good" Morning...Status Update

As somewhat expected, the night brought a lot of problems. Many game worlds are currently making trouble or are offline altogether with table repairs running all over the place. 

Quick overview:

  • Croydon and Idlewild: Already failed to restart last night and have been offline since. Now waiting for table repairs, then it's just "hoping and waiting" because we can't really do anything until the data center has switched that dreaded failed disk.
  • Nicosia, Pearls, Fornebu and Tempelhof have received the patch for the new database format which stabilized them somewhat. The statistics transfer (which proceeded surprisingly well) is causing a very high memory load though which causes problems. I will stop the data transfer for them as one of the first things in the morning.
  • Kaitak and Meigs will get the same treatment after as the 4 above.
  • We will be moving Gatow to new hardware (instead of Devau) this morning. Devau might receive new hardware this week as well, since we intend to split them up to use separate servers.
  • All others: No reports right now, Stapleton continues to do great.
33

One question - warning might be dumb, I dont know - but would it help to write clients that we as users could run on our pcs to help your servers to recover backlogs and manage huge workloads? Just like folding@home does it (http://folding.stanford.edu/). Of course it would cost time to build such a client in the first place, but me personally I have on gaming rig here (I7 4790K, 16 GB (non-ECC), SSD) and also a small file server (dual core with 8 gigs of ECC Ram) and I would be happy to allow you guys to use them as a power resource when I don´t need the (full) power.

Just a crazy Idea ;)

34

Status Update

Gatow has been moved to new hardware. The move went surprisingly quick and the game world has no backlog already. Moving on the the next items on my to-do list...

@Nils: I don't think this would be feasible in our situation.

35

Oh, forgot: DNS record for Gatow has changed again...if you can't access it yet, it's probably because of your ISP not having the new address yet.

36

Current status:

All game worlds except Croydon are at least up at the moment. Backlogs vary.

Idlewild and Croydon are the only game worlds remaining in "old configuration" until the failed disk has been switched. Then we'll upgrade them as well. Since Idlewild is picking up a lot of backlog right now, we'll leave Croydon offline for the moment.

37

What do you mean with ‘hoping’ for Croydon and Idle wild? Is there a possibility that the additional hardware fault on top of the software changes made these gameworlds go bust…??

38

What do you mean with 'hoping' for Croydon and Idle wild? Is there a possibility that the additional hardware fault on top of the software changes made these gameworlds go bust....??

No, just hoping that they can somewhat keep up performance-wise (with the old configuration) until the disk has been switched and we can upgrade them. The data of both game worlds is unharmed.

39

Auf Server MEIGS starten heute meine Flüge nicht.

ERE 1   06:00     booked

ERE 2   07:49     booked

ERE 3   09:27     booked

ERE 11 04:58     booked

ERE 12 06:20     booked

ERE 13 08:13     booked

stimmt da etwas nicht auf dem Server, oder was ist die Ursache?

40

stimmt da etwas nicht auf dem Server, oder was ist die Ursache?

Ich hoffe die Frage ist ein Scherz: http://meigs.airlinesim.aero/app/info/status