Ticket #180 (new defect)

Opened 6 years ago

Last modified 6 years ago

Technical fault at the server, could not add events to MUA

Reported by: zrdo/s4plnimbpbgme0qo5hxlla=@… Owned by: Rhys Smith
Priority: major Milestone:
Component: Distribution - RPM Version:
Keywords: Cc:

Description

We have recently installed Raptor. However, it runs for several hours as expected then stops. The log raptor-mua-process.log shows the following message:

 INFO 2014-02-19 14:20:28,669 (u.a.c.r.s.i.MUAProcessImpl:157) ---> MUA has received 1111 entries from https://elibrary.exeter.ac.uk/raptor-ica [Exeter IT]
 WARN 2014-02-19 14:20:28,669 (u.a.c.r.s.i.MUAProcessImpl:162) ---> Error trying to add authentications to this MUA, Transaction 394657 currently in processing, client should retry automatically
 INFO 2014-02-19 14:20:28,670 (o.a.c.c.l.Slf4jLogger:90) ---> Application {http://impl.wsinterface.raptormua.cardiff.ac.uk/}MultiUnitAggregatorImplService#{http://sei.server.remoting.raptor.cardiff.ac.uk/}addAuthentications has thrown exception, unwinding now: org.apache.cxf.binding.soap.SoapFault: Technical fault at the server, could not add events to MUA [Exeter IT]

Restarting the MUA fixes the problem then after several hours the problem occurs again.

We are using RHEL 6 with a MySQL database. We have 3 ICAs, one development and two production servers. The two production ones are on load balanced servers with the same entityId in the ICA configuration.

Change History

comment:1 Changed 6 years ago by smith@…

Can you switch the logging to DEBUG (in MUA\conf\logging.xml), and let us know what the logs say then?

comment:2 Changed 6 years ago by zrdo/s4plnimbpbgme0qo5hxlla=@…

DEBUG 2014-02-21 10:15:18,764 (o.m.l.Slf4jLog:40) ---> loaded class org.apache.cxf.interceptor.ServiceInvokerInterceptor from sun.misc.Launcher$AppClassLoader@4cf2f585
DEBUG 2014-02-21 10:15:18,764 (o.m.l.Slf4jLog:40) ---> loaded class org.apache.cxf.phase.PhaseInterceptorChain from sun.misc.Launcher$AppClassLoader@4cf2f585
DEBUG 2014-02-21 10:15:18,764 (o.a.c.c.l.Slf4jLogger:88) ---> Application {http://impl.wsinterface.raptormua.cardiff.ac.uk/}MultiUnitAggregatorImplService#{http://sei.server.remoting.raptor.cardiff.ac.uk/}addAuthentications has thrown exception, unwinding now
org.apache.cxf.binding.soap.SoapFault: Technical fault at the server, could not add events to MUA [Exeter IT]
	at uk.ac.cardiff.raptormua.service.impl.MUAProcessImpl.addAuthentications(MUAProcessImpl.java:175) ~[raptor-mua.jar:1.1.4]
	at uk.ac.cardiff.raptormua.wsinterface.impl.MultiUnitAggregatorImpl.addAuthentications(MultiUnitAggregatorImpl.java:131) ~[raptor-mua.jar:1.1.4]
	at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.6.0_28]
	at java.lang.reflect.Method.invoke(Method.java:622) ~[na:1.6.0_28]
	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_28]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_28]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_28]
	at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:110) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:98) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:423) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:178) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:142) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) ~[geronimo-servlet_2.5_spec-1.2.jar:1.2]
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159) ~[cxf-bundle-2.2.8.jar:2.2.8]
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.Server.handle(Server.java:322) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:680) ~[jetty-6.1.23.jar:6.1.23]
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) ~[jetty-util-6.1.23.jar:6.1.23]
DEBUG 2014-02-21 10:15:18,765 (o.a.c.c.l.Slf4jLogger:88) ---> Invoking handleFault on interceptor org.apache.cxf.interceptor.ServiceInvokerInterceptor@10c60380
DEBUG 2014-02-21 10:15:18,765 (o.a.c.c.l.Slf4jLogger:88) ---> Invoking handleFault on interceptor org.apache.cxf.jaxws.interceptors.HolderInInterceptor@9d8062f
DEBUG 2014-02-21 10:15:18,765 (o.a.c.c.l.Slf4jLogger:88) ---> Invoking handleFault on interceptor org.apache.cxf.jaxws.interceptors.SwAInInterceptor@457dfcb5
Note: See TracTickets for help on using tickets.