Home » Eclipse Projects » Virgo » Eclipse Virgo shutsdown for no reason after some hours on EC2
Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990278] |
Tue, 11 December 2012 17:04 |
Matteo Cantarelli Messages: 30 Registered: June 2012 |
Member |
|
|
I'm having a problem of a clean install of Eclipse Virgo (3.5) shutting down apparently for no reason once it's started on a Ubuntu image 12.04 on Amazon EC2. I'm starting Virgo from SSH, startup works fine but after some hours the server stops, the log report the following (see the third entry):
[2012-12-10 11:57:14.804] INFO http-bio-8080-exec-9 org.springframework.web.servlet.handler.SimpleUrlHandlerMapping Mapped URL path [/splash] onto handler 'splashController'
[2012-12-10 11:57:14.840] INFO http-bio-8080-exec-9 org.springframework.web.servlet.DispatcherServlet FrameworkServlet 'splash': initialization completed in 92 ms
[2012-12-10 15:33:27.488] INFO System Bundle Shutdown o.e.g.b.extender.internal.activator.ContextLoaderListener Stopping [org.eclipse.gemini.blueprint.extender] bundle v.[1.0.0.RELEASE]
[2012-12-10 15:33:27.493] INFO kernel-dm-10 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext Unpublishing application context OSGi service for bundle Virgo Kernel Admin
Shell (org.eclipse.virgo.kernel.shell)
[2012-12-10 15:33:27.494] INFO kernel-dm-10 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext Closing OsgiBundleXmlApplicationContext(bundle=org.eclipse.virgo.kernel.she
ll, config=osgibundle:/META-INF/spring/*.xml): startup date [Mon Dec 10 11:40:04 UTC 2012]; root of context hierarchy
[2012-12-10 15:33:27.495] INFO kernel-dm-10 o.s.beans.factory.support.DefaultListableBeanFactory Destroying singletons in org.springframework.beans.factory.support.DefaultL
istableBeanFactory@1fc7875: defining beans [commandProcessor,singleCommandExecutor,ioManager,completerRegistry,modelAccessorHelper,stateService,installCommand,bundleCommands,serviceCommands,packageComma
nds,configCommands,planCommands,parCommands,shutdownCommand,exitCommand,helpCommand,helpAccessor,helpCommandCompleter,bundleCompleter,packageCompleter,configCompleter,parCompleter,planCompleter,installC
ompleter,longConverter,stringConverter,commandResolver,commandRegistry,commandInvoker,converterRegistry,quasiFrameworkFactory,moduleContextAccessor,eventLogger,configAdmin,org.eclipse.gemini.blueprint.s
ervice.exporter.support.OsgiServiceFactoryBean#0,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#1,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBe
an#2,userRegionBundleContext,deployer,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#3,regionDigraph,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactor
yBean#4,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#5,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#6,org.eclipse.gemini.blueprint.service
.exporter.support.OsgiServiceFactoryBean#7,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#8,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#9,o
rg.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#10,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#11,org.eclipse.gemini.blueprint.service.export
er.support.OsgiServiceFactoryBean#12,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#13,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#14,org.e
clipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#15,objectNameCreator,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#16,org.eclipse.gemini.blueprint.
service.exporter.support.OsgiServiceFactoryBean#17,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#18,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactor
yBean#19,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#20,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#21,org.eclipse.gemini.blueprint.serv
ice.exporter.support.OsgiServiceFactoryBean#22]; root of factory hierarchy
[2012-12-10 15:33:27.497] INFO kernel-dm-10 o.e.g.blueprint.service.exporter.support.OsgiServiceFactoryBean Unregistered service [ServiceRegistrationWrapper for {org.eclipse.virgo.ker
nel.shell.CommandCompleter}={org.eclipse.virgo.kernel.shell.completer.commmandNames=install, org.eclipse.gemini.blueprint.bean.name=installCompleter, org.springframework.osgi.bean.name=installCompleter,
osgi.service.blueprint.compname=installCompleter, Bundle-SymbolicName=org.eclipse.virgo.kernel.shell, Bundle-Version=3.5.0.RELEASE, service.id=216}
These are the server properties
Java VM Description: Oracle Corporation(OpenJDK 64-Bit Server VM) 23.2-b09
Java Version: Oracle Corporation 1.7.0_09
Operating System: Linux(amd64) 3.2.0-31-virtual Server
Time Zone: Etc/UTC Virgo
Server Version: 3.5.0.RELEASE
Any idea or suggestion? Thanks!
[Updated on: Tue, 11 December 2012 17:06] Report message to a moderator
|
|
| | | | | | | |
Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990792 is a reply to message #990682] |
Fri, 14 December 2012 10:01 |
Glyn Normington Messages: 1222 Registered: July 2009 |
Senior Member |
|
|
I tested your logger configuration (on a 3.6.0 build which has an improved ShutdownManager, but the principle is the same) and there were lots of extra log entries, e.g. when any bundle starts the ShutdownManager's ShutdownLoggingListener is driven. I suggest you test your changes locally to get the logging working before running again on EC2.
Another theory for the failure is that that JVM permgen or heap is too small and you are hitting an obscure out of memory or similar failure perhaps due to a "slow" memory leak either in Virgo or in an application. (Virgo is woven with an aspect which logs runtime exceptions thrown in Virgo code, so I would hope to see evidence of a Virgo memory leak resulting in an OOM in the log, but it's possible that the leak is elsewhere or an exception is not being logged for some reason.)
Anyway, getting the logging of ShutdownManager working for starters will help as we'll get some clues about whether the OSGi framework is shutting down vs the JVM exiting (and driving ShutdownManager's JVM shutdown hook) vs the JVM halting without driving shutdown hooks.
[Updated on: Fri, 14 December 2012 10:02] Report message to a moderator
|
|
| | | |
Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #992897 is a reply to message #990278] |
Sun, 23 December 2012 00:16 |
Matteo Cantarelli Messages: 30 Registered: June 2012 |
Member |
|
|
Hi All,
An update on this issue.
I tried again with release 3.6 after adding the log tracing to ShutdownManager, here's the log (it shuts down after few hours):
[2012-12-22 14:56:54.775] INFO start-signalling-2 o.e.v.k.i.artifact.internal.bundle.BundleThreadContextManager Thread context class loader 'KernelBundleClassLoader: [bundle=
org.eclipse.virgo.apps.repository-3.6.0.RELEASE-synthetic.context_3.6.0.RELEASE]' popped and set to 'KernelBundleClassLoader: [bundle=org.eclipse.virgo.web.core_3.6.0.RELEASE]'
[2012-12-22 14:56:54.777] INFO start-signalling-2 org.eclipse.virgo.medic.eventlog.default DE0005I Started par 'org.eclipse.virgo.apps.repository' version
'3.6.0.RELEASE'.
[2012-12-22 17:08:12.935] DEBUG Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager$1 > public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.1.run()
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager > static boolean org.eclipse.virgo.nano.core.internal.Shutdown
Manager.access$0(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager > private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping()
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager < private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping()
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager < static boolean org.eclipse.virgo.nano.core.internal.Shutdown
Manager.access$0(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager > static void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.access$1(org.eclipse.virgo.nano.core.internal.ShutdownManager, boolean)
[2012-12-22 17:08:12.936] TRACE Thread-8 org.eclipse.virgo.nano.core.internal.ShutdownManager > private void org.eclipse.virgo.nano.core.internal.ShutdownMa
nager.doShutdown(boolean)
[2012-12-22 17:08:12.938] DEBUG System Bundle Shutdown o.e.v.nano.core.internal.ShutdownManager$ShutdownLoggingListener > public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.ShutdownLoggingListener.bundleChanged(org.osgi.framework.BundleEvent)
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager > static org.osgi.framework.launch.Framework org.eclipse.virgo
.nano.core.internal.ShutdownManager.access$2(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager < static org.osgi.framework.launch.Framework org.eclipse.virgo
.nano.core.internal.ShutdownManager.access$2(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager > static org.eclipse.virgo.medic.eventlog.EventLogger org.ecli
pse.virgo.nano.core.internal.ShutdownManager.access$3(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager < static org.eclipse.virgo.medic.eventlog.EventLogger org.ecli
pse.virgo.nano.core.internal.ShutdownManager.access$3(org.eclipse.virgo.nano.core.internal.ShutdownManager)
[2012-12-22 17:08:12.939] INFO System Bundle Shutdown org.eclipse.virgo.medic.eventlog.default KE0010I Shutdown initiated.
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager > final void org.eclipse.virgo.nano.core.internal.ShutdownMana
ger.removeShutdownHook()
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager > private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping()
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager < private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping()
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown org.eclipse.virgo.nano.core.internal.ShutdownManager < final void org.eclipse.virgo.nano.core.internal.ShutdownMana
ger.removeShutdownHook()
[2012-12-22 17:08:12.939] DEBUG System Bundle Shutdown o.e.v.nano.core.internal.ShutdownManager$ShutdownLoggingListener < public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.ShutdownLoggingListener.bundleChanged(org.osgi.framework.BundleEvent)
[2012-12-22 17:08:12.939] INFO System Bundle Shutdown o.e.g.b.extender.internal.activator.ContextLoaderListener Stopping [org.eclipse.gemini.blueprint.extender] bundle v.[1.0
.2.RELEASE]
[2012-12-22 17:08:12.944] INFO kernel-dm-8 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext Unpublishing application context OSGi service for bundle Virgo
Kernel Runtime Artifact Model (org.eclipse.virgo.kernel.model)
[2012-12-22 17:08:12.946] INFO kernel-dm-8 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext Closing OsgiBundleXmlApplicationContext(bundle=org.eclipse.vir
go.kernel.model, config=osgibundle:/META-INF/spring/*.xml): startup date [Sat Dec 22 14:52:39 UTC 2012]; root of context hierarchy
[2012-12-22 17:08:12.947] INFO kernel-dm-8 o.s.beans.factory.support.DefaultListableBeanFactory Destroying singletons in org.springframework.beans.factory.sup
port.DefaultListableBeanFactory@305e9d7a: defining beans [org.eclipse.virgo.kernel.model.internal.bundle.ModelBundleListenerInitializer#0,org.eclipse.virgo.kernel.model.internal.configurati
onadmin.ModelConfigurationListenerInitializer#0,org.eclipse.virgo.kernel.model.internal.deployer.ModelInstallArtifactLifecycleListenerInitializer#0,bundleDependencyDeterminer,deployerCompos
iteArtifactDependencyDeterminer,artifactRepository,artifactRepositoryListener,springContextAccessor,runtimeArtifactModelObjectNameCreator,kernelConfig,org.springframework.context.annotation
.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProces
sor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#0,org.eclipse.gemini.blueprint.serv
ice.exporter.support.OsgiServiceFactoryBean#1,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#2,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServic
eFactoryBean#3,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#4,packageAdminUtil,configurationAdmin,quasiFrameworkFactory,runtimeArtifactModel,regionDigraph,gl
obalRegion,userRegion]; root of factory hierarchy
[2012-12-22 17:08:12.949] INFO iLogServiceListener@2d26c6ea org.osgi.service.log.LogService Bundle org.eclipse.virgo.kernel.model_3.6.0.RELEASE, Service 1
93, ServiceEvent UNREGISTERING
[2012-12-22 17:08:12.950] INFO iLogServiceListener@2d26c6ea org.osgi.service.log.LogService Bundle org.eclipse.virgo.kernel.model_3.6.0.RELEASE, Service 1
92, ServiceEvent UNREGISTERING
[2012-12-22 17:08:12.950] INFO iLogServiceListener@2d26c6ea org.osgi.service.log.LogService
Any hint there as to why is it shutting down?
@Hristo it happens usually after 2 hours, the longest I had to wait was 6 hours. I tried both on a free micro instance and in a paid medium one, it happens on both. Also I tried both with ubuntu and the amazon linux and it happens on both. It happens without any bundles installed, i.e. just unzipped virgo downloaded from the wesbite, scped to the server and started from an ssh connection using startup.sh. How are you starting it?
Thanks,
Matteo
|
|
| |
Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #993114 is a reply to message #992902] |
Sun, 23 December 2012 12:28 |
|
Hi Matteo,
When SSH client is disconnected the processes started in the session are stopped/killed.
To workaround this you can either use screen or similar program to manage the sessions or set your client to send keep-alive packages. You can also increase or completely disable the SSH server timeout.
When I tested Virgo on EC2 I've had these keep-alive set to 60 seconds, so I did not manage to reproduce the problem.
Better approach imho would be to create a init script/service/daemon to keep Virgo started even after OS reboot or crash.
Regards,
Hristo Iliev
|
|
|
Goto Forum:
Current Time: Fri Apr 19 13:59:44 GMT 2024
Powered by FUDForum. Page generated in 0.03177 seconds
|