Eclipse Community Forums: Virgo » Eclipse Virgo shutsdown for no reason after some hours on EC2

Home » Eclipse Projects » Virgo » Eclipse Virgo shutsdown for no reason after some hours on EC2

Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990278]

Tue, 11 December 2012 12:04

Eclipse User

I'm having a problem of a clean install of Eclipse Virgo (3.5) shutting down apparently for no reason once it's started on a Ubuntu image 12.04 on Amazon EC2. I'm starting Virgo from SSH, startup works fine but after some hours the server stops, the log report the following (see the third entry):

[2012-12-10 11:57:14.804] INFO  http-bio-8080-exec-9         org.springframework.web.servlet.handler.SimpleUrlHandlerMapping   Mapped URL path [/splash] onto handler 'splashController' 
[2012-12-10 11:57:14.840] INFO  http-bio-8080-exec-9         org.springframework.web.servlet.DispatcherServlet                 FrameworkServlet 'splash': initialization completed in 92 ms 
[2012-12-10 15:33:27.488] INFO  System Bundle Shutdown       o.e.g.b.extender.internal.activator.ContextLoaderListener         Stopping [org.eclipse.gemini.blueprint.extender] bundle v.[1.0.0.RELEASE] 
[2012-12-10 15:33:27.493] INFO  kernel-dm-10                 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext   Unpublishing application context OSGi service for bundle Virgo Kernel Admin
 Shell (org.eclipse.virgo.kernel.shell) 
[2012-12-10 15:33:27.494] INFO  kernel-dm-10                 o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext   Closing OsgiBundleXmlApplicationContext(bundle=org.eclipse.virgo.kernel.she
ll, config=osgibundle:/META-INF/spring/*.xml): startup date [Mon Dec 10 11:40:04 UTC 2012]; root of context hierarchy 
[2012-12-10 15:33:27.495] INFO  kernel-dm-10                 o.s.beans.factory.support.DefaultListableBeanFactory              Destroying singletons in org.springframework.beans.factory.support.DefaultL
istableBeanFactory@1fc7875: defining beans [commandProcessor,singleCommandExecutor,ioManager,completerRegistry,modelAccessorHelper,stateService,installCommand,bundleCommands,serviceCommands,packageComma
nds,configCommands,planCommands,parCommands,shutdownCommand,exitCommand,helpCommand,helpAccessor,helpCommandCompleter,bundleCompleter,packageCompleter,configCompleter,parCompleter,planCompleter,installC
ompleter,longConverter,stringConverter,commandResolver,commandRegistry,commandInvoker,converterRegistry,quasiFrameworkFactory,moduleContextAccessor,eventLogger,configAdmin,org.eclipse.gemini.blueprint.s
ervice.exporter.support.OsgiServiceFactoryBean#0,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#1,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBe
an#2,userRegionBundleContext,deployer,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#3,regionDigraph,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactor
yBean#4,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#5,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#6,org.eclipse.gemini.blueprint.service
.exporter.support.OsgiServiceFactoryBean#7,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#8,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#9,o
rg.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#10,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#11,org.eclipse.gemini.blueprint.service.export
er.support.OsgiServiceFactoryBean#12,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#13,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#14,org.e
clipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#15,objectNameCreator,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#16,org.eclipse.gemini.blueprint.
service.exporter.support.OsgiServiceFactoryBean#17,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#18,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactor
yBean#19,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#20,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#21,org.eclipse.gemini.blueprint.serv
ice.exporter.support.OsgiServiceFactoryBean#22]; root of factory hierarchy 
[2012-12-10 15:33:27.497] INFO  kernel-dm-10                 o.e.g.blueprint.service.exporter.support.OsgiServiceFactoryBean   Unregistered service [ServiceRegistrationWrapper for {org.eclipse.virgo.ker
nel.shell.CommandCompleter}={org.eclipse.virgo.kernel.shell.completer.commmandNames=install, org.eclipse.gemini.blueprint.bean.name=installCompleter, org.springframework.osgi.bean.name=installCompleter,
 osgi.service.blueprint.compname=installCompleter, Bundle-SymbolicName=org.eclipse.virgo.kernel.shell, Bundle-Version=3.5.0.RELEASE, service.id=216}

These are the server properties

Java VM Description: Oracle Corporation(OpenJDK 64-Bit Server VM) 23.2-b09
Java Version: Oracle Corporation 1.7.0_09
Operating System: Linux(amd64) 3.2.0-31-virtual Server
Time Zone: Etc/UTC Virgo
Server Version: 3.5.0.RELEASE

Any idea or suggestion? Thanks!

[Updated on: Tue, 11 December 2012 12:06] by Moderator

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990359 is a reply to message #990278]

Wed, 12 December 2012 02:13

Eclipse User

Do you have the Virgo console with default user/password? If so perhaps someone initiated shutdown?

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990404 is a reply to message #990278]

Wed, 12 December 2012 05:26

Eclipse User

It looks like someone (as per Hristo's comment) or something stopped the OSGi framework. I'm not aware of this ever happening spontaneously, so there must be some cause even if it's a stray startup timeout kicking in much later than expected.

Note that the shutdown involves the thread named "System Bundle Shutdown", so you could investigate where that comes from. I couldn't find anywhere obvious in the Virgo codebase that uses such a thread name, so I presume this comes from Equinox. However, I'm not certain that will get you any closer to knowing who/what initiated shutdown as it may be that this thread is always involved in shutdown.

We've occasionally seen the system bundle being stopped as a result of a refresh operation that somehow took down really basic bundles in the system, so that's another possibility. Shame there is nothing in the logs to show what was going on. Is there anything in the event log just before the shutdown which might give some clues? (There shouldn't be as Virgo is supposed to log event log activity in the trace log too.)

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990405 is a reply to message #990404]

Wed, 12 December 2012 05:30

Eclipse User

Unfortunately nobody knows the public DNS to stop it from console, every time I start the server it's a different DNS also, and it happens every time after 2 or 3 hours. I also thought about some sort of timeout but the timing when this happens is not consistent. I am at loss Sad

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990409 is a reply to message #990405]

Wed, 12 December 2012 05:45

Eclipse User

Perhaps one way forward is to selectively increase the logging of the package containing the ShutdownManager class in order to try to get some more clues. The package is org.eclipse.virgo.nano.core.internal and I suggest you set the log level to TRACE. See the User Guide plus the Logback documentation for how to do this. You can easily test the logger configuration is working by using the shutdown script locally and seeing what gets logged. Keep in touch - this is an interesting problem.

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990412 is a reply to message #990405]

Wed, 12 December 2012 05:49

Eclipse User

Can you as well check for some VM output or crash logs (should be something like hs_err_pid<pid>.log) ?

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990415 is a reply to message #990409]

Wed, 12 December 2012 06:25

Eclipse User

I added the following to serviceability.xml, will let you know as soon as it shuts down again, thanks.

         <logger level="TRACE" additivity="false" name="org.eclipse.virgo.nano.core.internal.ShutdownManager">
		<appender-ref ref="SIFTED_LOG_FILE" />
		<appender-ref ref="LOG_FILE" />
	</logger>

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990682 is a reply to message #990415]

Thu, 13 December 2012 14:11

Eclipse User

Nothing extra appears on the log. Is what I added wrong? I also tried to use a different image on Amazon, switching from Ubuntu to the Amazon distro (based on RedHat) which has JDK preinstalled. It just died. I don't know what to do, this seems pretty basic to me, I just created an Amazon instance copied in an empty Virgo 3.5 release and started it. Has anyone ever tried to use Virgo on EC2?

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990792 is a reply to message #990682]

Fri, 14 December 2012 05:01

Eclipse User

I tested your logger configuration (on a 3.6.0 build which has an improved ShutdownManager, but the principle is the same) and there were lots of extra log entries, e.g. when any bundle starts the ShutdownManager's ShutdownLoggingListener is driven. I suggest you test your changes locally to get the logging working before running again on EC2.

Another theory for the failure is that that JVM permgen or heap is too small and you are hitting an obscure out of memory or similar failure perhaps due to a "slow" memory leak either in Virgo or in an application. (Virgo is woven with an aspect which logs runtime exceptions thrown in Virgo code, so I would hope to see evidence of a Virgo memory leak resulting in an OOM in the log, but it's possible that the leak is elsewhere or an exception is not being logged for some reason.)

Anyway, getting the logging of ShutdownManager working for starters will help as we'll get some clues about whether the OSGi framework is shutting down vs the JVM exiting (and driving ShutdownManager's JVM shutdown hook) vs the JVM halting without driving shutdown hooks.

[Updated on: Fri, 14 December 2012 05:02] by Moderator

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990821 is a reply to message #990792]

Fri, 14 December 2012 06:00

Eclipse User

I tried locally with 3.6.0 M5 and logging worked also for me. Then I copied 3.6 on EC2 and tried to start it but I got the following error:

Date: 14-Dec-2012
Time: 10:54:29 UTC

Cause: error

Exception:
java.lang.IllegalStateException: BundleContext is no longer valid
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.checkValid(BundleContextImpl.java:931)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.getServiceReference(BundleContextImpl.java:528)
at org.eclipse.virgo.kernel.osgi.framework.OsgiFrameworkUtils.getService(OsgiFrameworkUtils.java:87)
at org.eclipse.virgo.shell.osgicommand.Activator.getPotentiallyDelayedService(Activator.java:110)
at org.eclipse.virgo.shell.osgicommand.Activator.access$0(Activator.java:105)
at org.eclipse.virgo.shell.osgicommand.Activator$PostStartInitialisationRunnable.run(Activator.java:146)
at java.lang.Thread.run(Thread.java:679)

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990832 is a reply to message #990278]

Fri, 14 December 2012 06:33

Eclipse User

OK, that was my fault, forgot to clean the server. It's running now, waiting for it to fail again...

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #990957 is a reply to message #990832]

Sat, 15 December 2012 07:00

Eclipse User

I'm running Virgo 3.5 on Ubuntu with Oracle JDK 1.6 on EC2 for more than 8 hours but I cannot reproduce the issue you are facing.

Do you have some additional bundles/PARs/plans installed or some bundles in the Virgo repository?

How much time does it take on your side? Do you use free tier or a paid instance?

[Updated on: Sat, 15 December 2012 09:46] by Moderator

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #992897 is a reply to message #990278]

Sat, 22 December 2012 19:16

Eclipse User

Hi All,
An update on this issue.
I tried again with release 3.6 after adding the log tracing to ShutdownManager, here's the log (it shuts down after few hours):

[2012-12-22 14:56:54.775] INFO  start-signalling-2           o.e.v.k.i.artifact.internal.bundle.BundleThreadContextManager     Thread context class loader 'KernelBundleClassLoader: [bundle=
org.eclipse.virgo.apps.repository-3.6.0.RELEASE-synthetic.context_3.6.0.RELEASE]' popped and set to 'KernelBundleClassLoader: [bundle=org.eclipse.virgo.web.core_3.6.0.RELEASE]' 
[2012-12-22 14:56:54.777] INFO  start-signalling-2           org.eclipse.virgo.medic.eventlog.default                         DE0005I Started par 'org.eclipse.virgo.apps.repository' version
 '3.6.0.RELEASE'. 
[2012-12-22 17:08:12.935] DEBUG Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager$1            > public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.1.run()
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              > static boolean org.eclipse.virgo.nano.core.internal.Shutdown
Manager.access$0(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              > private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping() 
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              < private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping() 
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              < static boolean org.eclipse.virgo.nano.core.internal.Shutdown
Manager.access$0(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              > static void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.access$1(org.eclipse.virgo.nano.core.internal.ShutdownManager, boolean) 
[2012-12-22 17:08:12.936] TRACE Thread-8                     org.eclipse.virgo.nano.core.internal.ShutdownManager              > private void org.eclipse.virgo.nano.core.internal.ShutdownMa
nager.doShutdown(boolean) 
[2012-12-22 17:08:12.938] DEBUG System Bundle Shutdown       o.e.v.nano.core.internal.ShutdownManager$ShutdownLoggingListener  > public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.ShutdownLoggingListener.bundleChanged(org.osgi.framework.BundleEvent) 
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              > static org.osgi.framework.launch.Framework org.eclipse.virgo
.nano.core.internal.ShutdownManager.access$2(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              < static org.osgi.framework.launch.Framework org.eclipse.virgo
.nano.core.internal.ShutdownManager.access$2(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              > static org.eclipse.virgo.medic.eventlog.EventLogger org.ecli
pse.virgo.nano.core.internal.ShutdownManager.access$3(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.938] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              < static org.eclipse.virgo.medic.eventlog.EventLogger org.ecli
pse.virgo.nano.core.internal.ShutdownManager.access$3(org.eclipse.virgo.nano.core.internal.ShutdownManager) 
[2012-12-22 17:08:12.939] INFO  System Bundle Shutdown       org.eclipse.virgo.medic.eventlog.default                         KE0010I Shutdown initiated. 
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              > final void org.eclipse.virgo.nano.core.internal.ShutdownMana
ger.removeShutdownHook() 
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              > private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping() 
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              < private boolean org.eclipse.virgo.nano.core.internal.Shutdow
nManager.compareAndSetHookStopping() 
[2012-12-22 17:08:12.939] TRACE System Bundle Shutdown       org.eclipse.virgo.nano.core.internal.ShutdownManager              < final void org.eclipse.virgo.nano.core.internal.ShutdownMana
ger.removeShutdownHook() 
[2012-12-22 17:08:12.939] DEBUG System Bundle Shutdown       o.e.v.nano.core.internal.ShutdownManager$ShutdownLoggingListener  < public void org.eclipse.virgo.nano.core.internal.ShutdownMan
ager.ShutdownLoggingListener.bundleChanged(org.osgi.framework.BundleEvent) 
[2012-12-22 17:08:12.939] INFO  System Bundle Shutdown       o.e.g.b.extender.internal.activator.ContextLoaderListener         Stopping [org.eclipse.gemini.blueprint.extender] bundle v.[1.0
.2.RELEASE] 
[2012-12-22 17:08:12.944] INFO  kernel-dm-8                  o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext   Unpublishing application context OSGi service for bundle Virgo
 Kernel Runtime Artifact Model (org.eclipse.virgo.kernel.model) 
[2012-12-22 17:08:12.946] INFO  kernel-dm-8                  o.e.g.blueprint.context.support.OsgiBundleXmlApplicationContext   Closing OsgiBundleXmlApplicationContext(bundle=org.eclipse.vir
go.kernel.model, config=osgibundle:/META-INF/spring/*.xml): startup date [Sat Dec 22 14:52:39 UTC 2012]; root of context hierarchy 
[2012-12-22 17:08:12.947] INFO  kernel-dm-8                  o.s.beans.factory.support.DefaultListableBeanFactory              Destroying singletons in org.springframework.beans.factory.sup
port.DefaultListableBeanFactory@305e9d7a: defining beans [org.eclipse.virgo.kernel.model.internal.bundle.ModelBundleListenerInitializer#0,org.eclipse.virgo.kernel.model.internal.configurati
onadmin.ModelConfigurationListenerInitializer#0,org.eclipse.virgo.kernel.model.internal.deployer.ModelInstallArtifactLifecycleListenerInitializer#0,bundleDependencyDeterminer,deployerCompos
iteArtifactDependencyDeterminer,artifactRepository,artifactRepositoryListener,springContextAccessor,runtimeArtifactModelObjectNameCreator,kernelConfig,org.springframework.context.annotation
.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProces
sor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#0,org.eclipse.gemini.blueprint.serv
ice.exporter.support.OsgiServiceFactoryBean#1,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#2,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServic
eFactoryBean#3,org.eclipse.gemini.blueprint.service.exporter.support.OsgiServiceFactoryBean#4,packageAdminUtil,configurationAdmin,quasiFrameworkFactory,runtimeArtifactModel,regionDigraph,gl
obalRegion,userRegion]; root of factory hierarchy 
[2012-12-22 17:08:12.949] INFO  iLogServiceListener@2d26c6ea org.osgi.service.log.LogService                                   Bundle org.eclipse.virgo.kernel.model_3.6.0.RELEASE, Service 1
93, ServiceEvent UNREGISTERING 
[2012-12-22 17:08:12.950] INFO  iLogServiceListener@2d26c6ea org.osgi.service.log.LogService                                   Bundle org.eclipse.virgo.kernel.model_3.6.0.RELEASE, Service 1
92, ServiceEvent UNREGISTERING 
[2012-12-22 17:08:12.950] INFO  iLogServiceListener@2d26c6ea org.osgi.service.log.LogService

Any hint there as to why is it shutting down?
@Hristo it happens usually after 2 hours, the longest I had to wait was 6 hours. I tried both on a free micro instance and in a paid medium one, it happens on both. Also I tried both with ubuntu and the amazon linux and it happens on both. It happens without any bundles installed, i.e. just unzipped virgo downloaded from the wesbite, scped to the server and started from an ssh connection using startup.sh. How are you starting it?

Thanks,
Matteo

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #992902 is a reply to message #992897]

Sat, 22 December 2012 19:38

Eclipse User

Could it be the case that 2 hours after the ssh connection dies the server shuts down? Maybe when it lasted longer it's because I was still using the machine and the ssh connection was open but as soon as I shut it down and the connection was lost a 2hrs countdown started?

Re: Eclipse Virgo shutsdown for no reason after some hours on EC2 [message #993114 is a reply to message #992902]

Sun, 23 December 2012 07:28

Eclipse User

Hi Matteo,

When SSH client is disconnected the processes started in the session are stopped/killed.

To workaround this you can either use screen or similar program to manage the sessions or set your client to send keep-alive packages. You can also increase or completely disable the SSH server timeout.

When I tested Virgo on EC2 I've had these keep-alive set to 60 seconds, so I did not manage to reproduce the problem.

Better approach imho would be to create a init script/service/daemon to keep Virgo started even after OS reboot or crash.

Regards,
Hristo Iliev

Previous Topic:	Snaps Host using Spring
Next Topic:	jdbc code in mysql

Goto Forum:

-=] Back to Top [=-

Current Time: Sat Jul 26 11:09:22 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter