Eclipse Community Forums: Virgo

Help

Home

Home » Eclipse Projects » Virgo » Shutdown issues

Show: Today's Messages :: Show Polls :: Message Navigator

Shutdown issues [message #662608]

Thu, 31 March 2011 09:39

Lorie Pisicchio

Messages: 44
Registered: October 2010

Member

Hello all
I am still facing some unexplained issues on Virgo shutdown with shutdown.sh.
The command never runs successfully.

I don't know where to look for, because I'm running a very simple instance of Virgo (V2.1.0.RELEASE), with no custom bundle deployed. The only difference is a bit of customization of the dmk.sh, additionnal bundles available in repository, and switch from spring 3.0.0 to 3.0.5.

I get the following error code by outputting System.err on the console.

[2011-03-31 11:16:17.929] System.err                             31 mars 2011 11:16:17 org.apache.coyote.http11.Http11Protocol destroy 
[2011-03-31 11:16:17.929] System.err                             INFO: Stopping Coyote HTTP/1.1 on http-8080 
[2011-03-31 11:16:17.930] System.err                             31 mars 2011 11:16:17 org.apache.coyote.http11.Http11Protocol destroy 
[2011-03-31 11:16:17.930] System.err                             INFO: Stopping Coyote HTTP/1.1 on http-8443

See the log file attached for details.

Please let me know if I can help furthermore in investigating this issue..

Best Regards,
Lorie.

Attachment: log.log
(Size: 1.38MB, Downloaded 191 times)

Report message to a moderator

Re: Shutdown issues [message #662633 is a reply to message #662608]

Thu, 31 March 2011 11:18

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

I don't think those Coyote messages are necessarily involved in the failure to shut down.

I noticed the following message in log.log during the shutdown sequence:

[2011-03-31 10:54:55.351] startup-tracker              org.eclipse.virgo.medic.eventlog.default                         KE0001I Kernel starting.

I presume log.log shows multiple attempts to start and shut down Virgo, but please confirm as I wouldn't expect Virgo to restart during the shutdown sequence.

That's all I can glean from log.log. Please could you use jps and jstack to obtain a thread stack dump after shutdown has stopped making progress (check on a CPU monitor). That will at least show us any thread waits.

Report message to a moderator

Re: Shutdown issues [message #662638 is a reply to message #662633]

Thu, 31 March 2011 11:40

Lorie Pisicchio

Messages: 44
Registered: October 2010

Member

The "Kernel Starting" line in the log systematically appears when I start the server without the clean option. I checked : no other Virgo instance is running...
As soon as I add the clean option to the command line, it doesn't show up in the log anymore.
But in both case, shutdown doesn't work.
Find attached the dump extracted with jstack.
Please let me know if I can give you more information.

Attachment: jstack.txt
(Size: 8.32KB, Downloaded 643 times)

Report message to a moderator

Re: Shutdown issues [message #662652 is a reply to message #662638]

Thu, 31 March 2011 12:48

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

Thanks for the thread dump. It shows the following non-daemon threads (which could be holding up JVM termination):

"DestroyJavaVM" prio=10 tid=0x00007fa6e812a000 nid=0x3762 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RMI Reaper" prio=10 tid=0x00000000412a3800 nid=0x376a in Object.wait() [0x00007fa6efd31000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000077682cc38> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <0x000000077682cc38> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
	at java.lang.Thread.run(Thread.java:662)

"VM Thread" prio=10 tid=0x00000000411d4800 nid=0x3765 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000041166800 nid=0x3763 runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000041168800 nid=0x3764 runnable 

"VM Periodic Task Thread" prio=10 tid=0x0000000041278000 nid=0x3773 waiting on condition

Of these the only threads which are not runnable are the DestroyJavaVM thread and the VM Periodic Task Thread, neither of which I would expect to hold up VM termination, and the RMI Reaper thread which appears to be stuck in a wait.

Another possible clue to the problem in the thread stack dump is the heap stats section:

Heap
 PSYoungGen      total 663552K, used 252258K [0x00000007d22b0000, 0x0000000800000000, 0x0000000800000000)
  eden space 576192K, 37% used [0x00000007d22b0000,0x00000007df5fc3b0,0x00000007f5560000)
  from space 87360K, 41% used [0x00000007faab0000,0x00000007fcdbc5e8,0x0000000800000000)
  to   space 87360K, 0% used [0x00000007f5560000,0x00000007f5560000,0x00000007faab0000)
 PSOldGen        total 1398144K, used 411K [0x0000000776800000, 0x00000007cbd60000, 0x00000007d22b0000)
  object space 1398144K, 0% used [0x0000000776800000,0x0000000776866db0,0x00000007cbd60000)
 PSPermGen       total 43904K, used 43813K [0x0000000766800000, 0x00000007692e0000, 0x0000000776800000)
  object space 43904K, 99% used [0x0000000766800000,0x00000007692c9538,0x00000007692e0000)

Permgen looks pretty full and I wonder if some code is trying to allocate permgen space during shutdown (Virgo wouldn't do that itself AFAIK, but some libraries might).

You might try increasing permgen by 10% to see if that helps.

However, there seems to be a general issue with the RMI Reaper thread. I'm wondering if Spring 3.0.5 or the additional bundles in the repository (if they are auto-installed as optional dependencies of Virgo, Spring, etc. during startup) are running into this. I'll ask around to see if this is a known issue with Spring 3.0.5, but that's a long shot.

So please could you try deleting the additional bundles from the repository and start Virgo with the -clean option and see if that affects the shutdown behaviour.

Report message to a moderator

Re: Shutdown issues [message #662656 is a reply to message #662652]

Thu, 31 March 2011 12:56

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

Hi Lorie. Please would you raise a bug to track this issue. Regardless of the cause, I think others are likely to hit it.

Report message to a moderator

Re: Shutdown issues [message #662657 is a reply to message #662656]

Thu, 31 March 2011 12:58

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

I consulted the Spring framework lead engineer and he is not aware of any changes in Spring post 3.0.0 which are likely to cause this problem. Doesn't definitely rule that possibility out though...

Report message to a moderator

Re: Shutdown issues [message #662666 is a reply to message #662657]

Thu, 31 March 2011 13:32

Lorie Pisicchio

Messages: 44
Registered: October 2010

Member

Thanks for your answers Glyn.
I am trying to investigate on my side on this issue. I started from a fresh Virgo distribution (2.1.0), and progressively integrate changes to get to my faulty Virgo state.
For now, I have added in the repository each bundles that are available in my old Virgo (spring 3.0.5, and all my dependencies), and the server starts and stops fine.
As soon as I have a better idea of where the problem comes from, I will open a bug.

Again : thanks a lot for your help.

Report message to a moderator

Re: Shutdown issues [message #662671 is a reply to message #662666]

Thu, 31 March 2011 13:56

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

You're welcome! It's a pleasure working with users like yourself who are so constructive in helping diagnose problems.. Smile

Report message to a moderator

Re: Shutdown issues [message #662902 is a reply to message #662671]

Fri, 01 April 2011 12:37

Lorie Pisicchio

Messages: 44
Registered: October 2010

Member

I have identified 2 distinct problematics :
- When I deploy my application plan on Virgo, the shutdown is very slow. It can take up to 3 or 4 minutes. I tried increasing MaxPermSize, but it didn't change anything.

- On the other hand, I had made some small changes in dmk.sh because we had some port opening issues with JMX, in the start block, I changed the JMX_OPTS :

        -Dcom.sun.management.jmxremote.port=$JMX_PORT \

        -Dcom.springsource.rmiregistry.port=$JMX_PORT \
        -Dcom.springsource.rmiserver.port=$JMX_PORT \
        -Djava.rmi.server.hostname=$JMX_HOST_NAME \

(JMX_HOST_NAME declared in the script)
And I added one argument to the command line :

        -javaagent:$KERNEL_HOME/lib/jmx-agent-1.0.2.jar \

(jmx-agent-1.0.2.jar is available in the lib directory).

With those changes, shutdown command never (or at least during the 15 minutes I left Virgo process alive) succeeds.
(Jstack attached.)

Attachment: jstack.txt
(Size: 6.66KB, Downloaded 322 times)

Report message to a moderator

Re: Shutdown issues [message #662910 is a reply to message #662902]

Fri, 01 April 2011 12:59

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

This is going to get confusing if we are not careful as there are now two lines of investigation. I suggest you start a new forum thread for the slow shutdown case and keep this thread for the RMI Reaper issue.

In the slow shutdown case, I'm not sure why deploying your application would make such a difference unless the application is somehow blocking termination. Perhaps you could take a thread dump 2 minutes into termination?

In the non-terminating case with your dmk.sh changes JVM termination is again prevented by the RMI Reaper thread. Did you make any other changes compared to a fresh Virgo install such as upgrading Spring, adding bundles to the repository, or deploying applications?

Report message to a moderator

Re: Shutdown issues [message #662912 is a reply to message #662910]

Fri, 01 April 2011 13:14

Lorie Pisicchio

Messages: 44
Registered: October 2010

Member

OK I will open a new topic for the slow shutdown.
In the non-terminating case : I have other modifications on Virgo instances, but the shutdown issue can be reproduced as soon as I copy my dmk.sh, and add the jmx-agent-1.0.2.jar to the lib directory.

Attached : dmk.sh

Attachment: dmk.sh
(Size: 5.50KB, Downloaded 178 times)

Report message to a moderator

Re: Shutdown issues [message #662969 is a reply to message #662912]

Fri, 01 April 2011 15:53

Glyn Normington

Messages: 1222
Registered: July 2009

Senior Member

I'm afraid I'm not familiar with jmx-agent-1.0.2.jar, so it's hard to know how that might be affecting Virgo. Also, I can't vouch for the changes you made to dmk.sh without doing a lot more research.

This feels like it is unlikely to be a Virgo-specific problem. You might try running some kind of simple "hello world" Java program with the same agent and set up. I found a bug in Java by doing that the other day.

Or you might like to raise an enhancement request to cover the use case that led you into changing dmk.sh and adding the agent in the first place.

Another option, of course, is to purchase commercial support for Virgo so that someone can spend more time on your issue.

Hoep that gives you some ideas.

Report message to a moderator

Previous Topic:	Loading ActiveMQ Spring client on Virgo
Next Topic:	Tomcat export package error

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 19 22:33:38 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter