Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Hudson » Java at 100% CPU(How to approach/investigate)
Java at 100% CPU [message #1059205] Fri, 17 May 2013 09:58 Go to next message
Maarten van de Waarsenburg is currently offline Maarten van de WaarsenburgFriend
Messages: 4
Registered: May 2013
Junior Member
After upgrading Hudson from version 2.2.1 to 3.0.1-1b, he (Hudson) becomes unresponsive after an hour. When that happens, the Hudson Java process runs at 100%.

It is likely to have something to do with our converted configuration (jobs and slaves) as I have Hudson running on a test machine for a over month and it runs fine.

Some facts:
- Hudson runs on Red Hat Enterprise Linux 5 (up-to-date)
- Virtual machine 2 processors, 4 GB memory
- JRE 1.6.0_26
- Hudson Server # of executors is 0
- 20 slaves
- 72 jobs

Initially I only replaced the hudson.war and performed the upgrade. When I ran into problems, I unmounted my hudson disk on /var/lib/hudson and performed an uninstall/install cycle using the RPM. After that I restored the /etc/sysconfig/hudson file and mounted my hudson disk on /var/lib/hudson again. So eventually I have a clean Hudson installation with a converted /var/lib/hudson configuration.

How to approach this problem? I do not have any particular Java knowledge but I am reasonably familiar with Linux.
Re: Java at 100% CPU [message #1059349 is a reply to message #1059205] Fri, 17 May 2013 15:22 Go to previous messageGo to next message
Winston Prakash is currently online Winston PrakashFriend
Messages: 448
Registered: August 2011
Location: Fremont, CA USA
Senior Member
When you upgrade did you install all the plugin updates and the compatibility plugin suggested by the initial setup. Also make sure you have version 1.0.2 of the xpath-provider plugin.

Since you have 4 GB memory in the Linux VM, you could set aside 2 GB for Hudson Java process in the /etc/sysconfig/hudson/hudson file

Change

HUDSON_JAVA_OPTIONS="-Djava.awt.headless=true"


to

HUDSON_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx2048 -XX:MaxPermSize=196m"


Also look at Hudson Management -> System logs and see if any error reported.

BTW, Java knowledge is not required to run Hudson, except for tweaking memory as I mentioned above or if you want to write Hudson plugin.


Winston Prakash
Eclipse Hudson team
Re: Java at 100% CPU [message #1059744 is a reply to message #1059349] Tue, 21 May 2013 13:27 Go to previous messageGo to next message
Maarten van de Waarsenburg is currently offline Maarten van de WaarsenburgFriend
Messages: 4
Registered: May 2013
Junior Member
Hi Winston,

Thanks for your reply. Excuse me for my late response, all companies in The Netherlands were closed on Monday because of a common holiday.

All plugins are up-to-date, including the XPath Provider Plugin (1.0.2).

I've added the -Xmx2048 option to the hudson sysconfig file. It then takes longer before the CPU time of java climbs to 100% but eventually it will be 100%. At least, that is what I experience up till now. I didn't try the -XX:MaxPermSize=196m option yet but I will add it the next time I have to kill and start Hudson again.

I do not see any unexpected errors or warnings in the log.

Regards,
Maarten.
Re: Java at 100% CPU [message #1060019 is a reply to message #1059744] Wed, 22 May 2013 14:36 Go to previous messageGo to next message
Maarten van de Waarsenburg is currently offline Maarten van de WaarsenburgFriend
Messages: 4
Registered: May 2013
Junior Member
==SOLVED==

It appears to be an old slave.jar on one of the nodes. It was accidentally skipped when updating all slaves.

After upgrading this slave and restarting Hudson this morning, Hudson has been running continuously.

[Updated on: Thu, 23 May 2013 06:07]

Report message to a moderator

Re: Java at 100% CPU [message #1060044 is a reply to message #1060019] Wed, 22 May 2013 15:51 Go to previous messageGo to next message
Winston Prakash is currently online Winston PrakashFriend
Messages: 448
Registered: August 2011
Location: Fremont, CA USA
Senior Member
Hi Maarten, Glad to see it is resolved for you.

Winston Prakash
Eclipse Hudson team
Re: Java at 100% CPU [message #1060105 is a reply to message #1060044] Thu, 23 May 2013 07:02 Go to previous messageGo to next message
k z is currently offline k zFriend
Messages: 71
Registered: October 2012
Member
How could slave.jar be skipped?

From the slave log file I can see that slave.jar is downloaded every time new connection between master and slave is established (SSH).

[05/22/13 15:16:37] [SSH] Checking java version of java
[05/22/13 15:16:39] [SSH] java -version returned 1.6.0_20.
[05/22/13 15:16:39] [SSH] Starting sftp client.
[05/22/13 15:16:39] [SSH] Copying latest slave.jar...
[05/22/13 15:16:39] [SSH] Copied 269,117 bytes.


Or you used JNLP to set up a slave - master connection?

Regards!
Re: Java at 100% CPU [message #1060109 is a reply to message #1060105] Thu, 23 May 2013 07:17 Go to previous messageGo to next message
Maarten van de Waarsenburg is currently offline Maarten van de WaarsenburgFriend
Messages: 4
Registered: May 2013
Junior Member
To k z:

That's right, for the Windows nodes I use JNLP. In the past this worked best for me. Maybe it's time to try a master initiated connection mechanism.

The other nodes (Linux) were updated automatically.

Regards.
Re: Java at 100% CPU [message #1060112 is a reply to message #1060109] Thu, 23 May 2013 07:36 Go to previous messageGo to next message
k z is currently offline k zFriend
Messages: 71
Registered: October 2012
Member
Im using SSH for all platforms. On Windows I have to addionally install SSH client since Windows do not go with any. It's really pain in the neck but allows automatic CI slave process startup after system reboot and does not require interactive user session (unlike JNLP).Hudson team should really fix 'slave as a windows service' problem Sad
Re: Java at 100% CPU [message #1060230 is a reply to message #1060112] Thu, 23 May 2013 15:51 Go to previous messageGo to next message
Winston Prakash is currently online Winston PrakashFriend
Messages: 448
Registered: August 2011
Location: Fremont, CA USA
Senior Member
K.Z, I have fixed the windows as slave problem and put an interim release as discussed in this thread.

http://www.eclipse.org/forums/index.php/t/457969/

Did you get a chance to test it?


Winston Prakash
Eclipse Hudson team
Re: Java at 100% CPU [message #1060913 is a reply to message #1060230] Wed, 29 May 2013 07:02 Go to previous message
k z is currently offline k zFriend
Messages: 71
Registered: October 2012
Member
I've just tiried that. I'm using role-based authorization strategy and active directory plugins for security. The service installed itself successfully but there is a problem with connection.

If I do not specify credentials I got 403 exception
Failing to obtain https://<MyTestHudsonURL>/computer/nemedia/slave-agent.jnlp
java.io.IOException: Failed to load https://<MyTestHudsonURL>/computer/nemedia/slave-agent.jnlp: 403 Forbidden
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:222)
	at hudson.remoting.Launcher.run(Launcher.java:184)
	at hudson.remoting.Launcher.main(Launcher.java:160)
Waiting 10 seconds before retry


If I specify credentials (-jnlpCredentials <user>:<password>) I got NoClassDefFoundError exception
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:213)
	at hudson.remoting.Launcher.run(Launcher.java:184)
	at hudson.remoting.Launcher.main(Launcher.java:160)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.codec.binary.Base64
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	... 3 more


The only way I was able to get slave connected to master was to either grant Anonymous user Read permission (which is not acceptable in production env) or download slave-agent.jnlp to my file system and point out to that file in config. In both cases exception was thrown but the connection was successful.
May 29, 2013 8:37:54 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
May 29, 2013 8:37:54 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://<MyTestHudsonURL>/, https://<MyTestHudsonURL>:443/]
May 29, 2013 8:37:55 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <MyTestHudsonURL>:51996
May 29, 2013 8:37:55 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
May 29, 2013 8:37:56 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
May 29, 2013 8:37:58 AM org.eclipse.hudson.jna.NativeUtils <init>
INFO: Error getting Native Support Extensions
java.lang.NullPointerException
	at org.eclipse.hudson.jna.NativeUnixSupport.all(NativeUnixSupport.java:46)
	at org.eclipse.hudson.jna.NativeUtils.getAvailableNativeUnixSupports(NativeUtils.java:74)
	at org.eclipse.hudson.jna.NativeUtils.<init>(NativeUtils.java:48)
	at org.eclipse.hudson.jna.NativeUtils.<clinit>(NativeUtils.java:44)
	at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
	at java.io.ObjectStreamClass.computeDefaultSUID(Unknown Source)
	at java.io.ObjectStreamClass.access$100(Unknown Source)
	at java.io.ObjectStreamClass$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.io.ObjectStreamClass.getSerialVersionUID(Unknown Source)
	at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
	at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
	at java.io.ObjectInputStream.readClassDesc(Unknown Source)
	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
	at java.io.ObjectInputStream.readSerialData(Unknown Source)
	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.readObject(Unknown Source)
	at hudson.remoting.UserRequest.deserialize(UserRequest.java:171)
	at hudson.remoting.UserRequest.perform(UserRequest.java:91)
	at hudson.remoting.UserRequest.perform(UserRequest.java:41)
	at hudson.remoting.Request$2.run(Request.java:276)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:53)
	at java.lang.Thread.run(Unknown Source)


Are you planning to integrate slave windows service somehow better with hudson security model or I should use workaround with local slave-agent.jnlp file ?

[Updated on: Wed, 29 May 2013 07:03]

Report message to a moderator

Previous Topic:HTML Publisher 1.1 error
Next Topic:reviewboard plugin
Goto Forum:
  


Current Time: Thu Dec 18 17:27:33 GMT 2014

Powered by FUDForum. Page generated in 0.01883 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software