Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Hudson » "java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh
"java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh [message #905801] Fri, 31 August 2012 01:01 Go to next message
Brian Cavagnolo is currently offline Brian Cavagnolo
Messages: 3
Registered: August 2012
Junior Member
Hello,

I've been using this linux slave node for quite some time. But suddenly I get an OOM error when I attempt to launch the slave (see below). Notably, I have an identical slave that seems to be working just fine. Any thoughts or debugging tips?

Thanks,
Brian

[08/30/12 21:56:15] [SSH] Opening SSH connection to riyadh:22.
[08/30/12 21:56:15] [SSH] Authenticating as hudson/******.
[08/30/12 21:56:15] [SSH] Authentication successful.
[08/30/12 21:56:15] [SSH] The remote users environment is:
BASH=/bin/bash
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="2" [2]="24" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.2.24(1)-release'
DIRSTACK=()
EUID=2002
GROUPS=()
HOME=/home/hudson
HOSTNAME=riyadh
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=en_US.UTF-8
LOGNAME=hudson
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/hudson
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PIPESTATUS=([0]="0")
PPID=7066
PS4='+ '
PWD=/home/hudson
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='169.229.130.76 48992 22'
SSH_CONNECTION='169.229.130.76 48992 169.229.130.90 22'
TERM=dumb
UID=2002
USER=hudson
XDG_SESSION_COOKIE=39710d5371497900693104820000032c-1346388888.763341-1638202249
_=']'
[08/30/12 21:56:15] [SSH] Checking java version of java
[08/30/12 21:56:15] [SSH] java -version returned 1.7.0_06.
[08/30/12 21:56:15] [SSH] Starting sftp client.
[08/30/12 21:56:15] [SSH] Copying latest slave.jar...
[08/30/12 21:56:15] [SSH] Copied 236,330 bytes.
[08/30/12 21:56:15] [SSH] Starting slave process: cd '/var/hudson' && java -jar slave.jar
<===[HUDSON REMOTING CAPACITY]===>java.io.IOException: Remote call on riyadh failed
at hudson.remoting.Channel.call(Channel.java:659)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:326)
at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:383)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:203)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.OutOfMemoryError: Java heap space
[08/30/12 21:56:16] [SSH] Connection closed.
ERROR: Connection terminated
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1024)
ERROR: [08/30/12 21:56:16] slave agent was terminated
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1024)
Re: "java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh [message #905890 is a reply to message #905801] Fri, 31 August 2012 05:06 Go to previous messageGo to next message
Steve Christou is currently offline Steve Christou
Messages: 125
Registered: June 2012
Location: Milwaukee, Wisconsin
Senior Member

It depends on how you are starting up the hudson slave node. If you are launching a slave via:

JNLP:
There is an option in the Node configuration page called JVM Options you can use, and set it to -Xmx512mb or something higher.

Execution of command on the master:
Launch command option would be the place to make the modification. When you run java -jar slave.jar, you need to specify -Xmx512 as part of that argument list, so the Launch Command would look something like "ssh SERVERNAME java -Xmx512m -jar slave.jar"


/**
 * @author Steven Christou
 * @dev    Hudson-ci
 */
Re: "java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh [message #906111 is a reply to message #905890] Fri, 31 August 2012 12:31 Go to previous messageGo to next message
Brian Cavagnolo is currently offline Brian Cavagnolo
Messages: 3
Registered: August 2012
Junior Member
Hmm. An experiment reveals that the default max heap on this machine is 28GB (as reported with Runtime.getRuntime().maxMemory()). Also, I have identical slaves running this same code just fine. And this slave was working just fine up until yesterday. Anyway, I tried your proposed experiment and get the same result (BTW, I'm using ssh to launch the slave, but it also has JVM options.)

Any other thoughts?

Thanks,
Brian

[Updated on: Fri, 31 August 2012 12:35]

Report message to a moderator

Re: "java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh [message #906170 is a reply to message #906111] Fri, 31 August 2012 15:20 Go to previous messageGo to next message
Steve Christou is currently offline Steve Christou
Messages: 125
Registered: June 2012
Location: Milwaukee, Wisconsin
Senior Member

Can you follow the guide on this page: http://wiki.hudson-ci.org/display/HUDSON/I'm+getting+OutOfMemoryError and send me the heap dump from the slave machine using one of the methods they mention on there. We need a heap dump in order to find/fix the problem.

Thanks,
Steve.


/**
 * @author Steven Christou
 * @dev    Hudson-ci
 */

[Updated on: Fri, 31 August 2012 15:20]

Report message to a moderator

Re: "java.lang.OutOfMemoryError: Java heap space" while launching slave over ssh [message #906247 is a reply to message #906170] Fri, 31 August 2012 19:37 Go to previous message
Brian Cavagnolo is currently offline Brian Cavagnolo
Messages: 3
Registered: August 2012
Junior Member
Hey guys,

Thanks for the prompt help. I think the root cause was that I was running OOM on the master, not the slave. After writing my last post, hudson started failing to load the full console with OOM, and my jobs appeared to be frozen. This all may have been because I recently increased the number of slaves that we're using, and the number of concurrent jobs. The console output from these jobs gets really big (tens of megs, if you can believe that). And I may have many users viewing many different consoles at a time. Anyway, the heap was at 128M. I cranked it up to 512M and restarted. I guess this doesn't explain why it worked for several weeks. Probably just had fewer users at that time.

Anyway, thanks for your help.

Ciao,
Brian
Previous Topic:LDAP roles as parameters to scripts
Next Topic:Hudson instance stopped unexpectly
Goto Forum:
  


Current Time: Thu Jul 31 09:41:55 EDT 2014

Powered by FUDForum. Page generated in 0.01631 seconds