Re: [ee4j-build] Unexpected Jenkins failure with "git checkout" returnin

[Scott Marlow <smarlow@xxxxxxxxxx>]

On Mon, Jan 20, 2020 at 10:50 AM Tom Jenkinson <tom.jenkinson@xxxxxxxxxx> wrote:
>
> https://github.com/moby/moby/issues/21083#issuecomment-194763283 links to http://tldp.org/LDP/abs/html/exitcodes.html. Perhaps a "kill -9" could have been executed on the git process. I guess that could have happened for a number of reasons but along the idea of memory, then perhaps the Linux Out of Memory Killer uses kill -9? If so, maybe the git command needs a lot of ram to checkout that commit (or not) but perhaps the overall environment executing that command determined it wanted to reclaim some memory or something like that?

It could be that memory needs to be reclaimed.  Around the same time
that the linked test was running, another Jenkins job was also running
that experienced a timeout from a kubernetes container that was trying
to execute (Jakarta EE 8) TCK tests, from
https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck/job/8.0/39/execution/node/552:

"
java.io.IOException: Timed out waiting for websocket connection. You
should increase the value of system property
org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator.websocketConnectionTimeout
currently set at 30 seconds
"

We got about 36 seconds into the (Compat13) TCK tests, which may mean
we loaded GlassFish, Derby DB server and possibly started tests but
likely didn't get too far.  I'm not sure if this timeout occurred at
the same time, as the above linked "exit code is 137" failure, but
perhaps it did cause the problem.  I think it would be good to
understand how the above timeout failure is handled with regard to
cleaning up the container (e.g. close the app server, DB server +
other test processes), as not properly cleaning up the container and
then reusing the same container, could explain why this problem only
occurred when I ran multiple concurrent TCK runs.  I don't see
timestamps in the job output, so it is hard to know for sure if these
activities are related to each other.

https://pastebin.com/DpF1TreK shows a copy/paste of the steps that
were executing in
https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck/job/8.0/39/execution/node/552
(failure is on the last line "Shell Script").

Some possibly related links:

https://issues.jenkins-ci.org/browse/JENKINS-56939 seems to hit the
same timeout and references
https://issues.jenkins-ci.org/browse/JENKINS-58290 to release a thread
handle sooner via Jenkins "durable-task 1.30".  So, we could check if
our Jenkins server is running
https://wiki.jenkins.io/display/JENKINS/Durable+Task+Plugin 1.3.0
(released July of 2019).

Scott


>
> On Mon, 20 Jan 2020 at 13:21, arjan tijms <arjan.tijms@xxxxxxxxx> wrote:
>>
>> Hi,
>>
>> I worked a lot with the CI but can't remember having seen 137 ever from the git command, or any other commands for that matter.
>>
>> Kind regards,
>> Arjan
>>
>> On Fri, Jan 17, 2020 at 4:44 AM Scott Marlow <smarlow@xxxxxxxxxx> wrote:
>>>
>>> I ran the same job [2] again and it didn't fail this time.  Note that
>>> the [2] job is started by [3].  I did see the same failure yesterday
>>> as well.
>>>
>>> Scott
>>>
>>> [2] https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck/job/8.0/41/console
>>> [3] https://ci.eclipse.org/jakartaee-tck/job/eftl-jakartaeetck-certification/
>>>
>>> On Thu, Jan 16, 2020 at 10:14 PM Scott Marlow <smarlow@xxxxxxxxxx> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Any insight as to what this failure [1] represents and how to resolve?
>>> >  The command being executed is "git checkout -f
>>> > c457c40af61d307b0dea0e0489fed7ab4c3e8490" and the exit code is 137,
>>> > which could mean a memory allocation issue (from googling for that
>>> > exit code) but I'm not sure.  Have other seen this on our Jenkins and
>>> > solved it?
>>> >
>>> > Scott
>>> >
>>> > [1] https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck/job/8.0/40/console
>>>
>>> _______________________________________________
>>> ee4j-build mailing list
>>> ee4j-build@xxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://www.eclipse.org/mailman/listinfo/ee4j-build
>>
>> _______________________________________________
>> ee4j-build mailing list
>> ee4j-build@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://www.eclipse.org/mailman/listinfo/ee4j-build
>
> _______________________________________________
> ee4j-build mailing list
> ee4j-build@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/ee4j-build