Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[glassfish-dev] Issues with Jenkins

Hi all, 

as you probably noticed last two weeks or even more, we are struggling with Jenkins performance, which randomly significantly decreases.

The slowdown is not so much random, as I started watching it, however at this moment I don't see any simple solution working without reducing automated quality assurance.

What we could do:

Prefered Solution
Backup Alternative 1
  • We will change the Jenkinsfile, so it will stop running expensive parts
    • Compiling asciidoc documentation
      • The most frequently killed part of the build
      • I tried to improve JVM settings, tried to reproduce issues from Jenkins, but nothing fixed that.
      • Documentation should be up to date and should pass the processing.
    • "Ant based zombie army" how I am calling that for several years
      • Obsoleted slow expensive part of the build, however covering some critical features
      • Has probably the highest score of "new bug kills"
      • Is slowly converted to the application-tests maven module, which is much faster. However our conversion tempo is slow - help is welcome!
Backup Alternative 2
  • We can temporarily configure Jenkinsfile to mark failed builds just as warnings.
  • Then reviewers could rerun builds locally and report the result, or builds can be reexecuted on Jenkins too, which is much slower.
Backup Alternative 3
  • Move from Jenkins to GitHub Actions
  • I am already refusing it, but just for the record
    • The problem is that on GHA we have just 4 thread CPU for the build and it is not suitable for integration testing. 
    • It is even slower than Jenkins CI.
    • GHA applies some fairness policy, so jobs can be suspended for a while if they "eat" too much.
Combination

As I am thinking about it now, we can probably also combine all three alternatives:

  1. Move the compilation of the documentation to GitHub Actions.
  2. Temporarily configure Jenkinsfile to mark failures as warnings.
  3. Reviewers then decide how to interpret failed steps and what to do with that.
  4. Jenkins CI admins will fix that without blocking us.

What do you think about it? Did I forgot something? Yeah, maybe I could check Jenkins plugins, if we can add some configuration ie. for monitoring the performance ...

-- 
David Matejcek | OmniFish
david.matejcek@xxxxxxxxxxx

Back to the top