| Re: [mdt-papyrus.dev] Gerrit builds time out |
On Wed, Dec 2, 2015 at 9:58 AM, Christian W. Damus <give.a.damus@xxxxxxxxx> wrote:
Yeah, I’m going to have a look at that breaking CDO API change now (it’s in a feature I implemented in Kepler).cWOn Wed, Dec 2, 2015 at 9:57 AM, LETAVERNIER Camille <Camille.LETAVERNIER@xxxxxx> wrote:
Hi all,
The same patch has been applied to the Mars/Maintenance branch. Builds shouldn’t time out anymore
To benefit from this patch on your own feature/branch build, you need to set the following property on Hudson:
eclipse.download=file:/home/data/httpd/download.eclipse.org
This property defaults to http://download.eclipse.org when not explicitly set, so this change should be fully transparent to all builds (i.e. builds won’t fail if you don’t change anything).
Gerrit builds on master still fail, but on a different issue: I’ve updated the POM Dependencies to the latest versions, and it seems that the CDO Extra-plugin doesn’t compile anymore on Neon (With CDO M3)
Regards,
Camille
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] De la part de LETAVERNIER Camille
Envoyé : mercredi 2 décembre 2015 12:13
À : Papyrus Project list <mdt-papyrus.dev@xxxxxxxxxxx>
Objet : [PROVENANCE INTERNET] Re: [mdt-papyrus.dev] Gerrit builds time out
Hi,
Both workarounds have been applied:
- The Gerrit job doesn’t clean plug-ins from repo.eclipse and repo.maven before each new build anymore, which reduces the need for downloading anything there
- The Tycho build on Master has been updated to use a variable referencing the Eclipse Download server
o By default, it is set to http://download.eclipse.org,
o Hudson Master jobs (+ Gerrit) override it to file:/.../download.eclipse.org to bypass the HTTP protocol (and proxies)
o The POM Updater scripts have been updated accordingly (And POM dependencies have been updated as well, which might cause build errors eventually)
The build seems much faster now (Not much faster than before, but at least faster than the past couple of days)
Eclipse Webmasters are still investing the actual Proxy issue (Bug 483118)
483118: Contacting build.eclipse.org during builds seems very slow (times out?)
https://bugs.eclipse.org/bugs/show_bug.cgi?id=483118
Regards,
Camille
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] De la part de LETAVERNIER Camille
Envoyé : mardi 1 décembre 2015 15:28
À : Papyrus Project list <mdt-papyrus.dev@xxxxxxxxxxx>
Objet : [PROVENANCE INTERNET] Re: [mdt-papyrus.dev] Gerrit builds time out
And indeed, after 25 minutes, the build now has started to download dependencies from download.eclipse.org, and it’s just as slow
I’ve opened Bug 483381:
483381: Very slow access from HIPP to repo.eclipse.org and download.eclipse.org
https://bugs.eclipse.org/bugs/show_bug.cgi?id=483381
Regards,
Camille
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] De la part de LETAVERNIER Camille
Envoyé : mardi 1 décembre 2015 15:06
À : Papyrus Project list <mdt-papyrus.dev@xxxxxxxxxxx>
Objet : [PROVENANCE INTERNET] Re: [mdt-papyrus.dev] Gerrit builds time out
Hi,
It seems to be the HTTPs access to repo.eclipse.org that is very slow
I can workaround that, but this is probably an issue on the Eclipse Servers side. Anyway, reducing the number of calls to repo.eclipse.org (and repo.maven.apache.org) can only be beneficial.
Direct access to Eclipse Update Sites via file:/ rather than http:/ could help too, for the dependencies, but it is a little bit more complex to set up (We need to add a variable in all POM Dependencies & job configurations)
Tycho always shows 0kb/s when the download speed is too fast for it to provide an accurate estimation. So that’s actually a good sign :)
After 20 minutes, I still didn’t complete the step to download all build plug-ins. I haven’t even started to download actual Papyrus dependencies. This used to take only a few minutes (And can be reduced to a few seconds by not re-downloading them each time)
Regards,
Camille
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] De la part de Johan Hardy
Envoyé : mardi 1 décembre 2015 15:04
À : Papyrus Project list <mdt-papyrus.dev@xxxxxxxxxxx>
Objet : Re: [mdt-papyrus.dev] Gerrit builds time out
Hi,
Confirmed Christian, I experienced the same problem this monring but on another project.
Johan
From: mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] On Behalf Of Christian W. Damus
Sent: mardi 1 décembre 2015 15:01
To: Papyrus Project list
Cc: Papyrus Project list
Subject: Re: [mdt-papyrus.dev] Gerrit builds time out
Thanks, Camille,
I’m glad you’re looking into it. The JAR downloads in my builds’ console logs all show 0 kB/s, and they tick by very slowly, which suggests that it’s setting up the initial HTTP connection that takes time and the actual transfer then is so fast that Maven doesn’t report any progress intervals (with non-zero kB/s) on them. But, I’m just guessing now.
I don’t think dependency resolution is any slower than before; it really looks to me like the dependency downloads are the bottleneck.
Cheers,
Christian
On Tue, Dec 1, 2015 at 8:51 AM, LETAVERNIER Camille <Camille.LETAVERNIER@xxxxxx> wrote:
Hi Christian,
I see green builds from yesterday on Gerrit. And now it takes more than an hour to simply “resolve & download dependencies” (Without compiling anything)
So there’s definitely something wrong. It doesn’t seem to be the actual download speed, as it takes less than 1 second to download each artifact, so I suspect the “Resolve dependencies” step is slower than usual. This might be a simple server overload issue, or there might be something else.
AFAIK, Papyrus used to be the only project on our HIPP instance a few months ago, but this doesn’t seem to be the case anymore, as I frequently see the server quite busy even when no Papyrus jobs are running
I can tweak the Gerrit Job configuration a little bit to improve performances (Similar to what I did for the Extra & Tests jobs), but that would only save a few minutes.
Regarding the download speeds for download.eclipse.org, we are currently referencing our dependency update sites via HTTP. We could improve this by using the local filesystem instead, for all update sites located under http://download.eclipse.org. This would also improve the performances, but once again, I don’t believe that download speeds are the source of the issue here.
I’m running a dummy Gerrit job to see more precisely what actually takes time
Regards,
Camille
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx] De la part de Christian W. Damus
Envoyé : mardi 1 décembre 2015 14:02
À : Papyrus Project list <mdt-papyrus.dev@xxxxxxxxxxx>
Objet : [mdt-papyrus.dev] Gerrit builds time out
Hi, team,
It appears that the Gerrit builds are now routinely timing out. Looking at the glacial pace of the console’s progress in my browser, it appears that fetching JARs from the download server is bizarrely slow. Don’t these downloads just all route locally in Eclipse Foundation’s network, at presumably blazing speeds? Is there something in our Hudson instance making this so slow, that we should re-start it? Or maybe this is going to be normal, and we need to increase our build time-out setting?
Any advice or other information will be appreciated.
Thanks,
Christian
------------------------------------------------------------------------------
E-MAIL DISCLAIMER
The present message may contain confidential and/or legally privileged information. If you are not the intended addressee and in case of a transmission error, please notify the sender immediately and destroy this E-mail. Disclosure, reproduction or distribution of this document and its possible attachments is strictly forbidden.
SPACEBEL denies all liability for incomplete, improper, inaccurate, intercepted, (partly) destroyed, lost and/or belated transmission of the current information given that unencrypted electronic transmission cannot currently be guaranteed to be secure or error free.
Upon request or in conformity with formal, contractual agreements, an originally signed hard copy will be sent to you to confirm the information contained in this E-mail.
SPACEBEL denies all liability where E-mail is used for private use.
SPACEBEL cannot be held responsible for possible viruses that might corrupt this message and/or your computer system.
-------------------------------------------------------------------------------