Re: [tycho-user] Mirror p2 artifacts from zipped archive deployed as a M

["Sievers, Jan" <jan.sievers@xxxxxxx>]

no it's not a crazy idea I think.
there is an existing enhancement request for this
https://bugs.eclipse.org/bugs/show_bug.cgi?id=398238


would be interesting to have some measurements that would show the performance benefit before investing time though.
p2 downloads should be multithreaded (I think 4 threads by default, see [1]), and it only downloads those bundles actually needed so in theory it should not be slower than dowloading the whole zipped repo upfront.

You need to be careful what you compare here because p2 will:

- download the artifacts/component.jar/xml index metadata
- parse it and build up the dependency model in-memory
- resolve the given dependency constraints
- download required artifacts according to dependency resolution

there can be considerable time spent on the client in the above steps. 

> Additionally, due to the
> large number of network connections involved (I guess) we're also exposed to network irregularities, resulting in occational 
> SocketTimeoutExceptions whilst mirroring p2 artifacts.  

I am not sure this is a function of your particular network environment.

Enabling HTTP trace [2] may give you some more insights.

Regardless of performance it's probably a good idea to have an alternative solution that does not require any server-side logic for people that don't/can't use nexus unzip plugin or the artifactory solution you described.


<repository>
  <id>id</id>
  <url>mvn://groupId/artifactId/version</url>
  <layout>p2-maven</layout>
</repository>

maybe we don't even need a separate layout and url schema. just handling p2 repo urls that end with .zip by downloading them may also be enough for a first proof of concept.

Regards
Jan



[1] https://wiki.eclipse.org/Equinox/p2/HiddenOptions
[2] https://wiki.eclipse.org/Tycho/FAQ#How_to_switch_on_HTTP_tracing_during_p2_dependency_download.3F 

On 03.05.17, 18:07, "tycho-user-bounces@xxxxxxxxxxx on behalf of Stewart Francis" <tycho-user-bounces@xxxxxxxxxxx on behalf of stewartfrancis@xxxxxxxxxx> wrote:

Hi,
 
I'm working for an IBM team using Tycho to build a variety of different Eclipse plugin and RCP based products.  Our products have interdependencies, we're currently controlling compatible versions of those dependencies by using a corporate master
 pom.xml which has the versions set as properties.  We're using Artifactory as our artifact repository, and publishing entire eclipse-repositories, rather than individual plugins and features, which are excluded.  We take advantage of Artifactory's ability
 to reference files within zip archives to reference those zip files as p2 repositories in our maven builds with the support that Tycho provides for p2 repository layout.  We build our target platform by composing combinations of p2 repositories, which is working
 really well for us.
 
However, a significant portion of our build time is spent mirroring p2 artifacts.  My working assumption is that this is because p2 mirroring is single threaded, and negotiating the transfer of each file adds time too.  Additionally, due to the
 large number of network connections involved (I guess) we're also exposed to network irregularities, resulting in occational SocketTimeoutExceptions whilst mirroring p2 artifacts.  I've not dug into these too far.  We're running our builds in clean Docker
 environments, so we're hitting these kind of issues in most of our builds.  As a result of some experimentation, I've found it's significantly faster for us to use Maven's dependency resolution mechanism to mirror entire p2 repositories, rather than p2's mirroring
 support to mirror individual artifacts.
 
I wanted to ask your opinions on whether it'd be possible to extend Tycho's capability for p2 repository layouts with support for zipped p2 repositories, identified by Maven co-ordinates?  Is that sort of thing sensible / feasible?  I've some
 experience with developing Maven plugins (though nothing as sophisticated as Tycho!), and have had a look at the Tycho source, and it seems like it might be feasible, but could potentially have wide-reaching consequences, depending on implementation.  I'm
 happy to have a go at prototyping it, if it's not a crazy idea.  I was thinking something like:
 
<repository>
  <id>id</id>
  <url>mvn://groupId/artifactId/version</url>
  <layout>p2-maven</layout>
</repository>
 
I'd be interested to hear your thoughts.
 
Thanks,
 
Stew Francis


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU