Home » Eclipse Projects » Oomph » Using bundle pools in eclipse installer simple mode(Why does it try to retrieve plugins from the internet instead of using the bundle pool)
| | | | | |
Re: Using bundle pools in eclipse installer simple mode [message #1780889 is a reply to message #1780783] |
Mon, 29 January 2018 15:47 |
Mario Jauvin Messages: 94 Registered: October 2015 |
Member |
|
|
See my comments below:
Ed Merks wrote on Fri, 26 January 2018 23:44The ~/.p2/org.eclipse.equinox.p2.repository cache folder avoids repeatedly downloading the large content.xml/jar and artifacts.xml/jar from update sites, when their timestamp hasn't changed since the last time it was read. But p2 doesn't do such a good job of caching all the things that need to be cached and you can't tell what exactly it has cached because of the horrible hash codes. Oomph supports an offline mode where you can reproduce an p2 resolution or Targlet resolution that worked once online. It augments the p2 cache via ~/.eclipse/org.eclipse.oomph.p2/cache and here you can really see what is cached and where it came from. Problems like "java.net.UnknownHostException: public.dhe.ibm.com" is fundamentally a network problem that we can do nothing about.
- How is the offline mode augmented oomph cache populated and utilized?
- Does the population mode work in both simple and advanced mode?
- Can that cache be utilized both in simple and advanced mode?
- How can I avoid the "java.net.UnknownHostException: public.dhe.ibm.com" error when eclipse-installer is running behind a firewall with no proxy configured?
Ed Merks wrote on Fri, 26 January 2018 23:44
Oomph also maintains a cache of setups (of external resources) that it downloads in ~/.eclipse/org.eclipse.oomph.setup/cache. This is also to support offline mode and to reduce the extent to which the same file is downloaded multiple times unnecessarily.
An index archive is always loaded before loading the index itself (in org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.handleArchiveRedirection(URIConverter)). The default location is http://www.eclipse.org/setups/setups.zip but this location to maybe redirected. The index is produced by the IApplication org.eclipse.oomph.setup.internal.core.SetupArchiver which runs on an Eclipse host machine every five minutes, walking the entire set of reachable resources and including them in the zip file; these induce implicit redirections at runtime so that no one ever directly accesses the URIs reachable by the Index org.eclipse.setup. This avoids, for example, http://git.eclipse.org/c/* from being hammered to death, and improves performance because the whole index can be loaded by downloading this single file. Like all resources downloaded by org.eclipse.oomph.setup.internal.core.util.ECFURIHandlerImpl, this is cached in ~/.eclipse/org.eclipse.oomph.setup/cache .
I am redirecting the index to a file url. Are you indicating that I also must redirect the index archive? Why do I need the archive AND the index. I would think the archive would suffice.Ed Merks wrote on Fri, 26 January 2018 23:44
Note that the installer can manage multiple indexes so you can for example drag and drop a setups.zip as produced by SetupArchiver to make the installer switch to a different index, and it remembers them so the user can switch between difference indices.
What is the point of multiple indexes? I understand only one is used.
|
|
| | |
Re: Using bundle pools in eclipse installer simple mode [message #1780920 is a reply to message #1780903] |
Tue, 30 January 2018 04:52 |
Ed Merks Messages: 33217 Registered: July 2009 |
Senior Member |
|
|
I see.
1. As a user you don't need to do anything. The underlying technology uses so-called transport layers, and they do caching as a side effect of downloading whatever is needed, utilizing the cache avoid avoid transport. So p2 has its own cache, but its opaque, so you can't tell what is in the cache from the file names without looking into each file, and its somewhat incomplete, so there is not sufficient information present that one could do the same p2 task twice without internet access. Oomph's p2 extension builds a layer on top, using file names that are an encoded version of the full URI that was accessed with complete information such that an offline mode can be supported. Oomph also needs access to setup models. Those are transported by a different layer and into a different cache. In all cases, the cache is populated in a demand driven way based on which p2 update sites are accessed and which setup models are loaded.
To test the installer, I typically launch it with -Duser.home=D:\sandbox\USER-HOME and -Doomph.setup.user.home.redirect=true so that it thinks it's in a different user home than my real user home while running, and sets -Duser.home=D:\sandbox\USER-HOME into the installation it creates that when it launches, it thinks it's in that same fake user home. Then one can see easily what and where things are all populated into the fresh new user home.
But the fundamental answer to 1 is that you don't do have to do anything.
2. Again, you don't need to do anything. The transport layer generally looks at what's being downloaded and what's in the cache, doing time stamp comparison for the two p2 caches and etag checking for the setup cache. If the cache is fresh, it's used. If not, the thing is downloaded and cached new. If the internet is inaccessible, the cache can be used as a fallback. The same mechanisms are used regardless of the mode. And in fact in the installations that are created, the same mechanisms is also used, even for "Help -> Install New Software".
3. Three things happen when the installer starts. 1) It looks at its self update [p2 site, -Doomph.installer.update.url=... to check for updates. 2) It loads the setup archive http://www.eclipse.org/setups/setups.zip (can be redirected with -Doomph.redirection.<an-arbitrary-name>) using that to populate implicit redirections. 3) It looks for the Index in index:/org.eclipse.setup. By default that's redirected as index:/->http://git.eclipse.org/c/oomph/org.eclipse.oomph.git/plain/setups/, i.e., a folder direction to where all the setups are located in the plain text web view of Oomph's git repository. The setup archive that was already loaded in turn redirects all those URIs into the setup archive. So to work completely offline, you need to redirect those to disk. When the installation is created, all the p2 update sites referenced by all the p2 tasks need to be inside the firewall or on disk.
The option -Doomph.setup.ecf.trace is useful for tracing the internet access of to setup models. If you've properly redirected all setup locations, there should be no trace. If you run under debug control, you can monitor what happens in org.eclipse.oomph.p2.internal.core.CachingTransport, or set org.eclipse.oomph.p2.internal.core.CachingTransport.DEBUG to true; no -D option for that, but I suppose that would be a nice to have.
I think provisioning an Oomph development environment to launch the installer in debug mode (with user redirections as I described above) is likely to be helpful. Both p2 and Oomph use ECF for the transport layers, which in turn uses Apache HTTP libraries, so it's likely to provide a ways to discover answers and because it can populate a clean user home that you can completely delete, you'll get a good idea of what's happening on disk.
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
| | |
Re: Using bundle pools in eclipse installer simple mode [message #1781414 is a reply to message #1781393] |
Wed, 07 February 2018 05:22 |
Ed Merks Messages: 33217 Registered: July 2009 |
Senior Member |
|
|
It's important to note (as I mentioned above) that all resource sets for Setup models are initialized by org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.configureResourceSet(ResourceSet) which (except for the ArchiverApplication), always calls org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.handleArchiveRedirection(URIConverter) which always loads http://www.eclipse.org/setups/setups.zip (or whatever that is redirected to). This is done as a last step of configuring the resource set and it's done after the call to org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.configureRedirections(Map<URI, URI>) which configures the redirections specified by -Doomph.redirection.*.
Why? What is the purpose of that? The archive name structure of the zip entries is such that the name of each zip entry can be used to reconstruct the URI that the zip entry represents. The handleArchiveRedirection method uses this to induce additional redirections, i.e., the URI of each zip entry induces a redirection from that URI to the URI of the zip entry itself, but only if that URI isn't already redirected to something else by the configureRedirections previous setup. So, as Eike mentions, this is a huge optimization because all setup models can be loaded from the archive because all reachable resources have been stored in the archive.
But now suppose we redirect (with -Doomph.redirection.*) the index, or the folder of the index. The setup archive is still loaded, but anything in the archive that has already been redirected elsewhere will not be redirected to the archive. This is why you can redirect the index to something else. But note that the other redirections are still in effect. So if you copy the stuff in the Oomph setups folder, change it, and redirect to that changed version, that works nicely, and most importantly, all the other things in the archive that you've not redirected still redirect to the archive. So you don't have to copy all reachable resources to use them in your index. If you reuse nothing from the index we provide, you could redirect the archive to archive containing only the setups/models folder, or probably better, to an archive built from your index so that you have the same nice optimization of loading the entire set of reachable resources in a single http request. Note in particular that the setups/models folder is super important for dynamically loading models whose implementing plugins are not actually installed. These models can change with each commit we make in Oomph. So if you redirect the archive itself, you must maintain an update-to-date version of this folder in your archive.
So the best thing to have 100% control over what is loaded and from where it's loaded, you build your own index folder with your own stuff, use the SetupArchiver application to build archive for it, and then also redirect http://www.eclipse.org/setups/setups.zip to your archive's location. Alternatively, make a copy of http://www.eclipse.org/setups/setups.zip periodically and keep it at a different location to which you redirect.
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
Goto Forum:
Current Time: Tue Sep 24 09:31:23 GMT 2024
Powered by FUDForum. Page generated in 0.09260 seconds
|