Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Oomph » Using bundle pools in eclipse installer simple mode(Why does it try to retrieve plugins from the internet instead of using the bundle pool)
Using bundle pools in eclipse installer simple mode [message #1780687] Thu, 25 January 2018 17:45 Go to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
When I run the installer in simple mode, it tries to retrieve some (non eclipse) plugins from the internet even though the bundle pool contains the plugins. When I run in advanced mode it does not attempt to retrieve them from the internet.

I know this because I disable proxy settings and I am running behind a firewall.
Re: Using bundle pools in eclipse installer simple mode [message #1780711 is a reply to message #1780687] Fri, 26 January 2018 08:55 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 29552
Registered: July 2009
Senior Member
I'm not sure about the specific details, but the installer does check for self updates. Where it checks can be controlled via the system property oomph.update.url and that property is generally set in the eclipse-inst.ini. The default, when no system property is specified, is http://download.eclipse.org/oomph/updates/milestone/latest and the installer itself doesn't use the shared bundle pool for its "self" installation but rather places the artifact in it's own local bundle pool nested in the installer's installation folder. Perhaps that's what you're seeing?
Re: Using bundle pools in eclipse installer simple mode [message #1780768 is a reply to message #1780711] Fri, 26 January 2018 18:53 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
Here is what I did:


  1. deleted the cache folder under the .p2\org.eclipse.equinox.p2.repository folder
  2. ran once in advanced mode
  3. switched to simple mode


This is the error that then occurs in simple mode:

[2018-01-26 12:41:13] Repository http://download.eclipse.org/technology/epp/packages/oxygen
[2018-01-26 12:41:13] Repository http://download.eclipse.org/releases/oxygen/201712201001
[2018-01-26 12:41:13] Repository http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/updates/wdt/oxygen_comp/
[2018-01-26 12:41:13] Repository https://dl.bintray.com/subclipse/releases/subclipse/4.2.x/
[2018-01-26 12:41:13] ERROR: org.eclipse.equinox.p2.transport.ecf code=1006 Unknown Host: http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/updates/wdt/oxygen_comp/compositeContent.xml
java.net.UnknownHostException: public.dhe.ibm.com
  at java.net.InetAddress.getAllByName0(Unknown Source)
  at java.net.InetAddress.getAllByName(Unknown Source)
  at java.net.InetAddress.getAllByName(Unknown Source)
  at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
  at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:262)
  at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:161)
  at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
  at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
  at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
  at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:263)
  at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
  at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)



What is also strange is before I deleted the cache folder under the .p2\org.eclipse.equinox.p2.repository folder the same error used to occur with the dl.bintray.com URL instead.

A side question is what is the cache folder? I understand that the .p2/pool folder is a cache of the resources from all of the update sites where oomph downloaded from. What I do not understand is what is the cache folder in the org.eclipse.equinox.p2.repository folder is for.
Re: Using bundle pools in eclipse installer simple mode [message #1780769 is a reply to message #1780768] Fri, 26 January 2018 19:04 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
I am also puzzled by the following observation. When I start the eclipse installer using the -consoleLog and -Dosgi.debug set to a file that has most tracing set to true, I observe the following:
> ECF: http://www.eclipse.org/setups/setups.zip uri=http://www.eclipse.org/setups/setups.zip
> ECF: http://www.eclipse.org/setups/setups.zip cacheURI=file:/C:/Users/zzzzz/.eclipse/org.eclipse.oomph.setup/cache/http___www.eclipse.org_setups_setups.zip
> ECF: http://www.eclipse.org/setups/setups.zip eTag="8d0d6-563af18e277b8"
> ECF: http://www.eclipse.org/setups/setups.zip expectedETag=null
> ECF: http://www.eclipse.org/setups/setups.zip authorizationHandler=org.eclipse.oomph.setup.internal.core.util.ECFURIHandlerImpl$AuthorizationHandlerImpl@f6c03cb authorizations: {} securePreferences: org.eclipse.oomph.preferences.util.PreferencesUtil$AutoRefreshSecurePreferencesWrapper@bc2b4f7c uiServices: org.eclipse.oomph.p2.internal.ui.P2ServiceUI@18518ccf
> ECF: http://www.eclipse.org/setups/setups.zip trying=0
> ECF: http://www.eclipse.org/setups/setups.zip triedReauthorization=0
> ECF: http://www.eclipse.org/setups/setups.zip authorization=Authorization [user=, password=]
> ECF: http://www.eclipse.org/setups/setups.zip triedProxyReauthorization=0
> ECF: http://www.eclipse.org/setups/setups.zip proxyAuthorization=null
> ECF: http://www.eclipse.org/setups/setups.zip transferLister.exception

What is the cache at that location: file:/C:/Users/zzzzz/.eclipse/org.eclipse.oomph.setup/cache/http___www.eclipse.org_setups_setups.zip? This seems like a third cache location and the .eclipse is not an agent/bundle pool in my configuration.
Re: Using bundle pools in eclipse installer simple mode [message #1780770 is a reply to message #1780769] Fri, 26 January 2018 19:07 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
When the unknownhostexception occurs (because of the firewall and lack of proxies), I debugged the oomph installer and noticed that the compositeContent<hashcode>.jar or .xml file would not be found in the repository cache for that ibm.com url.
Re: Using bundle pools in eclipse installer simple mode [message #1780783 is a reply to message #1780770] Sat, 27 January 2018 04:44 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 29552
Registered: July 2009
Senior Member
The ~/.p2/org.eclipse.equinox.p2.repository cache folder avoids repeatedly downloading the large content.xml/jar and artifacts.xml/jar from update sites, when their timestamp hasn't changed since the last time it was read. But p2 doesn't do such a good job of caching all the things that need to be cached and you can't tell what exactly it has cached because of the horrible hash codes. Oomph supports an offline mode where you can reproduce an p2 resolution or Targlet resolution that worked once online. It augments the p2 cache via ~/.eclipse/org.eclipse.oomph.p2/cache and here you can really see what is cached and where it came from. Problems like "java.net.UnknownHostException: public.dhe.ibm.com" is fundamentally a network problem that we can do nothing about.

Oomph also maintains a cache of setups (of external resources) that it downloads in ~/.eclipse/org.eclipse.oomph.setup/cache. This is also to support offline mode and to reduce the extent to which the same file is downloaded multiple times unnecessarily.

An index archive is always loaded before loading the index itself (in org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.handleArchiveRedirection(URIConverter)). The default location is http://www.eclipse.org/setups/setups.zip but this location to maybe redirected. The index is produced by the IApplication org.eclipse.oomph.setup.internal.core.SetupArchiver which runs on an Eclipse host machine every five minutes, walking the entire set of reachable resources and including them in the zip file; these induce implicit redirections at runtime so that no one ever directly accesses the URIs reachable by the Index org.eclipse.setup. This avoids, for example, http://git.eclipse.org/c/* from being hammered to death, and improves performance because the whole index can be loaded by downloading this single file. Like all resources downloaded by org.eclipse.oomph.setup.internal.core.util.ECFURIHandlerImpl, this is cached in ~/.eclipse/org.eclipse.oomph.setup/cache .

Note that the installer can manage multiple indexes so you can for example drag and drop a setups.zip as produced by SetupArchiver to make the installer switch to a different index, and it remembers them so the user can switch between difference indices.
Re: Using bundle pools in eclipse installer simple mode [message #1780889 is a reply to message #1780783] Mon, 29 January 2018 15:47 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
See my comments below:

Ed Merks wrote on Fri, 26 January 2018 23:44
The ~/.p2/org.eclipse.equinox.p2.repository cache folder avoids repeatedly downloading the large content.xml/jar and artifacts.xml/jar from update sites, when their timestamp hasn't changed since the last time it was read. But p2 doesn't do such a good job of caching all the things that need to be cached and you can't tell what exactly it has cached because of the horrible hash codes. Oomph supports an offline mode where you can reproduce an p2 resolution or Targlet resolution that worked once online. It augments the p2 cache via ~/.eclipse/org.eclipse.oomph.p2/cache and here you can really see what is cached and where it came from. Problems like "java.net.UnknownHostException: public.dhe.ibm.com" is fundamentally a network problem that we can do nothing about.

  1. How is the offline mode augmented oomph cache populated and utilized?
  2. Does the population mode work in both simple and advanced mode?
  3. Can that cache be utilized both in simple and advanced mode?
  4. How can I avoid the "java.net.UnknownHostException: public.dhe.ibm.com" error when eclipse-installer is running behind a firewall with no proxy configured?
Ed Merks wrote on Fri, 26 January 2018 23:44

Oomph also maintains a cache of setups (of external resources) that it downloads in ~/.eclipse/org.eclipse.oomph.setup/cache. This is also to support offline mode and to reduce the extent to which the same file is downloaded multiple times unnecessarily.

An index archive is always loaded before loading the index itself (in org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.handleArchiveRedirection(URIConverter)). The default location is http://www.eclipse.org/setups/setups.zip but this location to maybe redirected. The index is produced by the IApplication org.eclipse.oomph.setup.internal.core.SetupArchiver which runs on an Eclipse host machine every five minutes, walking the entire set of reachable resources and including them in the zip file; these induce implicit redirections at runtime so that no one ever directly accesses the URIs reachable by the Index org.eclipse.setup. This avoids, for example, http://git.eclipse.org/c/* from being hammered to death, and improves performance because the whole index can be loaded by downloading this single file. Like all resources downloaded by org.eclipse.oomph.setup.internal.core.util.ECFURIHandlerImpl, this is cached in ~/.eclipse/org.eclipse.oomph.setup/cache .
I am redirecting the index to a file url. Are you indicating that I also must redirect the index archive? Why do I need the archive AND the index. I would think the archive would suffice.Ed Merks wrote on Fri, 26 January 2018 23:44

Note that the installer can manage multiple indexes so you can for example drag and drop a setups.zip as produced by SetupArchiver to make the installer switch to a different index, and it remembers them so the user can switch between difference indices.
What is the point of multiple indexes? I understand only one is used.
Re: Using bundle pools in eclipse installer simple mode [message #1780896 is a reply to message #1780889] Mon, 29 January 2018 17:04 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 29552
Registered: July 2009
Senior Member
Explaining how the caching works is just too much technical detail. Yes, it's used in the engine in all cases, even in your installed IDEs. I don't know where exactly your accessing the network for this error?

You can use the installer to install from different indexes. When there exists only one of course there's not much point, but when you have one and there is also the Eclipse.org one, then one could switch between them.
Re: Using bundle pools in eclipse installer simple mode [message #1780903 is a reply to message #1780896] Mon, 29 January 2018 19:24 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
I think you misinterpreted my questions. I am not interested in a detailed explanation of how the caching works. Given as you have stated before the documentation is sparse and one cannot try to understand the source as it is too complex to quickly gain sufficient understanding, I am, as most people, reverting to the forums. Is it not the only to gain traction for oomph and have it used by more and more people?

In my previous questions, I was not interested in technical details but rather use case scenarios. Let me rephrase the questions:

1. What do I need to do as a user in order to populate all three caches?
What I am looking for is something like: Delete .p2 and .eclipse folder and perform 1 installation with properly configured proxy settings using eclipse-installer in advanced mode
2. How can I make use of those cached artifacts/resources?
Again, I am looking at something like: invoke the eclipse-installer using the xxx option after performing step documented in question 1. This works only in advanced mode (or not, whatever)
3. What do I have to do to be able to run eclipse-installer when I do not have access to the internet?
Re: Using bundle pools in eclipse installer simple mode [message #1780920 is a reply to message #1780903] Tue, 30 January 2018 04:52 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 29552
Registered: July 2009
Senior Member
I see.

1. As a user you don't need to do anything. The underlying technology uses so-called transport layers, and they do caching as a side effect of downloading whatever is needed, utilizing the cache avoid avoid transport. So p2 has its own cache, but its opaque, so you can't tell what is in the cache from the file names without looking into each file, and its somewhat incomplete, so there is not sufficient information present that one could do the same p2 task twice without internet access. Oomph's p2 extension builds a layer on top, using file names that are an encoded version of the full URI that was accessed with complete information such that an offline mode can be supported. Oomph also needs access to setup models. Those are transported by a different layer and into a different cache. In all cases, the cache is populated in a demand driven way based on which p2 update sites are accessed and which setup models are loaded.

To test the installer, I typically launch it with -Duser.home=D:\sandbox\USER-HOME and -Doomph.setup.user.home.redirect=true so that it thinks it's in a different user home than my real user home while running, and sets -Duser.home=D:\sandbox\USER-HOME into the installation it creates that when it launches, it thinks it's in that same fake user home. Then one can see easily what and where things are all populated into the fresh new user home.

But the fundamental answer to 1 is that you don't do have to do anything.

2. Again, you don't need to do anything. The transport layer generally looks at what's being downloaded and what's in the cache, doing time stamp comparison for the two p2 caches and etag checking for the setup cache. If the cache is fresh, it's used. If not, the thing is downloaded and cached new. If the internet is inaccessible, the cache can be used as a fallback. The same mechanisms are used regardless of the mode. And in fact in the installations that are created, the same mechanisms is also used, even for "Help -> Install New Software".

3. Three things happen when the installer starts. 1) It looks at its self update [p2 site, -Doomph.installer.update.url=... to check for updates. 2) It loads the setup archive http://www.eclipse.org/setups/setups.zip (can be redirected with -Doomph.redirection.<an-arbitrary-name>) using that to populate implicit redirections. 3) It looks for the Index in index:/org.eclipse.setup. By default that's redirected as index:/->http://git.eclipse.org/c/oomph/org.eclipse.oomph.git/plain/setups/, i.e., a folder direction to where all the setups are located in the plain text web view of Oomph's git repository. The setup archive that was already loaded in turn redirects all those URIs into the setup archive. So to work completely offline, you need to redirect those to disk. When the installation is created, all the p2 update sites referenced by all the p2 tasks need to be inside the firewall or on disk.

The option -Doomph.setup.ecf.trace is useful for tracing the internet access of to setup models. If you've properly redirected all setup locations, there should be no trace. If you run under debug control, you can monitor what happens in org.eclipse.oomph.p2.internal.core.CachingTransport, or set org.eclipse.oomph.p2.internal.core.CachingTransport.DEBUG to true; no -D option for that, but I suppose that would be a nice to have.

I think provisioning an Oomph development environment to launch the installer in debug mode (with user redirections as I described above) is likely to be helpful. Both p2 and Oomph use ECF for the transport layers, which in turn uses Apache HTTP libraries, so it's likely to provide a ways to discover answers and because it can populate a clean user home that you can completely delete, you'll get a good idea of what's happening on disk.



Re: Using bundle pools in eclipse installer simple mode [message #1781380 is a reply to message #1780920] Tue, 06 February 2018 15:52 Go to previous messageGo to next message
Mario Jauvin is currently offline Mario JauvinFriend
Messages: 34
Registered: October 2015
Member
Here is a follow-up on question 3.

What is the difference between the setup archive and the index? I make a copy of the setups folder from https://git.eclipse.org/r/oomph/org.eclipse.oomph , saved it on my disk and made modification to it. I redirect the index to this disk copy and I thought this was all I needed to do. Can you explain what the setup archive is?

[Updated on: Tue, 06 February 2018 15:54]

Report message to a moderator

Re: Using bundle pools in eclipse installer simple mode [message #1781393 is a reply to message #1781380] Tue, 06 February 2018 16:51 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6394
Registered: July 2009
Senior Member
The index is the definitive root of Oomph's setup model tree. It contains the product and project catalogs.

The setup archive is an optimization of that setup model tree. It's a zip file that contains everything that's reachable from the root/index. This way the installer and other Oomph facilities can load all models with just a single http request. And they only do it if at least one zip entry has changed since the last download.
Re: Using bundle pools in eclipse installer simple mode [message #1781414 is a reply to message #1781393] Wed, 07 February 2018 05:22 Go to previous message
Ed Merks is currently offline Ed MerksFriend
Messages: 29552
Registered: July 2009
Senior Member
It's important to note (as I mentioned above) that all resource sets for Setup models are initialized by org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.configureResourceSet(ResourceSet) which (except for the ArchiverApplication), always calls org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.handleArchiveRedirection(URIConverter) which always loads http://www.eclipse.org/setups/setups.zip (or whatever that is redirected to). This is done as a last step of configuring the resource set and it's done after the call to org.eclipse.oomph.setup.internal.core.util.SetupCoreUtil.configureRedirections(Map<URI, URI>) which configures the redirections specified by -Doomph.redirection.*.

Why? What is the purpose of that? The archive name structure of the zip entries is such that the name of each zip entry can be used to reconstruct the URI that the zip entry represents. The handleArchiveRedirection method uses this to induce additional redirections, i.e., the URI of each zip entry induces a redirection from that URI to the URI of the zip entry itself, but only if that URI isn't already redirected to something else by the configureRedirections previous setup. So, as Eike mentions, this is a huge optimization because all setup models can be loaded from the archive because all reachable resources have been stored in the archive.

But now suppose we redirect (with -Doomph.redirection.*) the index, or the folder of the index. The setup archive is still loaded, but anything in the archive that has already been redirected elsewhere will not be redirected to the archive. This is why you can redirect the index to something else. But note that the other redirections are still in effect. So if you copy the stuff in the Oomph setups folder, change it, and redirect to that changed version, that works nicely, and most importantly, all the other things in the archive that you've not redirected still redirect to the archive. So you don't have to copy all reachable resources to use them in your index. If you reuse nothing from the index we provide, you could redirect the archive to archive containing only the setups/models folder, or probably better, to an archive built from your index so that you have the same nice optimization of loading the entire set of reachable resources in a single http request. Note in particular that the setups/models folder is super important for dynamically loading models whose implementing plugins are not actually installed. These models can change with each commit we make in Oomph. So if you redirect the archive itself, you must maintain an update-to-date version of this folder in your archive.

So the best thing to have 100% control over what is loaded and from where it's loaded, you build your own index folder with your own stuff, use the SetupArchiver application to build archive for it, and then also redirect http://www.eclipse.org/setups/setups.zip to your archive's location. Alternatively, make a copy of http://www.eclipse.org/setups/setups.zip periodically and keep it at a different location to which you redirect.
Previous Topic:Oomph headless
Next Topic:Problem with redirection in eclipse-inst.ini
Goto Forum:
  


Current Time: Tue Sep 25 13:25:38 GMT 2018

Powered by FUDForum. Page generated in 0.02696 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top