| Re: [justj-dev] Justj project |
Hi, Lets move this discussion to mailing list. So I am responding to mailing list now. From the questions I feel There are two paths here. Packaging JRE in the EPP and generate a jre based on components selected in oomph/installer and use it.
In the packaging stage there is a place where product contents are created with all the features and their dependencies. We use this to generate jdeps and in turn required java . Also if we have a folder named jre at the same level as eclipse launcher executable, launcher tries to create jvm using $ECLIPSE_HOME/jre/bin/java. This works in linux and windows not in mac. This could be one way
Thanks Sravan From: Daniel Megert <daniel_megert@xxxxxxxxxx> > I wonder if we should be discussing this via the mailing list? Ed Merks ---30.04.2020 08:13:07---Sravan, I wonder if we should be discussing this via the mailing list?
I wonder if we should be discussing this via the mailing list? https://accounts.eclipse.org/mailing-list/justj-dev There are those who will complain about private conversations. Of course at this point, we're the only ones... ____________________________________ This is a long email with lots of details to answer your question, but at the end there are some "open questions". Please be patient while I iron out the details on my prototyping effort and can produce results that can be further/properly tested... ____________________________________ I've been doing a lot of prototyping to automate as much of the processing as possible, so I have quite a bit already working already and tested already... So you'll see that I have a few "test/prototype" jobs running already on the build server: For example this one: https://ci.eclipse.org/justj/job/SimRel%20jdeps/lastBuild/ It uses jdep on all plugins in the current 2020-06 staging repository. That produces this list of modules: https://ci.eclipse.org/justj/job/SimRel%20jdeps/lastSuccessfulBuild/artifact/actual-deps/*view*/ java.base There's also this job: https://ci.eclipse.org/justj/job/Tools/ For this I uploaded a zip of a local git clone with some "releng" JustJ tools that I hope/intend to use. So you can find that zip here: https://ci.eclipse.org/justj/job/Tools/lastSuccessfulBuild/artifact/ So right now the job just unzips this clone and does a Maven/Tycho build that builds the update site for the tools: The purpose of the tools is to generate the entire scaffolding necessary to build a p2 update site with JREs. It defines a model that allows the following file to be loaded and that is used to generate the all the necessary structure for building the JREs: This is still a work in progress, but is fully tested locally. It generates the attached zip file, and that's something you should be able to unzip and build locally. But the point is it doesn't contain any of the actual JREs, it's just the scaffolding, so the build will not produce something that works until you copy a JRE into the "jre" folder of one of the fragments. Finally there is this job for actually building the JREs using jlink: https://ci.eclipse.org/justj/job/Build%20JREs/ This is of course also a work in progress and produces these results: https://ci.eclipse.org/justj/job/Build%20JREs/lastBuild/artifact/ You can view the job configuration to see what it does. That is the script I'm current working on and I am waiting on this: https://bugs.eclipse.org/bugs/show_bug.cgi?id=562477 I.e., I need agents that run natively on Mac and Windows to run jlink natively to produce the Mac and Windows JREs. (Apparently corporate sponsorship will be needed long term for these agents.) You'll see the above script curls the JDK from open JDK and curls a recent Eclipse SDK. It first produces a "full" JRE with all the modules in the JDK. It also produces a JRE with the minimal hard-coded modules that I computed are needed for the Eclipse Installer. It also produces a JRE for the modules computed (in the script) to be needed by the Eclipse SDK. I'm currently trying to understand the size reductions. What I noticed is that the JRE produced by the full set of "simrel" required modules is only slightly larger (2M) than the minimal set needed for the installer and by the minimal set needed by the Eclipse SDK, so it's probably of little value to produce those smaller variants... I do already have the above working locally using my Windows machine as well as my Linux and Mac virtual boxes, and I have tested that the Eclipse SDK launches with each of them. I'm also looking at launching this Ant build.xml: <?xml version="1.0"?> I.e., launching that via the Eclipse SDK's antRunner and using the actual jlink-generated JRE, e.g., something like this: eclipse/eclipse -application org.eclipse.ant.core.antRunner -nosplash -emacs -vm jre.simrel.minimal.compressed/bin This would let me not only test that Eclipse launches with that JRE, but also to see which packages OSGi computes for this JRE: org.osgi.framework.system.packages=com.sun.jdi, com.sun.jdi.connect, com.sun.jdi.connect.spi, com.sun.jdi.event, com.sun.jdi.request, com.sun.management, com.sun.nio.file, com.sun.tools.attach, com.sun.tools.attach.spi, java.applet, java.awt, java.awt.color, java.awt.datatransfer, java.awt.desktop, java.awt.dnd, java.awt.event, java.awt.font, java.awt.geom, java.awt.im, java.awt.im.spi, java.awt.image, java.awt.image.renderable, java.awt.print, java.beans, java.beans.beancontext, java.io, java.lang, java.lang.annotation, java.lang.constant, java.lang.instrument, java.lang.invoke, java.lang.management, java.lang.module, java.lang.ref, java.lang.reflect, java.lang.runtime, java.math, java.net, java.net.spi, java.nio, java.nio.channels, java.nio.channels.spi, java.nio.charset, java.nio.charset.spi, java.nio.file, java.nio.file.attribute, java.nio.file.spi, java.rmi, java.rmi.activation, java.rmi.dgc, java.rmi.registry, java.rmi.server, java.security, java.security.cert, java.security.interfaces, java.security.spec, java.sql, java.text, java.text.spi, java.time, java.time.chrono, java.time.format, java.time.temporal, java.time.zone, java.util, java.util.concurrent, java.util.concurrent.atomic, java.util.concurrent.locks, java.util.function, java.util.jar, java.util.logging, java.util.prefs, java.util.regex, java.util.spi, java.util.stream, java.util.zip, javax.accessibility, javax.annotation.processing, javax.crypto, javax.crypto.interfaces, javax.crypto.spec, javax.imageio, javax.imageio.event, javax.imageio.metadata, javax.imageio.plugins.bmp, javax.imageio.plugins.jpeg, javax.imageio.plugins.tiff, javax.imageio.spi, javax.imageio.stream, javax.lang.model, javax.lang.model.element, javax.lang.model.type, javax.lang.model.util, javax.management, javax.management.loading, javax.management.modelmbean, javax.management.monitor, javax.management.openmbean, javax.management.relation, javax.management.remote, javax.management.timer, javax.naming, javax.naming.directory, javax.naming.event, javax.naming.ldap, javax.naming.ldap.spi, javax.naming.spi, javax.net, javax.net.ssl, javax.print, javax.print.attribute, javax.print.attribute.standard, javax.print.event, javax.rmi.ssl, javax.script, javax.security.auth, javax.security.auth.callback, javax.security.auth.kerberos, javax.security.auth.login, javax.security.auth.spi, javax.security.auth.x500, javax.security.cert, javax.security.sasl, javax.sound.midi, javax.sound.midi.spi, javax.sound.sampled, javax.sound.sampled.spi, javax.sql, javax.sql.rowset, javax.sql.rowset.serial, javax.sql.rowset.spi, javax.swing, javax.swing.border, javax.swing.colorchooser, javax.swing.event, javax.swing.filechooser, javax.swing.plaf, javax.swing.plaf.basic, javax.swing.plaf.metal, javax.swing.plaf.multi, javax.swing.plaf.nimbus, javax.swing.plaf.synth, javax.swing.table, javax.swing.text, javax.swing.text.html, javax.swing.text.html.parser, javax.swing.text.rtf, javax.swing.tree, javax.swing.undo, javax.tools, javax.transaction.xa, javax.xml, javax.xml.catalog, javax.xml.datatype, javax.xml.namespace, javax.xml.parsers, javax.xml.stream, javax.xml.stream.events, javax.xml.stream.util, javax.xml.transform, javax.xml.transform.dom, javax.xml.transform.sax, javax.xml.transform.stax, javax.xml.transform.stream, javax.xml.validation, javax.xml.xpath, org.ietf.jgss, org.w3c.dom, org.w3c.dom.bootstrap, org.w3c.dom.css, org.w3c.dom.events, org.w3c.dom.html, org.w3c.dom.ls, org.w3c.dom.ranges, org.w3c.dom.stylesheets, org.w3c.dom.traversal, org.w3c.dom.views, org.w3c.dom.xpath, org.xml.sax, org.xml.sax.ext, org.xml.sax.helpers, sun.misc, sun.reflect So I can verify that indeed it's got fewer packages than the full JRE: org.osgi.framework.system.packages=com.sun.jarsigner, com.sun.java.accessibility.util, com.sun.jdi, com.sun.jdi.connect, com.sun.jdi.connect.spi, com.sun.jdi.event, com.sun.jdi.request, com.sun.management, com.sun.net.httpserver, com.sun.net.httpserver.spi, com.sun.nio.file, com.sun.nio.sctp, com.sun.security.auth, com.sun.security.auth.callback, com.sun.security.auth.login, com.sun.security.auth.module, com.sun.security.jgss, com.sun.source.doctree, com.sun.source.tree, com.sun.source.util, com.sun.tools.attach, com.sun.tools.attach.spi, com.sun.tools.javac, com.sun.tools.jconsole, java.applet, java.awt, java.awt.color, java.awt.datatransfer, java.awt.desktop, java.awt.dnd, java.awt.event, java.awt.font, java.awt.geom, java.awt.im, java.awt.im.spi, java.awt.image, java.awt.image.renderable, java.awt.print, java.beans, java.beans.beancontext, java.io, java.lang, java.lang.annotation, java.lang.constant, java.lang.instrument, java.lang.invoke, java.lang.management, java.lang.module, java.lang.ref, java.lang.reflect, java.lang.runtime, java.math, java.net, java.net.http, java.net.spi, java.nio, java.nio.channels, java.nio.channels.spi, java.nio.charset, java.nio.charset.spi, java.nio.file, java.nio.file.attribute, java.nio.file.spi, java.rmi, java.rmi.activation, java.rmi.dgc, java.rmi.registry, java.rmi.server, java.security, java.security.cert, java.security.interfaces, java.security.spec, java.sql, java.text, java.text.spi, java.time, java.time.chrono, java.time.format, java.time.temporal, java.time.zone, java.util, java.util.concurrent, java.util.concurrent.atomic, java.util.concurrent.locks, java.util.function, java.util.jar, java.util.logging, java.util.prefs, java.util.regex, java.util.spi, java.util.stream, java.util.zip, javax.accessibility, javax.annotation.processing, javax.crypto, javax.crypto.interfaces, javax.crypto.spec, javax.imageio, javax.imageio.event, javax.imageio.metadata, javax.imageio.plugins.bmp, javax.imageio.plugins.jpeg, javax.imageio.plugins.tiff, javax.imageio.spi, javax.imageio.stream, javax.lang.model, javax.lang.model.element, javax.lang.model.type, javax.lang.model.util, javax.management, javax.management.loading, javax.management.modelmbean, javax.management.monitor, javax.management.openmbean, javax.management.relation, javax.management.remote, javax.management.remote.rmi, javax.management.timer, javax.naming, javax.naming.directory, javax.naming.event, javax.naming.ldap, javax.naming.ldap.spi, javax.naming.spi, javax.net, javax.net.ssl, javax.print, javax.print.attribute, javax.print.attribute.standard, javax.print.event, javax.rmi.ssl, javax.script, javax.security.auth, javax.security.auth.callback, javax.security.auth.kerberos, javax.security.auth.login, javax.security.auth.spi, javax.security.auth.x500, javax.security.cert, javax.security.sasl, javax.smartcardio, javax.sound.midi, javax.sound.midi.spi, javax.sound.sampled, javax.sound.sampled.spi, javax.sql, javax.sql.rowset, javax.sql.rowset.serial, javax.sql.rowset.spi, javax.swing, javax.swing.border, javax.swing.colorchooser, javax.swing.event, javax.swing.filechooser, javax.swing.plaf, javax.swing.plaf.basic, javax.swing.plaf.metal, javax.swing.plaf.multi, javax.swing.plaf.nimbus, javax.swing.plaf.synth, javax.swing.table, javax.swing.text, javax.swing.text.html, javax.swing.text.html.parser, javax.swing.text.rtf, javax.swing.tree, javax.swing.undo, javax.tools, javax.transaction.xa, javax.xml, javax.xml.catalog, javax.xml.crypto, javax.xml.crypto.dom, javax.xml.crypto.dsig, javax.xml.crypto.dsig.dom, javax.xml.crypto.dsig.keyinfo, javax.xml.crypto.dsig.spec, javax.xml.datatype, javax.xml.namespace, javax.xml.parsers, javax.xml.stream, javax.xml.stream.events, javax.xml.stream.util, javax.xml.transform, javax.xml.transform.dom, javax.xml.transform.sax, javax.xml.transform.stax, javax.xml.transform.stream, javax.xml.validation, javax.xml.xpath, jdk.dynalink, jdk.dynalink.beans, jdk.dynalink.linker, jdk.dynalink.linker.support, jdk.dynalink.support, jdk.incubator.foreign, jdk.javadoc.doclet, jdk.jfr, jdk.jfr.consumer, jdk.jshell, jdk.jshell.execution, jdk.jshell.spi, jdk.jshell.tool, jdk.management.jfr, jdk.nashorn.api.scripting, jdk.nashorn.api.tree, jdk.net, jdk.nio, jdk.nio.mapmode, jdk.security.jarsigner, jdk.swing.interop, netscape._javascript_, org.ietf.jgss, org.w3c.dom, org.w3c.dom.bootstrap, org.w3c.dom.css, org.w3c.dom.events, org.w3c.dom.html, org.w3c.dom.ls, org.w3c.dom.ranges, org.w3c.dom.stylesheets, org.w3c.dom.traversal, org.w3c.dom.views, org.w3c.dom.xpath, org.xml.sax, org.xml.sax.ext, org.xml.sax.helpers, sun.misc, sun.reflect So the thinking at this point is that there will be three jobs that run natively on each of the three supported platforms to produce a full JRE and one or more minimized JREs. Here is some size information from running a more up-to-date version of the script locally: $du -s -k *.tar I computed the tar sizes because the Windows installer will package up a tar file and this size is close to the uncompressed size on disk for the folder. I would prefer to package up a zip file but I don't believe there is any EPL compatible native C library for processing a zip. The tar format it easier to process natively with relatively simple C logic... In any case, you can see the relative size differences, also taking note that stripping the debug information significantly reduces the size... So the general idea is that the three jobs will produce *.tar.gz and *.zip of the JRE variants, likely along with information about the contained modules and perhaps also the available Java packages. The generator model can be updated based on these inputs. Another thing (very rough still), is something that can process the *.tar.gz to analyze file permissions (even on Windows where this can't be done using the native file system): This can be used to determine the touchpoint instructions that are needed. I.e., the necessary chmod +x on the excutable files and well as any neccessary symbolic links. There are many symbolic links in the legal files, but I doubt that saves much space... But I've seen that the adoptOpenJDK versions contain (or should contain) important symbolic links to executables, so that might/will be problem if they are not handled properly (as we've seen on Mac already). In any case, the generator produces all the scaffolding (projects with poml.xml) for building JREs (with Tycho) and it should unpackage the *.tar.gz and *.zip into that scaffolding. This is mostly working already locally manaully, so locally I've been able to produce an update site with the JREs packaged as IUs (on Windows). Using this, I've updated Oomph's installer product to require the minimal JRE during the build. Also, I have modified the Windows native executable to be able to extract a *.tar of the installer product containing the minimal JRE. This launches properly and will always launch on Windows even if there is no appropriate version of Java installed. Then I hacked the product catalog locally so that the product (Eclipse SDK product) requires the full JRE IU and then used the installer to create an installation that uses (has a -vm option for) that embedded JRE; the JRE IU is actually in the shared bundle pool so is reused by multiple installations. This all works properly and I can create Java projects and run them. On Windows and Linux, the IDE will even use the JRE of the IDE as the JRE in the Java preferences; on Mac that is not the case. One gotcha that I noticed is that one cannot debug launch when using the minimized JRE. I believe this is because one or more of these agents are missing: jdk.hotspot.agent@14.0.1 _______________________________________ Some open questions.
Regards, On 30.04.2020 05:11, Sravan K Lakkimsetti wrote: Hi,
Please note I did not run full test suite with these. I just launched eclipse. My suspicion would be p2 would require more modules to be included. [attachment "org.eclipse.justj.zip" deleted by Daniel Megert/Zurich/IBM] |