Re: [jdt-dev] Installs are consistently failing

[Ed Merks <ed.merks@xxxxxxxxx>]

Comments below.

On 30.03.2020 10:32, R Steiger wrote:

Ed,

 

Thanks for confirming my suspicions. 

 

I tried your suggested cloning-avoidance technique, copying the git directory from an earlier jdt-master, first by simply doing an OS-level copy (which doesn’t do a deep copy in the face of any symlinks), and which did indeed fail with the now-familiar “No more authentication methods available” fault.   I then repeated the procedure, zipping the source git dir, then unzipping in the new jdt-master root dir, and that, too failed, in the same way.

    If the clone task is running, you've not put the clones in the
    physical location to which the task tries to clone.  I don't believe
    there are any symlinks;  I'm not sure how that would work on
    Windows.

 

Leaving the following Qs:

1. Shouldn’t the zip/unzip procedure do a proper deep copy?  If not, can you suggest a CLI command sequence that does?

    Any of these approaches should properly copy the clone.

1. 
   
2. What’s the root-cause of the auth fault? 

It's really hard to say.   I know for example that a build last night failed like this:

!ENTRY org.eclipse.osgi 4 0 2020-03-30 20:31:51.129
!MESSAGE Application error
!STACK 1
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1DiagnosticWrappedException: java.io.IOException: Server returned HTTP response code: 504 for URL: https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/plain/simrel.aggr

    It seems that 50* errors have been a plague on and off for the past
    weeks.

1. In my environment, its gone from an occasional annoyance (maybe 10-20% of installs, usually doesn’t recur on restarts), to damned nearly 100% of installs, now at least zero for 10.  Fixing at its root looks like my only path forward at this point.

Perhaps asking on the EGit forum would be good.

https://www.eclipse.org/forums/index.php/f/48/

It would also be good when the problem is easily reproducible to also test if git from the command line has the same problems.

Perhaps the settings for Team -> Git SSH client or HTTP client change the behavior you're seeing...

 

Lastly, thanks for confirming the heavy-lifting schedule, the git server has been way faster tonight 😊.

 

Thanks,

 

-rjs

 

From: Ed Merks <ed.merks@xxxxxxxxx>
Sent: Sunday, March 29, 2020 10:34 PM
To: R Steiger <rsteiger@xxxxxxxxxxxxxxxx>; stephan.herrmann@xxxxxxxxx
Cc: Eclipse JDT general developers list. <jdt-dev@xxxxxxxxxxx>
Subject: Re: Installs are consistently failing

 

Comments below.

On 29.03.2020 23:59, R Steiger wrote:

Hey guys,

 

I’ve run into a mystery, and a show-stopper:

 

• Mystery: I had a perfectly-functioning jdt-master workspace, up until in a reorganization, I moved it (and all other eclipse development files) to another drive.  I was careful to fix the hard filepaths in the workspace of which I’m aware (didn’t attempt to fix any in .metadata).  When launched, the transplanted workspace now comes up with no projects.

Moving a workspace is generally not workable.  The .metadata is not generally portable and will contain absolute locations in binary files.

•  

• Any idea why this might be happening?
• Do you know of a straightforward way to repair the workspace and get it back on the air?

No, I don't think this is repairable.  I think you must delete the workspace and create a new one following the same procedure as you used to create the original one.  If you use the standard layout choices of the installer, the installation will go into a folder with the following children:

eclipse
git
ws

If you want a new setup in a different drive/folder and want to avoid re-cloning all the repos.  You can deep-copy the current git folder into the new location before you start that process.  That will create a new installation "eclipse" and will create a new workspace "ws" and during the setup process once that installation is launched, the Git clone tasks will find the Git clones already exist and will skip the cloning process.

•  

• Show-stopper: since the above leaves me now without a working jdt-master, I’ve attempted (a bunch of times) to recreate one.

o    After attempting this using the Installer at least 6 times, each time, after grinding through a bunch of repo clones (at about the 20-35% progress mark), it’s consistently failing, emitting a batch of five “No more authentication methods available” stacktraces, and erroring-out.

It's possible that the git.eclipse.org server was having problems.   If you're in this state now, you could delete the "git" folder and copy over the "git" folder from the older location so that the setup process will skip the cloning task.

 

• None of the advice I’ve found on the ‘web provides useful diagnostics, deterministic fixes, nor an effective workaround.
• While I’d run into this failure a few times in the last week, and was able to get past it by restarting the installs, it’s seeming to be down hard now.  I’ve made no changes to local network, firewalls, creds, etc.

I did a pull on all 26 repos in my SDK setup just now, so perhaps the server is working better now.

•  

 

Related questions:

• Is there a daily major build process?  I noticed when trying to run installs between 2-4:30am PDT that progress would hang for 15 to 30 minutes at a time, suggesting to me that the git fleet was heavily loaded. 

Certainly Sunday seems to be the worst time for having such problems.  In the end, there are hundreds of build processes for the hundreds of project that are all using the git.eclipse.org server to do pulls or clones...

•  
• Several of the export filesets contain multiple versions of the same logical jar, e.g. org.eclipse.jdt.annotation_1.1.400.202003272004.jar and org.eclipse.jdt.annotation_2.2.400.202003272004.jar. 

• Is this to be expected?

Yes, there are expected to be two versions of this jar.

•  
• If so, how are they all accessed without collisions?  My understanding is that each plugin has it’s own classLoader, and that could allow such multi-version coexistence.  Is that correct?

Yes, that's correct.

•  

 

Thanks much, and hope y’all are staying safe!

Often these days I feel as if my life would be significantly better if I were a little less safe.

 

-rjs

 

From: jdt-dev-bounces@xxxxxxxxxxx <jdt-dev-bounces@xxxxxxxxxxx> On Behalf Of R Steiger
Sent: Thursday, March 26, 2020 12:14 AM
To: Ed Merks <ed.merks@xxxxxxxxx>; Eclipse JDT general developers list. <jdt-dev@xxxxxxxxxxx>
Subject: Re: [jdt-dev] discussion about my current "Enable Classpath Cycles"project

 

Hey Ed,

 

Thanks for getting back to me.  (I sure do understand daily mail floods!)

 

Responses inline.

 

From: Ed Merks <ed.merks@xxxxxxxxx>
Sent: Wednesday, March 25, 2020 9:46 PM
To: R Steiger <rsteiger@xxxxxxxxxxxxxxxx>; Eclipse JDT general developers list. <jdt-dev@xxxxxxxxxxx>
Subject: Re: [jdt-dev] discussion about my current "Enable Classpath Cycles"project

 

Sorry, I must have overlooked this in my daily flood of email.  Note that in the wiki it references https://bugs.eclipse.org/bugs/show_bug.cgi?id=536533 also for asking question.  In the Platform SDK Setup there is this preference task:

<?xml version="1.0" encoding="UTF-8"?>
<setup:CompoundTask
    xmi:version="2.0"
    xmlns:xmi="http://www.omg.org/XMI"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:setup="http://www.eclipse.org/oomph/setup/1.0"
    name="org.eclipse.oomph.setup.ui">
  <setupTask
      xsi:type="setup:PreferenceTask"
      key="/instance/org.eclipse.oomph.setup.ui/showToolBarContributions"
      value="true"/>
</setup:CompoundTask>

So I expect that the equivalent of Window -> Preferences -> Oomph -> Setup Tasks "Show tool bar contributions" would have been performed. 

[R Steiger] for some reason it appears this task didn’t execute, just judging from the absence of Oomph toolbar buttons.

If not, you can do that manually and even if not set you can also use Help -> Perform Setup Tasks  and Navigate -> Open Setup to access the same actions/menus.  So these instructions apply to an existing development environment's workspace from the previous steps of the tutorial, not to the installer.

[R Steiger] Thanks for confirming that section 8 assumes running in the dev env ws.   I looked at Help -> Perform Setup Tasks, under Manual Tasks, and see 4 Eclipse Ini -Doomph.redirection… tasks, none of them mention anything about setup:PreferenceTask nor showToolBarContributions .  Similar finding under Navigate -> Open Setup , nothing looks relevant to this setup step.  Btw, I didn’t find any affordance for manually entering a new task, so was unable to try your suggestion. 

So no still no Oomph toolbar buttons, but that might not be what I really need.

But if you're asking how to "build the IDE" and by that you don't mean to how to set up a development environment but rather how to  replicate the Maven/Tycho build locally to produce a p2 update site and the other artifacts produced on the build machine, that I don't know. 

[R Steiger] By "build the IDE"  I mean the first option, that I want to build a launchable IDE having the deltas I’ve coded and tested in the jdt-master workspace, specifically, so the compiler mods are operational.  I’ve been able to do some testing in the instance spawned with a debug configuration, but it’s tedious, and the key scenario requires importing several projects, all of which is lost when the debug instance exits.  I tried to switch the debugee’s workspace so as to snapshot the test setup, but got a popup saying “Unable to relaunch the workbench because the eclipse.vm property has not been set.”.  So, no, at this early point, I’m not (yet) trying to replicate the Maven/Tycho build, create a p2 update site, etc.

The other tack I looked at (as described in the “Eclipse Bible”) was exporting, but quickly realized that under a manual approach, I don’t know how to specify the IDE’s configuration, and would be even more ignorant trying to automate with Ant. 

Hence my asking how to do what I imagine is a pretty common step when working on JDT, PDE, etc., namely create or modify the feature(s), unit test them, build them into a new IDE package, then take it out on the road.

Thanks again,

-rjs

Generally that will involve invoking Maven on the appropriate pom.xml.

Regards,
Ed

On 26.03.2020 03:44, R Steiger wrote:

Folks,

 

I’m up against a deadline on this project, and been waiting for Ed’s reply, but he could easily be taking a break, out of town, whatever. 

 

Can anyone else point me to the first step on the path to building the SDK?  (I got stuck in section 8 of the provisioning wiki page, not seeing any Oomph toolbar buttons in jdt-master, leading me to try looking in the Installer, but leaving me unable to make sense of doing so with an existing workspace, since all the flow is aimed at building a new workspace.  And haven’t yet found any other guides covering IDE builds.)

 

Much thanks,

 

-rjs

 

From: jdt-dev-bounces@xxxxxxxxxxx <jdt-dev-bounces@xxxxxxxxxxx> On Behalf Of R Steiger
Sent: Tuesday, March 24, 2020 2:49 PM
To: Ed Merks <ed.merks@xxxxxxxxx>
Cc: Eclipse JDT general developers list. <jdt-dev@xxxxxxxxxxx>
Subject: Re: [jdt-dev] discussion about my current "Enable Classpath Cycles"project

 

Hi Ed,

I just saw your reply of 3/1 to Gayan Perera’s question about building a JDT distribution.  I have the same issue (see below). 

In reviewing https://wiki.eclipse.org/Eclipse_Platform_SDK_Provisioning, specifically section 8 (“Update the Installation and Workspace"), in what workspace/context is this to be performed?  E.g. I looked in my dev workspace, and couldn’t find any Oomph tool bar contributions.  Do these instructions assume running in the Eclipse Installer?

Thanks,

-rjs

 

From: jdt-dev-bounces@xxxxxxxxxxx <jdt-dev-bounces@xxxxxxxxxxx> On Behalf Of R Steiger
Sent: Tuesday, March 24, 2020 1:35 AM
To: jdt-dev@xxxxxxxxxxx
Subject: Re: [jdt-dev] discussion about my current "Enable Classpath Cycles"project

 

After being away from doing any eclipse work since last October, I’ve resumed getting ejc to allow project dependency cycles.  (All the following in on 2020-03.)

 

[Stephan and Andrey, Cc’ing you since you’ve both helped orient me on this project, and also in case you’re interested in the changes I’m proposing, especially if you see problems and/or have suggestions.]

 

The mods I’ve made and tested are, briefly:

• Added an Ignore option to  ... -> Compiler -> Building -> Build path problems -> Circular dependencies:
• In JavaProject:createClasspathProblemMarker, when Ignore is selected, in the absence of any other classpath problems, detected dependency cycles are ignored.  The net effect of this is to suppress adding a buildpath problem marker to the project, altogether.  This approach of ditching markers at the earliest opportunity proved to be surgically clean, and avoided “chasing” after markers, then suppressing them downstream in the Ignore case.
• I hacked MultiProjectTests:testCycle*, setting CORE_CIRCULAR_CLASSPATH to JavaCore.IGNORE instead of JavaCore.WARNING, but only tested testCycle1, which covers my core use-case.
• Before submitting these changes, I’d like to properly parameterize the testCycle* methods, and have them run twice, once with IGNORE, once with WARNING.  While I have an idea how to do this without bloating the code, I’d feel better making this change after discussing how best to handle such parameterized tests with someone who’s familiar with the existing testing rubric, and maybe has implemented such parameterization.

 

The next step is to road-test these mods.  My thought is to locally build a stock Eclipse IDE for Java Developers package, having the above mods, and put it into daily use for a couple of weeks, using it to work on a large code-base.  What’s the recipe for building the IDE?  I’d like to use the most lightweight path, e.g. don’t need to create an update site, doesn’t require pushing to git, etc.

 

Thanks,

 

-rjs