Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Virgo » Virgo dumps - 64bit vs 32bit OS?(Virgo starts 100% of the time on 64bit OS and 50% of the time on 32bit OS.)
Virgo dumps - 64bit vs 32bit OS? [message #1027944] Wed, 27 March 2013 16:21 Go to next message
joel mann is currently offline joel mannFriend
Messages: 7
Registered: October 2012
Junior Member
All-

We are having an interesting issue here: Our Virgo 3.6.0 application starts up successfully 100% of the time on our 64-bit machines (Windows7 in addition to Windows 2008 Server) and (on average) has about 50% successful startup rate on Windows 2003 Server 32-bit.

Our JAVA_HOME configuration on the boxes:

-Windows 7 (64bit OS): Java 1.6 32bit
-Windows 2008 (64bit OS): Java 1.6 64bit
-Windows 2003 (32bit OS): Java 1.6 32bit

When the server crashes on the Windows 2003 box, we simply get "dump generated" in serviceability directory. The dump nor log files do not give us anything to go off of really, so we are struggling the make any traction. We've gone as far as hooking up a debugger to the Virgo code, which did not reveal anything useful.

Other points to note here are that we have 36 bundles deployed to the server with less then 100 dependencies. Presently we suspect there to be some sort of a race condition on the 32bit OS, however we have no solid proof.

There is really no common denominator we can find here other then the server starts up successfully 100% of the time on the 64bit OS. Anyone have any hints/suggestions on where to head to find the culprit?



[Updated on: Wed, 27 March 2013 17:52]

Report message to a moderator

Re: Virgo dumps - 64bit vs 32bit OS? [message #1028821 is a reply to message #1027944] Thu, 28 March 2013 19:42 Go to previous messageGo to next message
joel mann is currently offline joel mannFriend
Messages: 7
Registered: October 2012
Junior Member
I see we have stumped even the experts here...? Laughing

So we have narrowed it down to a few possible bundles that are causing issues. Or more appropriately, narrowed it down to a few bundles that Virgo cannot resolve dependencies successfully 100% of the time..?

Our "solution" is to manually copy over the one of the suspect bundles to the pickup directory AFTER the server starts (with "-clean"). This works 100% of the time on the 32-bit OS box.

Anyone have any idea why Virgo starts up successfully 100% of the time when manually dropping a bundle in the pickup directory vs placing it in the plan? Does Virgo process items that are dropped in the pickup directory differently then if its placed in the plan? If it does process items the pickup directory differently, is there anyway to deploy them via the plan in the same manner?

FYI, The bundle is in question was and has always been placed last in the plan...

[Updated on: Thu, 28 March 2013 20:10]

Report message to a moderator

Re: Virgo dumps - 64bit vs 32bit OS? [message #1032594 is a reply to message #1027944] Wed, 03 April 2013 07:29 Go to previous messageGo to next message
Daniel McGreal is currently offline Daniel McGrealFriend
Messages: 27
Registered: April 2012
Junior Member
I doubt it's relevant, but we had a similar issue with bundle loading sequence where my development machine would always succeed but colleagues' machines wouldn't. It turned out directory queries on my Mac would return the listing in alphabetical order, whereas the Linux OS was something more arbitrary, causing the bundles to load in a different sequence. You're using a plan, of course, so I wouldn't think that this would be a possibility for you.
Re: Virgo dumps - 64bit vs 32bit OS? [message #1032996 is a reply to message #1032594] Wed, 03 April 2013 18:38 Go to previous messageGo to next message
joel mann is currently offline joel mannFriend
Messages: 7
Registered: October 2012
Junior Member
Thanks for the reply Daniel.

We have been feverishly attempting to resolve the issue over the past few days. Long story short we uncovered the fact that when Virgo dumps it takes over 30secs to calc dependencies and when it successfully loads it takes about 14secs. Apparently by default if the dependencies take longer then 30secs to load it simply dumps. Whoops! :-/ This was uncovered by turning on the debug logging in the Virgo/Eclipse code.

We are presently thinking we are one of the few teams that are using Virgo to this extent, which is why no one else has encountered this problem. A few days ago one of our developers dug up a somewhat hidden system property: -Dosgi.usesTimeout, which apparently is used to increase/set the amount of time it takes to resolve dependencies. Presently there are about 4 Google hits if you search for this property. Very Happy

We set it to 90000ms and this seemed to help the issue, however we actually uncovered a small bug in Virgo/eclipse (way down in the code) where this system property gets wiped out sometimes if you set it. I think we are planning to submit a patch/bug to Virgo/Eclipse to fix this issue. Smile I'll post up the bug # here for reference once we submit it.

Hopefully this will help someone out there in the future...

Re: Virgo dumps - 64bit vs 32bit OS? [message #1037467 is a reply to message #1032996] Tue, 09 April 2013 15:33 Go to previous messageGo to next message
Glyn Normington is currently offline Glyn NormingtonFriend
Messages: 1222
Registered: July 2009
Senior Member
Just noticed this thread and I'm relieved you are making progress. The Equinox resolver works hard to solve the NP complete resolution problem and occasionally it takes a long time to resolve a set of bundles. One other trick you might find helpful is to install some stuff using initialArtifacts (see the User Guide). These are installed after Virgo has come up, but before pickup is processed etc. Another question is why the resolution is so hard - are there lots of packages which are both imported and exported (yeah, I know it's an OSGi best practice, but it doesn't always help the resolver!)?

[Updated on: Tue, 09 April 2013 15:33]

Report message to a moderator

Re: Virgo dumps - 64bit vs 32bit OS? [message #1044999 is a reply to message #1037467] Fri, 19 April 2013 16:13 Go to previous messageGo to next message
Justin Griffin is currently offline Justin GriffinFriend
Messages: 3
Registered: April 2013
Junior Member
Glyn,

I work with "joel mann", and we've been using Virgo since the dm Server days with great success. Here's the short on our system:



  1. We deploy our app with a plan file, which has grown to about 35 bundles listed (10 are "snaps" bundles)
  2. We leave this plan file in the "/pickup" directory.
  3. On slower machines, we experience longer package resolution time, but never thought twice about it until this problem.


We do not currently list any of our bundles in the "initialArtifacts", however I do declare the "repository:plan/org.eclipse.virgo.snaps" core there.

Here's what I found, and how I "patched" the problem for ourselves. The code I reference, I got by cloning http://git.eclipse.org/gitroot/equinox/rt.equinox.framework.git

I narrowed it down to the way the "platform properties" (which contains the "osgi.usesTimeout" property we are now specifying) are passed around inside a "org.eclipse.osgi.internal.resolver.StateImpl" bean. Before this bean is passed for use, it is serialized and then deserialized (as a way to make a reference "copy") by "org.eclipse.osgi.internal.resolver.StateWriter#writeStateDeprecated(StateImpl state, DataOutputStream out)":
63  private void writeStateDeprecated(StateImpl state, DataOutputStream out) throws IOException {
64       out.write(StateReader.STATE_CACHE_VERSION);
65       if (writePrefix(state, out))
66           return;
67       out.writeLong(state.getTimeStamp());
68       // write the platform property keys
69       String[] platformPropKeys = state.getPlatformPropertyKeys();
70       writePlatformProp(platformPropKeys, out);
71       Dictionary<Object, Object>[] propSet = state.getPlatformProperties();
72       out.writeInt(propSet.length);
73       for (int i = 0; i < propSet.length; i++) {
74          Dictionary<Object, Object> props = propSet[i];
75           out.writeInt(platformPropKeys.length);
76           for (int j = 0; j < platformPropKeys.length; j++)
77               writePlatformProp(props.get(platformPropKeys[j]), out);
78       }
79       BundleDescription[] bundles = state.getBundles();
80       StateHelperImpl.getInstance().sortBundles(bundles);
81       out.writeInt(bundles.length);
82       if (bundles.length == 0)
83           return;
84       for (int i = 0; i < bundles.length; i++)
85           writeBundleDescription(bundles[i], out, false);
86     out.writeBoolean(state.isResolved());
87     // save the lazy data offset
88     out.writeInt(out.size());
89     for (int i = 0; i < bundles.length; i++)
90         writeBundleDescriptionLazyData(bundles[i], out);
91 }


The subtle bug is on line 69, where the call to state.getPlatformPropertyKeys() only returns a strict enumerated list of "standard" property keys, and not all of the keys of all of the properties that have been added. Therefore, when the state is reinflated by the reflexive StateReader class, only the standard properties are reconstituted (see line 162 below).

150 private boolean readStateDeprecated(StateImpl state, DataInputStream in, long expectedTimestamp) throws IOException {
151     if (in.readByte() != STATE_CACHE_VERSION)
152         return false;
153     byte tag = readTag(in);
154     if (tag != OBJECT)
155         return false;
156     int index = in.readInt();
157     long timestampRead = in.readLong();
158     if (expectedTimestamp >= 0 && timestampRead != expectedTimestamp)
159         return false;
160     addToObjectTable(state, index);
161     // read the platform property keys
162     String[] platformPropKeys = (String[]) readPlatformProp(in);
163     state.addPlatformPropertyKeys(platformPropKeys);
164     int numSets = in.readInt();
165     Dictionary<?, ?>[] platformProps = new Dictionary[numSets];
166     for (int i = 0; i < numSets; i++) {
167         Hashtable<Object, Object> props = new Hashtable<Object, Object>(platformPropKeys.length);
168         int numProps = in.readInt();
169         for (int j = 0; j < numProps; j++) {
170             Object value = readPlatformProp(in);
171             if (value != null && j < platformPropKeys.length)
172                 props.put(platformPropKeys[j], value);
173         }
174         platformProps[i] = props;
175     }
176     state.setPlatformProperties(platformProps);
177     numBundles = in.readInt();
178     if (numBundles == 0)
179         return true;
180     for (int i = 0; i < numBundles; i++) {
181         BundleDescriptionImpl bundle = readBundleDescription(in);
182         state.basicAddBundle(bundle);
183         if (bundle.isResolved())
184             state.addResolvedBundle(bundle);
185     }
186     state.setTimeStamp(timestampRead);
187     state.setResolved(in.readBoolean());
188     in.readInt(); // skip past the old offset
189     if (lazyLoad)
190         return true;
191     for (int i = 0; i < numBundles; i++)
192         readBundleDescriptionLazyData(in, 0);
193     return true;
194 }


PATCH SOLUTION

So, the simplest solution I could come up with was to change the call in StateWriter on line 69 to call a new method I wrote to return ALL property keys:
63  private void writeStateDeprecated(StateImpl state, DataOutputStream out) throws IOException {
...
68     // write the platform property keys
69     String[] platformPropKeys = getPlatformPropertyKeys(state);
70     writePlatformProp(platformPropKeys, out);
...
91  }

Note: The big difference here is I call "StateImpl#getPlatformProperties()" to get all of the keys, instead of "StateImpl#getPlatformPropertyKeys()":
    private String[] getPlatformPropertyKeys(StateImpl state) {
        List<String> toReturn = new ArrayList<String>();

        // get all the properties
        Dictionary[] props = state.getPlatformProperties();
        if (props != null && props.length > 0) {
            Enumeration keys = props[0].keys();
            while (keys.hasMoreElements()) {
                Object key = keys.nextElement();
                if (key != null) {
                    toReturn.add(String.valueOf(key));
                }
            }
        }

        return toReturn.toArray(new String[toReturn.size()]);
    }


I applied this minor change, and rebuilt the "org.eclipse.osgi" bundle. I then took only the "org.eclipse.osgi.internal.resolver.StateWriter.class" file and replaced the one in the "org.eclipse.osgi" bundle in Virgo. This was easier than attempting to rebuild the exact same version of "org.eclipse.osgi", however it forced me to "unsign" the jar (all of the equinox jars appear to be signed).

We are currently replacing this bundle with our "patched" vesion in the Virgo distribution when we assemble our distribution packaging, and have not had the issue since.

If it helps, I've attached a git patch file with my change.

Thanks!
-Justin
Re: Virgo dumps - 64bit vs 32bit OS? [message #1047478 is a reply to message #1044999] Tue, 23 April 2013 09:02 Go to previous messageGo to next message
Glyn Normington is currently offline Glyn NormingtonFriend
Messages: 1222
Registered: July 2009
Senior Member
Hi Justin

Thanks for sharing this! Please could you open a couple of bugzillas to capture the patches - one for Equinox and one for Virgo. I suspect there will be more debate on the Equinox side, but let's see. Smile

Regards,
Glyn
Re: Virgo dumps - 64bit vs 32bit OS? [message #1059845 is a reply to message #1047478] Tue, 21 May 2013 21:28 Go to previous message
Justin Griffin is currently offline Justin GriffinFriend
Messages: 3
Registered: April 2013
Junior Member
Thanks Glyn,

I have submitted two bugzilla tickets, as you suggested.

1) Equinox - https://bugs.eclipse.org/bugs/show_bug.cgi?id=408636

2) Virgo - https://bugs.eclipse.org/bugs/show_bug.cgi?id=408637

I linked the latter to the former.
Previous Topic:Bundle scopes and Config Admin service
Next Topic:SpringIntegration 2.2.3.RELEASE + Maven + Virgo
Goto Forum:
  


Current Time: Fri Nov 21 02:33:29 GMT 2014

Powered by FUDForum. Page generated in 0.01778 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software