Home » Modeling » EMF » EPackage.Registry concurrency issues
| EPackage.Registry concurrency issues [message #428863] |
Wed, 01 April 2009 07:45  |
 Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
Hi,
I've been experiencing concurrency issues lately when loading models from
XMI resources in multiple threads (Eclipse Jobs) at the same time. The
XMLHandler produces ClassNotFoundExceptions when instantiating the classes.
There is no other error message. It's just that someone down the call stack
simply returns null.
In accordance with Heisenberg's uncertainty principle, the system behaves
differently under observation. So if I debug the code, everything works
fine. Scheduling the Jobs to run sequentially also does the trick. So I've
settled with that as a workaround.
Out of curiosity, I'd still like to know the underlying cause of the problem
and I investigated a bit into the EMF codebase. So here's the situation: My
code is running in an Equinox runtime, all bundles are installed properly,
all EPackages are registered via extensions. Technically, everything is
fine.
Now I start a couple of Jobs that each reads one model that resides in the
workspace. Each thread uses its own resource set. All models are based on a
small number (3 or 4) of metamodels. So several Jobs ask the
EPackage.Registry for EPackages of the same URIs, basically at the same
time. Being registered via extensions, the Registry contains
EPackage.Descriptors. Upon first access, the Descriptors lazily resolve the
actual EPackages and EFactories. Since the default EPackage.Registry
implementation is not thread-safe, I'm guessing that something goes wrong
here and some threads receive not-fully-initialized EPackages.
_If_ that is the case, overriding the default EPackage.Registry with one
that is synchronized should solve these problems, right? Before I do that,
though, I'd like to hear your opinion if my logic is flawed or if there are
other possible concurrency hotspots in this scenario.
Thanks,
Achim
|
|
|
| Re: EPackage.Registry concurrency issues [message #428876 is a reply to message #428863] |
Wed, 01 April 2009 11:39  |
Ed Merks
Messages: 33139
Registered: July 2009 |
Senior Member |
|
|
Achim,
Comments below.
Achim Demelt wrote:
> Hi,
>
> I've been experiencing concurrency issues lately when loading models from
> XMI resources in multiple threads (Eclipse Jobs) at the same time.
There is nothing in EMF to make things that affect common data
structures be thread safe.
> The
> XMLHandler produces ClassNotFoundExceptions when instantiating the classes.
>
Stack traces are helpful. You mean the XMI one of the java.lang one?
> There is no other error message. It's just that someone down the call stack
> simply returns null.
>
Again, stacks are helpful to see.
> In accordance with Heisenberg's uncertainty principle, the system behaves
> differently under observation. So if I debug the code, everything works
> fine. Scheduling the Jobs to run sequentially also does the trick. So I've
> settled with that as a workaround.
>
> Out of curiosity, I'd still like to know the underlying cause of the problem
> and I investigated a bit into the EMF codebase. So here's the situation: My
> code is running in an Equinox runtime, all bundles are installed properly,
> all EPackages are registered via extensions. Technically, everything is
> fine.
>
That sounds good, but I've also never heard of a problem like this...
> Now I start a couple of Jobs that each reads one model that resides in the
> workspace. Each thread uses its own resource set. All models are based on a
> small number (3 or 4) of metamodels. So several Jobs ask the
> EPackage.Registry for EPackages of the same URIs, basically at the same
> time. Being registered via extensions, the Registry contains
> EPackage.Descriptors. Upon first access, the Descriptors lazily resolve the
> actual EPackages and EFactories. Since the default EPackage.Registry
> implementation is not thread-safe, I'm guessing that something goes wrong
> here and some threads receive not-fully-initialized EPackages.
>
I'm not sure how that's possible though. It should be thread safe from
a read-only point of view...
> _If_ that is the case, overriding the default EPackage.Registry with one
> that is synchronized should solve these problems, right? Before I do that,
> though, I'd like to hear your opinion if my logic is flawed or if there are
> other possible concurrency hotspots in this scenario.
>
I know EMF has been in use for a great many years and I've never heard
of a problem that was attributed to this cause. If there is an
underlying thread safety problem here, I'd want to fix it in EMF
itself. I know it's very hard to debug threading problems but here is a
very good trick. Set breakpoints at the place you suspect has the
problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
until both threads are at this breakpoint. Then step each thread
through the logic. If there is a problem, it's most likely to be here
in the resolution of the package descriptor.
public EPackage getEPackage()
{
// First try to see if this class has an eInstance
//
try
{
Class<?> javaClass =
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
Field field = javaClass.getField("eINSTANCE");
Object result = field.get(null);
return (EPackage)result;
}
I don't think it should be possible to get back a class that hasn't been
initialized. I think a class loader that returns a class that isn't
completely initialized is doing something bad. After all, how is one
supposed to use a class that might not be in a good state to be used
yet? Yet from your description, this sound likely to be at the root of
the problem. Another perhaps more likely cause is that there are
packages that aren't properly registered and hence when first accessed
add an entry to the global registry invalidating my comment/assumption
about read-only thread safety.
I do highly recommend you find the cause rather than just try to fix the
symptom.
> Thanks,
> Achim
>
>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EPackage.Registry concurrency issues [message #428877 is a reply to message #428876] |
Wed, 01 April 2009 12:09 |
Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
Ed,
Thanks for your comments. Below you'll find and exception stack trace that
occurred when loading a file ending with "*.modelgen". It's an XMI
ClassNotFoundException, not a java.lang one. In the particular log file I'm
looking at right now, I see this stack trace three times logged within one
second. However, there are 42 such "*.modelgen" files in the workspace, all
of which are loaded sooner or later, so the other 39 are ok. The files
themselves are ok. I can open them in the editor without any problems. Very
strange...
I don't think classloading is the issue here, since I would suspect other
exceptions (java.lang ones) popping up somewhere. I'll try out your
suggestion for debugging to find out more.
> Another perhaps more likely cause is that there are
> packages that aren't properly registered and hence when first accessed
> add an entry to the global registry invalidating my comment/assumption
> about read-only thread safety.
How would a package not be properly registered? All I can do is declare
these extensions, right?
<extension point="org.eclipse.emf.ecore.generated_package">
<package
uri = "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
class =
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
genModel = "model/modelgen.genmodel" />
</extension>
<extension
point="org.eclipse.emf.ecore.extension_parser">
<parser
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
type="modelgen">
</parser>
</extension>
Thanks,
Achim
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
2, 276)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
at
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
at
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
at org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
2, 276)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
at
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
at
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
at
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
at
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
at
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
at
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
... 11 more
Ed Merks wrote:
> Achim,
>
> Comments below.
>
> Achim Demelt wrote:
>> Hi,
>>
>> I've been experiencing concurrency issues lately when loading models from
>> XMI resources in multiple threads (Eclipse Jobs) at the same time.
> There is nothing in EMF to make things that affect common data
> structures be thread safe.
>> The
>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>> classes.
>>
> Stack traces are helpful. You mean the XMI one of the java.lang one?
>> There is no other error message. It's just that someone down the call
>> stack simply returns null.
>>
> Again, stacks are helpful to see.
>> In accordance with Heisenberg's uncertainty principle, the system behaves
>> differently under observation. So if I debug the code, everything works
>> fine. Scheduling the Jobs to run sequentially also does the trick. So
>> I've settled with that as a workaround.
>>
>> Out of curiosity, I'd still like to know the underlying cause of the
>> problem and I investigated a bit into the EMF codebase. So here's the
>> situation: My code is running in an Equinox runtime, all bundles are
>> installed properly, all EPackages are registered via extensions.
>> Technically, everything is fine.
>>
> That sounds good, but I've also never heard of a problem like this...
>> Now I start a couple of Jobs that each reads one model that resides in
>> the workspace. Each thread uses its own resource set. All models are
>> based on a small number (3 or 4) of metamodels. So several Jobs ask the
>> EPackage.Registry for EPackages of the same URIs, basically at the same
>> time. Being registered via extensions, the Registry contains
>> EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
>> the actual EPackages and EFactories. Since the default EPackage.Registry
>> implementation is not thread-safe, I'm guessing that something goes wrong
>> here and some threads receive not-fully-initialized EPackages.
>>
> I'm not sure how that's possible though. It should be thread safe from
> a read-only point of view...
>> _If_ that is the case, overriding the default EPackage.Registry with one
>> that is synchronized should solve these problems, right? Before I do
>> that, though, I'd like to hear your opinion if my logic is flawed or if
>> there are other possible concurrency hotspots in this scenario.
>>
> I know EMF has been in use for a great many years and I've never heard
> of a problem that was attributed to this cause. If there is an
> underlying thread safety problem here, I'd want to fix it in EMF
> itself. I know it's very hard to debug threading problems but here is a
> very good trick. Set breakpoints at the place you suspect has the
> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
> until both threads are at this breakpoint. Then step each thread
> through the logic. If there is a problem, it's most likely to be here
> in the resolution of the package descriptor.
>
> public EPackage getEPackage()
> {
> // First try to see if this class has an eInstance
> //
> try
> {
> Class<?> javaClass =
>
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
> Field field = javaClass.getField("eINSTANCE");
> Object result = field.get(null);
> return (EPackage)result;
> }
>
> I don't think it should be possible to get back a class that hasn't been
> initialized. I think a class loader that returns a class that isn't
> completely initialized is doing something bad. After all, how is one
> supposed to use a class that might not be in a good state to be used
> yet? Yet from your description, this sound likely to be at the root of
> the problem. Another perhaps more likely cause is that there are
> packages that aren't properly registered and hence when first accessed
> add an entry to the global registry invalidating my comment/assumption
> about read-only thread safety.
>
> I do highly recommend you find the cause rather than just try to fix the
> symptom.
>
>> Thanks,
>> Achim
>>
>>
|
|
|
| Re: EPackage.Registry concurrency issues [message #428885 is a reply to message #428877] |
Wed, 01 April 2009 13:12 |
Ed Merks
Messages: 33139
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
--------------090201030008020404020506
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Achim,
Comments below.
Achim Demelt wrote:
> Ed,
>
> Thanks for your comments. Below you'll find and exception stack trace that
> occurred when loading a file ending with "*.modelgen". It's an XMI
> ClassNotFoundException, not a java.lang one.
That's what I would have assumed....
> In the particular log file I'm
> looking at right now, I see this stack trace three times logged within one
> second. However, there are 42 such "*.modelgen" files in the workspace, all
> of which are loaded sooner or later, so the other 39 are ok. The files
> themselves are ok. I can open them in the editor without any problems. Very
> strange...
>
Indeed.
> I don't think classloading is the issue here, since I would suspect other
> exceptions (java.lang ones) popping up somewhere. I'll try out your
> suggestion for debugging to find out more.
>
Well, as I said, it would be very odd to see a null coming back for the
eINSTANCE. That should be impossible. But if you look at the rest of
the logic for getEPackage, I don't see how multiple threads executing
that same code, could ever have one of them returning null...
>
>> Another perhaps more likely cause is that there are
>> packages that aren't properly registered and hence when first accessed
>> add an entry to the global registry invalidating my comment/assumption
>> about read-only thread safety.
>>
>
> How would a package not be properly registered?
If the uri in the plugin.xml doesn't actually match
ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
it's possible for it to be out of date.
Now that I think of it some more, if the package was null, you ought to
see a package not found exception. You've not indicated which version
of EMF you are using, but the problem sounds like this one which was
fixed a very long time ago.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
> All I can do is declare
> these extensions, right?
>
> <extension point="org.eclipse.emf.ecore.generated_package">
> <package
> uri = "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
> class =
> " de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
> genModel = "model/modelgen.genmodel" />
> </extension>
> <extension
> point="org.eclipse.emf.ecore.extension_parser">
> <parser
> class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
> type="modelgen">
> </parser>
> </extension>
>
>
> Thanks,
> Achim
>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
> 'GeneratorConfiguration' not found.
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
> 2, 276)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
> at
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
> at
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
> at
> org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
> at
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
> at
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
> at
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
> at
> org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
> at org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
> at
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
> 'GeneratorConfiguration' not found.
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
> 2, 276)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
> at
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
> at
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
> at
> com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
> at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
> ... 11 more
>
>
> Ed Merks wrote:
>
>
>> Achim,
>>
>> Comments below.
>>
>> Achim Demelt wrote:
>>
>>> Hi,
>>>
>>> I've been experiencing concurrency issues lately when loading models from
>>> XMI resources in multiple threads (Eclipse Jobs) at the same time.
>>>
>> There is nothing in EMF to make things that affect common data
>> structures be thread safe.
>>
>>> The
>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>> classes.
>>>
>>>
>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>
>>> There is no other error message. It's just that someone down the call
>>> stack simply returns null.
>>>
>>>
>> Again, stacks are helpful to see.
>>
>>> In accordance with Heisenberg's uncertainty principle, the system behaves
>>> differently under observation. So if I debug the code, everything works
>>> fine. Scheduling the Jobs to run sequentially also does the trick. So
>>> I've settled with that as a workaround.
>>>
>>> Out of curiosity, I'd still like to know the underlying cause of the
>>> problem and I investigated a bit into the EMF codebase. So here's the
>>> situation: My code is running in an Equinox runtime, all bundles are
>>> installed properly, all EPackages are registered via extensions.
>>> Technically, everything is fine.
>>>
>>>
>> That sounds good, but I've also never heard of a problem like this...
>>
>>> Now I start a couple of Jobs that each reads one model that resides in
>>> the workspace. Each thread uses its own resource set. All models are
>>> based on a small number (3 or 4) of metamodels. So several Jobs ask the
>>> EPackage.Registry for EPackages of the same URIs, basically at the same
>>> time. Being registered via extensions, the Registry contains
>>> EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
>>> the actual EPackages and EFactories. Since the default EPackage.Registry
>>> implementation is not thread-safe, I'm guessing that something goes wrong
>>> here and some threads receive not-fully-initialized EPackages.
>>>
>>>
>> I'm not sure how that's possible though. It should be thread safe from
>> a read-only point of view...
>>
>>> _If_ that is the case, overriding the default EPackage.Registry with one
>>> that is synchronized should solve these problems, right? Before I do
>>> that, though, I'd like to hear your opinion if my logic is flawed or if
>>> there are other possible concurrency hotspots in this scenario.
>>>
>>>
>> I know EMF has been in use for a great many years and I've never heard
>> of a problem that was attributed to this cause. If there is an
>> underlying thread safety problem here, I'd want to fix it in EMF
>> itself. I know it's very hard to debug threading problems but here is a
>> very good trick. Set breakpoints at the place you suspect has the
>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>> until both threads are at this breakpoint. Then step each thread
>> through the logic. If there is a problem, it's most likely to be here
>> in the resolution of the package descriptor.
>>
>> public EPackage getEPackage()
>> {
>> // First try to see if this class has an eInstance
>> //
>> try
>> {
>> Class<?> javaClass =
>>
>>
> Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>
>> Field field = javaClass.getField("eINSTANCE");
>> Object result = field.get(null);
>> return (EPackage)result;
>> }
>>
>> I don't think it should be possible to get back a class that hasn't been
>> initialized. I think a class loader that returns a class that isn't
>> completely initialized is doing something bad. After all, how is one
>> supposed to use a class that might not be in a good state to be used
>> yet? Yet from your description, this sound likely to be at the root of
>> the problem. Another perhaps more likely cause is that there are
>> packages that aren't properly registered and hence when first accessed
>> add an entry to the global registry invalidating my comment/assumption
>> about read-only thread safety.
>>
>> I do highly recommend you find the cause rather than just try to fix the
>> symptom.
>>
>>
>>> Thanks,
>>> Achim
>>>
>>>
>>>
>
>
>
--------------090201030008020404020506
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Achim,<br>
<br>
Comments below.<br>
<br>
Achim Demelt wrote:
<blockquote cite="mid:gqvlgu$qnu$1@build.eclipse.org" type="cite">
<pre wrap="">Ed,
Thanks for your comments. Below you'll find and exception stack trace that
occurred when loading a file ending with "*.modelgen". It's an XMI
ClassNotFoundException, not a java.lang one. </pre>
</blockquote>
That's what I would have assumed....<br>
<blockquote cite="mid:gqvlgu$qnu$1@build.eclipse.org" type="cite">
<pre wrap="">In the particular log file I'm
looking at right now, I see this stack trace three times logged within one
second. However, there are 42 such "*.modelgen" files in the workspace, all
of which are loaded sooner or later, so the other 39 are ok. The files
themselves are ok. I can open them in the editor without any problems. Very
strange...
</pre>
</blockquote>
Indeed.<br>
<blockquote cite="mid:gqvlgu$qnu$1@build.eclipse.org" type="cite">
<pre wrap="">
I don't think classloading is the issue here, since I would suspect other
exceptions (java.lang ones) popping up somewhere. I'll try out your
suggestion for debugging to find out more.
</pre>
</blockquote>
Well, as I said, it would be very odd to see a null coming back for the
eINSTANCE. That should be impossible. But if you look at the rest of
the logic for getEPackage, I don't see how multiple threads executing
that same code, could ever have one of them returning null...<br>
<blockquote cite="mid:gqvlgu$qnu$1@build.eclipse.org" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> Another perhaps more likely cause is that there are
packages that aren't properly registered and hence when first accessed
add an entry to the global registry invalidating my comment/assumption
about read-only thread safety.
</pre>
</blockquote>
<pre wrap=""><!---->
How would a package not be properly registered? </pre>
</blockquote>
If the uri in the plugin.xml doesn't actually match
ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
it's possible for it to be out of date.<br>
<br>
Now that I think of it some more, if the package was null, you ought to
see a package not found exception. You've not indicated which version
of EMF you are using, but the problem sounds like this one which was
fixed a very long time ago.<br>
<blockquote><a
href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325">https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325</a><br>
</blockquote>
<blockquote cite="mid:gqvlgu$qnu$1@build.eclipse.org" type="cite">
<pre wrap="">All I can do is declare
these extensions, right?
<extension point="org.eclipse.emf.ecore.generated_package">
<package
uri = <a class="moz-txt-link-rfc2396E" href="http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore">"http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"</a>
class =
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
genModel = "model/modelgen.genmodel" />
</extension>
<extension
point="org.eclipse.emf.ecore.extension_parser">
<parser
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
type="modelgen">
</parser>
</extension>
Thanks,
Achim
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
2, 276)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
at
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
at
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
at
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
at org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
at
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
2, 276)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
at
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
at
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
at
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
at
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
at
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
at
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
at
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
... 11 more
Ed Merks wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Achim,
Comments below.
Achim Demelt wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I've been experiencing concurrency issues lately when loading models from
XMI resources in multiple threads (Eclipse Jobs) at the same time.
</pre>
</blockquote>
<pre wrap="">There is nothing in EMF to make things that affect common data
structures be thread safe.
</pre>
<blockquote type="cite">
<pre wrap=""> The
XMLHandler produces ClassNotFoundExceptions when instantiating the
classes.
</pre>
</blockquote>
<pre wrap="">Stack traces are helpful. You mean the XMI one of the java.lang one?
</pre>
<blockquote type="cite">
<pre wrap="">There is no other error message. It's just that someone down the call
stack simply returns null.
</pre>
</blockquote>
<pre wrap="">Again, stacks are helpful to see.
</pre>
<blockquote type="cite">
<pre wrap="">In accordance with Heisenberg's uncertainty principle, the system behaves
differently under observation. So if I debug the code, everything works
fine. Scheduling the Jobs to run sequentially also does the trick. So
I've settled with that as a workaround.
Out of curiosity, I'd still like to know the underlying cause of the
problem and I investigated a bit into the EMF codebase. So here's the
situation: My code is running in an Equinox runtime, all bundles are
installed properly, all EPackages are registered via extensions.
Technically, everything is fine.
</pre>
</blockquote>
<pre wrap="">That sounds good, but I've also never heard of a problem like this...
</pre>
<blockquote type="cite">
<pre wrap="">Now I start a couple of Jobs that each reads one model that resides in
the workspace. Each thread uses its own resource set. All models are
based on a small number (3 or 4) of metamodels. So several Jobs ask the
EPackage.Registry for EPackages of the same URIs, basically at the same
time. Being registered via extensions, the Registry contains
EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
the actual EPackages and EFactories. Since the default EPackage.Registry
implementation is not thread-safe, I'm guessing that something goes wrong
here and some threads receive not-fully-initialized EPackages.
</pre>
</blockquote>
<pre wrap="">I'm not sure how that's possible though. It should be thread safe from
a read-only point of view...
</pre>
<blockquote type="cite">
<pre wrap="">_If_ that is the case, overriding the default EPackage.Registry with one
that is synchronized should solve these problems, right? Before I do
that, though, I'd like to hear your opinion if my logic is flawed or if
there are other possible concurrency hotspots in this scenario.
</pre>
</blockquote>
<pre wrap="">I know EMF has been in use for a great many years and I've never heard
of a problem that was attributed to this cause. If there is an
underlying thread safety problem here, I'd want to fix it in EMF
itself. I know it's very hard to debug threading problems but here is a
very good trick. Set breakpoints at the place you suspect has the
problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
until both threads are at this breakpoint. Then step each thread
through the logic. If there is a problem, it's most likely to be here
in the resolution of the package descriptor.
public EPackage getEPackage()
{
// First try to see if this class has an eInstance
//
try
{
Class<?> javaClass =
</pre>
</blockquote>
<pre wrap=""><!----> Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
</pre>
<blockquote type="cite">
<pre wrap=""> Field field = javaClass.getField("eINSTANCE");
Object result = field.get(null);
return (EPackage)result;
}
I don't think it should be possible to get back a class that hasn't been
initialized. I think a class loader that returns a class that isn't
completely initialized is doing something bad. After all, how is one
supposed to use a class that might not be in a good state to be used
yet? Yet from your description, this sound likely to be at the root of
the problem. Another perhaps more likely cause is that there are
packages that aren't properly registered and hence when first accessed
add an entry to the global registry invalidating my comment/assumption
about read-only thread safety.
I do highly recommend you find the cause rather than just try to fix the
symptom.
</pre>
<blockquote type="cite">
<pre wrap="">Thanks,
Achim
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>
--------------090201030008020404020506--
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EPackage.Registry concurrency issues [message #428898 is a reply to message #428885] |
Wed, 01 April 2009 14:14 |
Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
--nextPart5903759.URsZ4UmYZ8
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7Bit
Ed,
This is wacky stuff! The key to the problem is concurrent access to the
package registry during the initialization of the generated EPackage. In the
debugger, it's very easy to produce the PackageNotFoundExceptions you
predicted. In some cases, I've actually seen those in my logs, too, so
you're not far off the mark. Here's how to do it:
1) Have two or more threads execute EPackage.Registry.getEPackage(...) for a
generated package registered via the extension point.
2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
3) Have another breakpoint in the generated PackageImpl's static init()
method.
4) With the first thread, step into the call to field.get(null). This
actually initializes the package class and thus calls the static init()
method. Execution should halt at the breakpoint.
5) With the second thread, step _over_ the call to field.get(null). This
time, the classloader apparently considers the class already initialized.
However, the eINSTANCE field is still null. So, yes, getEPackage() _will_
receive a null value for eINSTANCE.
When reading from an XMI file, this obviously produces a
PackageNotFoundException. But it gets even better!
Now I _don't_ halt execution at EPackageDescriptor.getEPackage(). Instead, I
only have a breakpoint in the package's static init() method directly
_after_ the the line "isInited = true;". I also have a conditional
breakpoint in XMLHandler.createObjectFromFactory(), with the condition
"factory != null && newObject == null".
Again, two or more threads reading from XMI. One thread halts at the
breakpoint in init(). The other thread halts at my conditional breakpoint.
Taking a look at the variables here, I see that I have indeed the factory I
want (my ModelgenFactoryImpl). It refers to the correct package (my
ModelgenPackageImpl). This package, however, has no name, no URI, and no
contents. In fact, its "isCreated" and "isInitalized" fields are false.
Both threads, the one the halted at init() and the one in
createObjectFromFactory() refer to the exact same ModelgenPackageImpl
instance. So it seems that the classloader does returns a non-null eINSTANCE
field if the thread is not halted in the debugger. But the package simply
hasn't been fully initialized yet. See attached screenshot.
Now, what can we do?
For the record, I'm using EMF 2.4. The generated code is up-to-date, NS_URIs
match.
Cheers,
Achim
Ed Merks wrote:
> Achim,
>
> Comments below.
>
> Achim Demelt wrote:
>> Ed,
>>
>> Thanks for your comments. Below you'll find and exception stack trace
>> that occurred when loading a file ending with "*.modelgen". It's an XMI
>> ClassNotFoundException, not a java.lang one.
> That's what I would have assumed....
>> In the particular log file I'm
>> looking at right now, I see this stack trace three times logged within
>> one second. However, there are 42 such "*.modelgen" files in the
>> workspace, all of which are loaded sooner or later, so the other 39 are
>> ok. The files themselves are ok. I can open them in the editor without
>> any problems. Very strange...
>>
> Indeed.
>> I don't think classloading is the issue here, since I would suspect other
>> exceptions (java.lang ones) popping up somewhere. I'll try out your
>> suggestion for debugging to find out more.
>>
> Well, as I said, it would be very odd to see a null coming back for the
> eINSTANCE. That should be impossible. But if you look at the rest of
> the logic for getEPackage, I don't see how multiple threads executing
> that same code, could ever have one of them returning null...
>>
>>> Another perhaps more likely cause is that there are
>>> packages that aren't properly registered and hence when first accessed
>>> add an entry to the global registry invalidating my comment/assumption
>>> about read-only thread safety.
>>>
>>
>> How would a package not be properly registered?
> If the uri in the plugin.xml doesn't actually match
> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
> it's possible for it to be out of date.
>
> Now that I think of it some more, if the package was null, you ought to
> see a package not found exception. You've not indicated which version
> of EMF you are using, but the problem sounds like this one which was
> fixed a very long time ago.
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>
>> All I can do is declare
>> these extensions, right?
>>
>> <extension point="org.eclipse.emf.ecore.generated_package">
>> <package
>> uri =
>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>> class =
>>
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>> genModel = "model/modelgen.genmodel" />
>> </extension>
>> <extension
>> point="org.eclipse.emf.ecore.extension_parser">
>> <parser
>>
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>> type="modelgen">
>> </parser>
>> </extension>
>>
>>
>> Thanks,
>> Achim
>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>> 'GeneratorConfiguration' not found.
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>> 2, 276)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>> at
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>> at
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>> at
>>
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>> at
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>> at
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>> at
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>> at
>>
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>> at
>> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>> at
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>> 'GeneratorConfiguration' not found.
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>> 2, 276)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>> at
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>> at
>>
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>> at
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>> at
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>> at
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>> at
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>> at
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>> at
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>> at
>>
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>> at
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>> at
>> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>> at
>>
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>> at
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>> ... 11 more
>>
>>
>> Ed Merks wrote:
>>
>>
>>> Achim,
>>>
>>> Comments below.
>>>
>>> Achim Demelt wrote:
>>>
>>>> Hi,
>>>>
>>>> I've been experiencing concurrency issues lately when loading models
>>>> from XMI resources in multiple threads (Eclipse Jobs) at the same time.
>>>>
>>> There is nothing in EMF to make things that affect common data
>>> structures be thread safe.
>>>
>>>> The
>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>> classes.
>>>>
>>>>
>>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>>
>>>> There is no other error message. It's just that someone down the call
>>>> stack simply returns null.
>>>>
>>>>
>>> Again, stacks are helpful to see.
>>>
>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>> behaves differently under observation. So if I debug the code,
>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>> does the trick. So I've settled with that as a workaround.
>>>>
>>>> Out of curiosity, I'd still like to know the underlying cause of the
>>>> problem and I investigated a bit into the EMF codebase. So here's the
>>>> situation: My code is running in an Equinox runtime, all bundles are
>>>> installed properly, all EPackages are registered via extensions.
>>>> Technically, everything is fine.
>>>>
>>>>
>>> That sounds good, but I've also never heard of a problem like this...
>>>
>>>> Now I start a couple of Jobs that each reads one model that resides in
>>>> the workspace. Each thread uses its own resource set. All models are
>>>> based on a small number (3 or 4) of metamodels. So several Jobs ask the
>>>> EPackage.Registry for EPackages of the same URIs, basically at the same
>>>> time. Being registered via extensions, the Registry contains
>>>> EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
>>>> the actual EPackages and EFactories. Since the default
>>>> EPackage.Registry implementation is not thread-safe, I'm guessing that
>>>> something goes wrong here and some threads receive
>>>> not-fully-initialized EPackages.
>>>>
>>>>
>>> I'm not sure how that's possible though. It should be thread safe from
>>> a read-only point of view...
>>>
>>>> _If_ that is the case, overriding the default EPackage.Registry with
>>>> one that is synchronized should solve these problems, right? Before I
>>>> do that, though, I'd like to hear your opinion if my logic is flawed or
>>>> if there are other possible concurrency hotspots in this scenario.
>>>>
>>>>
>>> I know EMF has been in use for a great many years and I've never heard
>>> of a problem that was attributed to this cause. If there is an
>>> underlying thread safety problem here, I'd want to fix it in EMF
>>> itself. I know it's very hard to debug threading problems but here is a
>>> very good trick. Set breakpoints at the place you suspect has the
>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>> until both threads are at this breakpoint. Then step each thread
>>> through the logic. If there is a problem, it's most likely to be here
>>> in the resolution of the package descriptor.
>>>
>>> public EPackage getEPackage()
>>> {
>>> // First try to see if this class has an eInstance
>>> //
>>> try
>>> {
>>> Class<?> javaClass =
>>>
>>>
>>
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>>
>>> Field field = javaClass.getField("eINSTANCE");
>>> Object result = field.get(null);
>>> return (EPackage)result;
>>> }
>>>
>>> I don't think it should be possible to get back a class that hasn't been
>>> initialized. I think a class loader that returns a class that isn't
>>> completely initialized is doing something bad. After all, how is one
>>> supposed to use a class that might not be in a good state to be used
>>> yet? Yet from your description, this sound likely to be at the root of
>>> the problem. Another perhaps more likely cause is that there are
>>> packages that aren't properly registered and hence when first accessed
>>> add an entry to the global registry invalidating my comment/assumption
>>> about read-only thread safety.
>>>
>>> I do highly recommend you find the cause rather than just try to fix the
>>> symptom.
>>>
>>>
>>>> Thanks,
>>>> Achim
>>>>
>>>>
>>>>
>>
>>
>>
--nextPart5903759.URsZ4UmYZ8
Content-Type: image/png; name="variables.png"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="variables.png"
iVBORw0KGgoAAAANSUhEUgAAAycAAAN9CAIAAAAG3pFvAAAACXBIWXMAAA7E AAAOxAGVKw4bAAAg
AElEQVR4nOzdd1gTyRsH8EGRUKS3ICqgFLEgCtiwgJ4dRSzYu6Igehawoth7 9+7sZ+/Yzt7O3rBz
AkpHmiQBBQQCJJvfH6trfkBCKCYRv58nj8/s7MzsmzWBl9nJRuUTN42I4X5M iQx7TgAAAACgEuwd
nI3Z5uI1KkzWxeRbPT29FRCa8rly/iRdKHnW4Ee7d/08IaRjN09FBwIA8Gsp KioSCASVHERVVbVW
rVpVEs9Pjf5dJp5FfM266JSru4cXIUTwJVOBISoP1doGdOHaxbN0AemXnN27 fh6JFwCAPEW9i3hw
9990zkehQFiB7jVVa5qamrXv6G7bqHGVx/aTunf9PJM/qHziptEpV7eefQRf MhQdmzJSrW1IF65f
uUCQe8kXEi8AALlJTIg/e+p4W9f2bLZZjZo1KzACJRR+/Jj2+OEDr0FDLCyt qjzCnxSTeKlE/fc8
Mux51+49i3J4io5K2dXSNiKE3Lh2BYmXPCHxAgCQj317drR0cjIzq+wvuLS0 lJcvXoydMLlKoqoe
6MRLld4QCQUiYZFiA1J+hZ/TCCG//db15s0bkWHPkXsBAEB18jEtuV5dr4rN comrV7f+5YvnqySk
akY1Mux5F3e3ws+pio7kp1H4ObWLuxsh5NbtO4QQJF4/WsdunpjuAgCQA6FA qKZWFavga9ao2LKw
aoz+XfZ1rovCRFd5FHxOJYR0cXdD4gUAANWJSKToCKo1VUKIiBKIBMi6yo3P S+zcqf2/dx8QJF4A
AKAcCosEmZ8+19bSqq2lUaEBfsq0S9/YTMreYrcmrdruJftKaa9Kfp5FXcYt PLivLspeLwd8boJ7
+za3HzwhlUi8Hr5LCd59I4uXpmtktmRiV9dGik/grGzs46MjFR0FAACUT2GR gMvLyMnJUVNTI0Tj
S25+Vna2uZmp7CPIPte19WzcNK8GFYmSkCsPYqU36Nm+YbkG/MRN0zc2E5WI XkVFpcyUi1ayLzNC
uSKR7mvWxVxhPPi86PjFh5cXu4k36rX4zhAP11HOP/aOZ6bOXunPz0pvI+lK aFVdITV19qIL4pGU
WsnI58a7tXW+8/g5qVDide1NwvK/LvpPGlSXbZz8kTt306kgP4/uzS0rELyy QeoGACBndMpFl7/k
5ienpOjp6ZVzDJnSrtCo7OzcotCorFa2OuUc/6vm7q6Sdr25/bACA37ipqmo qIgnT7KnXJVU7KBS
WtJXGIvoua5Dr8jxiw+HeLgWm/oa4uF6/OJDQlxHtih9lNgPaV6Tg5+d+5P1 /6vwsnJy+00KPrdz
ia62VplBf3x6sswpN0kNqnCujj53KioqH5+eJISwW3szNZKOks+N69S6+d2n FUm81h284z2sD1HX
Sv6cR9S1vIf1WXfwdvcNY2Tpa2Vjz5R1dLTbtG4VNH9uvbp1yxXAj1NmyoW0 DACgCn3k8AghDaws
4+IT8vPzORyOiYmJgV75siIZ57puPU8vKKJuPU93sdG59y73dVR2Pr9gjrdl uY515fBpSbv2nvhI
CBk/WGJmVirxxKu8KVcF5rSYa4tMX0kTZgxVQgglFIiEhYSQqORcQsgIh8Le y54N8XAd4VB4OEzt
+MWHF+e7HL9IopI/iRxKT54amBu2crA9dv7G6P6/idcfOH21ZZMGOpq16PEr T9I4VTX+9wFFIvok
ip9BKUfJ58R2cG5y/3m5E68sXhqnoOa792kFRQK1WjUMtbSyeOV4ldBZi0gk ysjM3Lf/kJ//7xfO
SXwRAwBANVZbS0tFhaLLOTk5dc3N1dXVCgoLBYKiwoKCYo319Q0kDFN63hB9 czAhxOa3E4SQ43fT
8gqEhJC8AuHxu6lDOpl1bKQppa8UPUcMkLRLSk4mBZ14ERnWchXrVYFjlVRm 6laDfLvCSAmLXr6O
aOnYmBIW0ZNblxN06KkvSljU0rHxy9cRlLDIY+Uz5kH3oh/+I3rtOH5ZUFTI 1BTw8/eFXJ88pHt8
UuqowPXWXcZbdBozbMYaDi+DbmDWbsTOY5daek6t4zqS3qTrJbUnhPxx6Hyz 3n4NOo+bvnwHn5/P
1NOFoqKCFX8da9rL18p93KSFW7Nzcuj6mw9fuA2fU6/jaJf+vx8+f0s87GKP lPv7xDNWJuVSUVFJ
ub9PSkdKWJTPiWnfwi4y7Dn3Y4rs/0O6RmaiQr6ZrqalkU4d3dqiQr6ukbQ1 fZL+m40MDX+f6hcd
HUPXWNnY7913oG0Htwa2jQkhQqFwzfqNzq1dGzV1nDZ9Vm5uLt3sQ1LShEl+ TZo7NWrSfPS4ibyM
4l9O8N/b8DbtO+07cJAec8euPS5t2ts3azF77oKCwq85aGFh4bwFi5q1cGnW wmV+0KLCb/XMVJyV
jf3RYyfau3WxbezQu69XROQ7Zq+VjT3T7O69+9169bFt7NDercvxk6fKexIA AH55lI6OXmFhESFE
T09PT0+XxVLX1NTS0dEzMjYVf0gZQiQBIaTLxEfRNweLRKLBHdlLRtsSQpaM th3ckS2pvYxMWaU/
KuwTN00+FxbJ/58uWQ5dg9CfYRQWMZfPRMKioY0+D/Fw/evorSEerkMbfRbf dT6gCfOge9GPpg3r
WJgZXbr9lKk5e/2Ro72VVR3DMXM2TRjQ+c25jW/ObWhQz2TxtqPM4V6Gx1zd HZR0eye9SddLaf/k
1bubfwc/PraKm5m1bncIU08Xth74J+xd/NXdQa/Prlevpbrir+N0/e/Ld88a 2yfqyrbTWwJfvo0W
D7vkI/nOrmK5qoqKSvKdXdJ70Y98TrRr84blSrwCR7ndvXBVT02lvqGWbi1y 98LVwFFuhJCH71J+
m7XfZfSq32btf/iu7NEyMjP/+GuHo2NzpubNm7CL58/ERUUQQv7asSs8POLi +TMvnj5ksVir1q6n
24z38R03ZtTL0EfPnjxo2KDB8pWrxcf89/adseN9li1eNHb0KLom9NnzK5fO 379zk8vjbd6yja5c
v3Hzx/T02zev/nvjSmpq2oZNW0qG9+Rp6OkTx968eNq9W9f5QYvIt1m6+OhI 5iLjzIA5M36f+t/r
5yePHn79OkzGEwgAALRaqqqEkKzsbEKImpoaISQ/Pz8ltZQkQPacKObWEPpB b3aZ+Igpfx9txS76
UbGwa6iU/qiW6NX0ReJ3jhAJio7HGB+/+NBvWJe/jt4ixHWINfdrS0FRv81R TMtz023Fx5oypNuK
XWd7t3egN3eeuLZu1nCRoOjWniC6hlVTdc5Yj7bDFjKHWzTJy6C2OrNJF6S0 X+w7wFBbnRCy2Lf/
4IAt88b3Fe94/NL9w6v92fq1CSHzxvft6rNixVRvQoi6Wq10TiaP96mOsd7a GUMrcJsM2bvkf4xq
19TqUZislxq7N7esPcMrePd1+jOMq2d4uTYyl32JvfjSLl1dnVPHjjCbQfPn GBp8nUA+cSrkwN97
2GxTQsjswJm9+3gtXxJMCLlx5etnP1ks1uyAGe07dWG6Hzp89I/tO/fu3tHc oRlTuShonpGhIV0Y
NnLMnMBZhJDzFy4dP3Lga/3CBcNGjp43J7BYnEsXL9LT0yWE+EwYt+3P7aWe CpY6i8vhfsr8VKeO
2eqVy8o8dQAAIE6lRs0PScn0avr8/HwOl5efn6+ro0MIEREinsZImYsq4OeJ b3Ye+jchhKioECIS
fXlPiKjz0L8v7Rlo2+0gIST4QBQhZDEh2q6tcx4+LdZXRpW9Eb5ClXc12Nf7 ddHX6b5eRuxgRV9Y
/M0wLtPD9fjFh95TrV6+jmrp2JgS5p+Z+v3LLIt9crB1U8saNVQevoxo29zm 7vN3etqaDjbmlLDo
RUT8qr0XwmOS8/iFdIhMR2M9TfFB6LKU9uYmOnTZ3FiXm5nN1NOFj7zPbmOX ip8Lun7HorFbj1zb
ePCStpZ68OT+v7VpIuWMWHSfXuzlSK/xSry2WcZzmvcxqm3jBo9lTrxcG5nf /P/l87IvsWcmirKy
sg8cOrx42YojB/fRNSYmJkyz9HTOb917MZvMq+Tlq1er1qwPj4jMz88n///q 2fP3/gH9+4mnXIQQ
Zql+vbp1ebyvlyMzMjLq16tHl+vXq5uRkVkyTjrlIoSoq6sLBIJSz8P2P7Zu +3P75m1/aNfWXhQ0
r0tn91KbAQBAqbi8jJo1a9Kr6XNycnJycvT09HR1dehfauK/2KReAfy/NCJk 8/eZrQETgwkhp3cv
ad7/OF0zf3A9QgjZQHIePi3ZV0aGfRaWrMy48BP87V2B65j/d78u27p1Xr4m J+PrnPYrIiRRJCSD
6icO8qt7Mr4OIfG2dfVFwmzpw/kNctt+4mabppa7Qm75DnKjh/Vbvj9oYp9O Tna1NVm5+QVNBy4U
v2Qp3r3M9h9S0i3rGBFCklJ5RnraxcYxNtA5u9GfbahbbECHhmZ7Fo0RiUR3 nr+fvfn4syOLJMVv
2StQfC0X+fbSpBOvhMvrZDurJC/tfRu7hk9kTryKqcASe11dnUk+E/7cvrPU vcbGRmdPnTA1NSlW
7zd1+oJ5c9w6dqhdu3Zubm6zFi7MrhNHDw0dMVpXR2fSxPFMZVJysqWFBSEk OSXF0PDrRJqhoSFT
/yEp2cBA0grNMjR3aLZn518ikejuvfuBcxc8e3y/YuMAAPyCvuTm16xZ00Bf j6Wurqen9/nzZz09
vTpmbIqiSjaWknQVm7xxHHCCLrw+PZgQcnr3EqaGaVwU8HUVSsWuCmZfWhZf Yo4smV+hsZTe19X0
9LKkgeYx9Dr6YsuV6KmvgeYxZS5s6uJsk8b7fOZmaMbnLx0dG9CV/MJCNVWi VlP0IZUzd8sp8m0Z
FlMotiml/bJd5zMyP2dkfl6683zfTg7FOg7v0WrellMJyemCwoJ3sUn+qw7R 9VNXH4pKSCkqLKCE
AoGQkhI/c15UVFTiL6yMv7BS/PUny9Iu5pGX9q61tUF5F9fTKrDEPicnZ8/e fQ0blH7DuhHDhsxd
sDDxwweBQPA+Kmra9Fl0fUFBgTqLpcZiJSUnzw8KFu/CZpueOHrwVMjp7Tu+ X6pfvnJNRmZmRmbm
shWr+/XtQ1f26d1z6fJVvIwMXkbGkmUr+nr0IrLR0dGOjYtnNqdOnxUTE1tU JBCJRJQQX+AFAFAO
tbU0DA30WCwWEYnM2Kb2jezM2Kb0Gm+RiIhE5Gvh66bEtEtFpfQHIeT07iUt Bp4Qr6ELtdYfpB/F
2suoJiHWmqU8qqWv9+tiLtX1Z0f2n2BQ7NLhyQkGhERSsv0enNTfNXDL6XXT vJhBVvv3W7b7ki/3
qKmh9sR+rhfvhxW7MsigN6W0d7av123K5jx+YU/XJtOHuBUbZ1L/djtPU8MX 7EnPzGlgbujv3Ymu
/83FdvKKw4kfMxuYG22c0V/6LVXpNCv23BK6Wey5JUziVd57seamRbZqYB1a /hmvwFFuy/+6OHhk
fx1tzazs3GsXrgb5eZTaklnXpaGh0crZaduWjaU2m+wzUYXsGT5qLIfDtbKy 9PebTNevWbli+co1
Kf6/m5qa+EwYd+HSZfFeJiYmx48cHDJitEAonDrFlxDi7NSyR6++X3Jze/fq MXP6tK/RzpqxaPEy
ty7dCSG9e/YImDldxqfpM2F8X6+BeXl59HXS7l27TJ4yNfFDkpWV5cYNa2Uc BAAAaEVFAlVVtdS0
NEJIHTOzkoW8vByhQEgJBWqqUhZTlZ4xtRh4smTl8mMfiNi6rnLNdjG3QlWp IXunn57K+SM7XCw0
8tKjFR1JtaVpavMsMd/ewblciRe+JkjcvevnO3bzVMihAQB+Fp+zsnV0dKOi owkhtjY2JQtfcrIo
oYAQoifhZl3Lg+dN9p1S3uPWXLufLghnj2Eqd2z/M2jJKukd95542HPEALva pe/dvON0ee+SquTu
XT/P3CX1J/gexp9UbmqEU13bF+Wc8Sq5xB4AAECK/Px8HR0dWxtrQgghopIF 6SkXTaX8U0/UnHFf
+5a3JyFXDp++Uv5eP6+vn2FUdBjVXH56VMs6DV5WdHE9AABAmUQi0cePH6U0 0FRXk55yEUJUasjv
gl81m8qSxdf7dSk6jOqPz41zNDJ7/TMnXvjORAAAZVbHjF35QSrwdYQgu6+f YVR0GL+Ewqy05gZf
IsOe37t+vgIfbAQAAPihaqrWpASCbx96rPiDEghqSluw/+vCFUa5KsrhOugK CCFhP/OkFwAAVEts
s7ppH9Pq1Kns76a0j2lss7pVElI1g7kueRPkfRLkfWpWm4tJLwAAUCqdf+t+ 6+aNDx8ShIIikYiq
wEMoKPrwIeHWzRudf+uu6GejjLCuSzGEBV+aagoIIW/DnkeGPSeElPfWEgAA AFXLwtKqp4fnw/t3
rly5JBRU5G7VNVVrstl1evXpZ2FpVXbrXw+uMCoMVcQnhDRhJdGb4WEE6RcA AChW42aODi1bVXIQ
QVGhoKiwSuKpZlTOH9nRhJWExEt5RBTh7wMAAIBqCHNdSqdxrfiyGwEAAMBP JaLI6lf69iMAAAAA
xVFlSnV7BykwDgAAAAAlx8/7wpQzOGkf4qJyv2RRlKhkyxo1VLS0detb2Rqa mPFub6YrVcVbiI8F
AAAAAKXK+pQRG/W2ob2jjoFRjRql3NCfokTZmbzYyNeqtdSYSlxhBAAAACif R3eu2zRtqWdkXGrK
RQipUUNFz8jYpmnLR3euf6+UV3gAAAAA1YSmpnpt3TK+SpwQUlvXQFNTndlE 1gXww71+E6atZ6jo
KAAAoMqoqNSQNMslrkYNFRWV77kWsi4AAACAyhIUFUa8eCz99rDIugAAAAAq hRIKI148+cTjvH/9
jIhK+UgjDVkXwE/DysZe0SEAAEBxIpHofdiLnKxPhJDPmbyY8NeSWiLrAgAA AKi4uMiwTE4as5me
8iEp5l2pLVVLraVZ2dh36tB+395dKioq4pXx0ZFVFSjAL8ulbYeHd2+pqamJ V2Zn53gPHXHi6CFd
XR1FBQYAAOXSsHFzB5cO6uqa4pV8fl5ubnaxlmXMdeno6h46fLSKowMAQlo5 O50KOVOs8uix4y0c
myPlAgD4ueTmZmdkfBR/lEy5SJlZ1/IlwQcPH4mLK+X7mD8kJU2Y5NekuVOj Js1Hj5vIy8ig661s
7I8cPd7RvatdE4duvfo8f/Ey5MzZzl172jZ26N3X631UFN1MKBSuWb/RubVr o6aO06bPys3Nrdzz
BfjJ+E722fP3foqimJrCwqIDh49MnDBO0puLUWyBF7OJtxUAgDIra65LR3v5 0sUzAmYXFRUV2zXe
x3fcmFEvQx89e/KgYYMGy1euZnY9ePToxNFDb16E9uvrMWa8z507944c3Bf2 MrR3rx5zFyyi2/y1
Y1d4eMTF82dePH3IYrFWrV1fpc8LQNk1bdK4fv26V659v2fxhYuXHJs7NLCy lPLmkg5vKwAAZVb2
avo2rVu1bdN609Y/itXfuHKxXds2LBZLW1t7dsCMBw8eMbtWLltqZsZWV1cf N2Z0bm7usiXBzObb
t+F0mxOnQoIXLmCzTbW0tGYHzrx+/WYVPiuAn4LvJJ/de/5mNnfv3Tdp4gQi 9c0lHd5WAADKTNpq
esbM6b8PHDy0s1snZ6eWTOXLV69WrVkfHhGZn59PCBFfca+vr0cX1NXVi20K BAK6nJ7O+a17L6aL
eHeAX0Sb1q1UatR4/ORp2zat795/oK+v59jcgUh9c0mHtxUAgNw8vHa+vF1k yrrU1GptXLdmsv+0
86dPMpV+U6cvmDfHrWOH2rVr5+bmNmvhUq4DGxsbnT11wtTUpHzxAlQvUyb7 7Ni1p22b1nv27vOd
NJGuLPPNpaqqyufz6b9qPn36zNTjbQUAIB+vX4fNX7FJRFEikUgkov8ViUTU t+L3+q1rljZ20aZ7
yXq/LmvrhiOGDQleupypKSgoUGex1FispOTk+UHB5Q13xLAhcxcsTPzwQSAQ vI+KmjZ9VnlHAKgG
unR2/5iefvbcPxkZmR07tKcry3xz2Tey27Xnbz6fn57OCVq0mKnH2woAQJmV 4y6po0eO4HF5zOaa
lSuWr1zTxKHl0BGjnZxalPfAk30mtnJ2Gj5qbGOHltNmBHTt2qW8IwBUAyoq Kr4+E2fPWzBp4nim
ssw316oVy67fuNmshcuAwUPbtW3D1ONtBQCgzFTOH9nRuFY8IaRu7yB+3hdF xwNQDb1+E9ahk3vO
5+I3gAAAgJ/U1rXLZL/COMxFmxASUWSFbwQCAAAAkAeZVtMDAAAAgLjod+Ei EUVEREQokYgQkUhE
KBFFF77OfRFCP75C1gUAAABQbpYNrCmhkKKEFCWkhNS3glBEUXSBooQURRHy /SY+37MuoVAoFAoV
ETZANUdRQpGI4P0FAPCzq1mzZmW6f8+6CgsLS37tDwBUXmN7e156iqKjAACA yqq6rKuoqLCwsNLx
AAAAAFRP9O2pK+x71lVUWFiIuS4AAACAH+N71vX4xjkFxgEAAACg5Lp6DqlM 9+9ZV7+RkysdDAAA
AEC1lZv9uexGkuEuqQAAAADygKwLAAAAQB6QdQEAAADIA7IuAAAAAHmQmHXd vXdPnnEAAAAAVG8S
s66uXbsrKvFSY2ko5LgA8EMxb+0qfI+vXrN2wEBvGY+rcP0HDFq7br2iowAA hZF2hXHo0OFKNeOl
PD86AUAZ5Obmbtu2be2aVfSm8v+IWLN65ZYtW/Lz8xUdCAAohrSs69SpU/JJ vGT8WVlYgB9VUN1k
ZWXNmhVoY9tIU0vblG02YKD3v7dvizfYsHGThmbtDRs3MTUTJvpM8Z8q3sZv iv9En0mS6gkhaiyN
5o4tRSIRs0skEjk0b1Fy8qnUN6PSZjPnzp13dXVt2LAhvVn5HxFqLA3xh5Gx KV1/5erVLr911dbR
Y5vVGT5iZGpqqvR6Go+X0aChjfjZs7Gxad26zYULFysZJwD8pKRlXW5ubhVO vBb8u1NrdX/nPb4L
/t1ZifAAqrnhw0cWCYpuXL+ak/357X//DR0yeOXKVcxeiqJ27ty1fv3aXbt2 UxRFV27dsvnRo8cn
TpykN48fP/HkydOtWzZLqqc3NTU1Ll++zIx84cJFLS1NeTzDH+nipUv9PD2r cMDCgnzmsW7d2gnj
x9H1mzZunjZ1atKHxOio902bNh0ydJj0etrSZcsmTfIpdgjvQQMvXETWBfCL KuMzjBVLvAacDOLk
JJ/1GujXwpGTkzzgZJCUxvQfgvRflkzlrl27rW3stGrruLRq/eZNmHhLQsjV a9ccWzhp1daxtrH7
++99sgcGoGzu3L27csVyS0vLmjVrGhsbDRw44OaN68zeq9eu6evrTfHzMzQ0 uHb9a72mpubxY0cC
A2dHRUVFRUXNmTP3xPGjGhoakurpXgGzZq1bv4EZed369YEBAeWNNj4+3qv/ QANDY20dvT59PDlc
Ll2vxtIo9T1bUFAwYaKPnr5hvXoW4tN1DKFQGBS00LxuPV09g+EjRubk5DAD btm6zaqBNUtdWmr4
4sXLNm1aM5vMj4gyj1umwsLCP//8a9q0r3OH169f9fTsq6enq62tPf33aa9f v5FeTwh59+7d9es3
fp82tdjIrVu3evbseQVCAoBqoOw7R7i5uU2ZMqVr1+55eXmyjDjn1l8aqkIP G5t8ocBQQ8PDxlpD
VTjn1l+S2tMXBeg/LpnKu/fu3bt7m5Oe5unp6evnV6zLuHHjgxctzMzg3v73 5tPQUFmiAlBO7dq2
9fWb8uTJ01LX+uzcscvP15cQMnnSpB3bv08b29nZrVmzesjQ4UOGDl+/fp2N jY30ekKIl1c/Dof7
+PETQsjde/cyMjL79Sv3LFE/rwHTpvmnJH9ITkq0tbUNDJzN7Cr1Pbt48RIe lxcV9e75i2d3bt8p
OeDqNWtfvnwV+vRJclKihobG/AXf/0ILDQ0NffqkgC/tx056ejqbzS5ZL+m4 xS4gMo+SIxw+fKRT
p4516tQpVp+bm7vtjz87u7uXWT9nzrzFwYvU1dWLtaxbt25aWpqUJwUA1ZjK +SM7GteKJ4RYeC4W
36HG0qBXgdy5c2fQoEHHjh3p1LGjLCNqrPIK7tCuhsq3fE6FUBS15P6j/Hln JXVRY2mIp1xqLI2P
aakGBvqEkLy8PCNj07zcHPFmDRraBATM9Ozb19zcvHxPF0DJfP6ctX79+ouX LsfFxRkbG/fz9AwK
WqCvr0cIiY+Pb9+hY2xMtLq6Op/Pb2ht8/DBfUtLS6Zvm7btVFVVH9wvPg9d sp5+7+zZs/fylatn
Tp/y8Ojr5dVv/PhxzHtKvFBqnCWXTOXl5dnY2qUkJxHJ71lLq4a3bl6n111F R0c3aepQ7HDWNnaX
L12wtbUlhKRzOM5OLklJiXSDxIQ4MzMz6WdPR1c/g8epVauW+NOUclwZURTV zMHxdMjJRo0aidfT
J8fE2Pju3dvMYrJS62/euhUcvPjB/XsqKirFfr4VFRUZGZtmfc6UPR4AUB7M 9zBuXbssMHg1JRRS
lJCihJSQ+lYQiiiKLlCUkKKog3t3DnPRJoREFFmVMddV3pSLEFKntk5CRmZG TnZGTnZGTlZGdnZC
Rmad2jrlelb0j29CiKampkAgKLb35InjN27ccnZpZdeo8aVLl8o1MoBS0dPT Xb582etXL7I+Z16+
dCE3L3fYsOH0rp07d3G5PB1dfTWWho6uPpfL27lzF9Px4MFDIpGoqKjo2LHj 4gNKqieEjBw54sXz
5ydOnPzvv/9GjBguKSTxtU3FJqEfP37i3rmLvoGRGktDT9+Qx8tgdpX6nk1P T2fSRCsrq5LHSk1N
bdqsOT3hVK+eBXPJkhBSZspFCGGz2SkpKSXryzyudKdPn7G1tS2WchFCCgvy edx0Xz9f+jMKUupn
z567ds0aFRWVkoMnJyeXOj8HAL8CaVlXBVIuQkj3Bs1jMz+ZqmuZsrTYrNqm 6lqxGZ+7N2he6VC/
c3Z2OnsmJDUlefPmjb6+U6pwZABFUVFRsbOz27hh/f0HDwghfD7/0OHD0VHv mAtNtjwAACAASURB
VNQn6n3kwUOH+Hw+ISQyMnLhouDjx44eO3pk7rz579+/pweRVE9jsVhT/KeM HTfef6o/i8WqQJBD
hw338/VNiI8t4OfxuOniH4oslampaUJCAl1mCuLYbHZCfCzzHKVfTyzJyanl 06elrDGQdFwZrzCu
XbcuMGBWqUfU0dGZOWN6yYVZxerfvn3r3rkLM7j4IZ4+DXVxcS7PswSA6kNa 1lWBlIsQ8lfvmaqq
2vc+JOvWYhUIqXsfklVr1f6r90wpXfT0dKOiomQ/xIiRoyIjI4uKikQikUBY fCYM4CfyW9duISGn
0zkcoVCYnJy8YMHC1q1bEUJOnjzVyqWVhYUF09LS0tKppdOpUyF5eXlDh43Y tnWLlZVVgwYNNm/a
OHTYiPz8fEn14oebHRiQl5sjKaUoE5+fr67OUldXT0hI8PUr+w+eIYO9AwLn cLk8Dpc7a1ZgyQY+
PhN9fafExcUJBIK3b98OHzGyXPF49O59KiRE9uOWnMYrNplHCLly9aqmpma7 dm3FK8eOG//u3TuB
QPAxPT04eHGb1q2l1xcbXPwQIadPe/TuXa6nCQDVhrSsqwIpF+3KyHU2JjbT HtwNSUi0MbG5MnKd
9PYzZsxo09ZV9nsC9e3TZ5D3ED19w/nzg/bvx2cY4Se2YMH8EydPOjg0r62t 26GjW25e7tGjRwgh
23fsmDS5+E0HJk2auH3HjqnTfu/W9be+ffvQlV5e/dw6dZo67XdJ9VUY7a6d OwMC5+jpG/7WtXux
vKRUixcH6+vrWdvYtmzh1LFTKT9MZgcGtHNt1617T109g5GjxpT3NhBeXv0e P34cGxtb3uNKsXbt
uoBZxf9K9Ojde/iIUTq6+s5OLhmZmUeOHpZeL0lMTMyTx088PfuWKyQAqDYk rqa/e+9exVIuAAC5
WbN2XWjos9MhJxUdiEwGDPRu3brV7MBy37MDAJREJVfTq0oaFykXACi/ObNL uXCptH6W7BAAfpCy
79cFAAAAAJWHrAsAAABAHpB1AQAAAMgDsi4AAAAAeUDWBQAAACAPyLoAAAAA 5OH7nSOKCsr3XRwA
AAAAIDvMdQEAAADIA7IuAAAAAHlA1gUAAAAgD8i6AAAAAOQBWRcAAACAPChF 1nX/wUO7xg5aOoaK
DgTgZ1Xm2wfvrzLJcorWrd80eOiIyo8jH95Dhm/YuFnRUQDAd6plN6mQqNgP 8fHxiYkpFhbmVlZW
tg3rS2m8ICh44/q1vXv1+EHBAABUXm5e3h9/7fj35lV6U0vHMDc7Q7EhSbdi +dLfuvXy852soaGu
6FgAgJAfNNcV+iIsPj5eX1/fza29vr5+fHx86IswKe3DIyK6de1SgQMpz9+U ABWjpWPo1KqdSCRi
akQiUUuXtsr/2tbSMRR/VKB7FUZSVUNJ988/F13btWnYwIrerHzKVewcss0t 6fobN2/16tPP0KRO
XQvrcRMmc7k8uj46JraP5wBj07rGpnX7eA6IjoktdRxmfBvrhq1bOV+8dLmS cQJAVfm/rOv+/YfF
3r0lH/fvP5Q+YlTsh9ev35qZmbm5ufH5fHV1dTMzs9ev30bFfpDUhc8vqFWr VtU8IYCfjaaGxpWr
15jNi5euaGlqKjAe2eVmZzAPRcciD5evXO3bp3cVDih+AlevWj5u7Gi6fuu2 v6ZP80+Mj371/Imu
rs7ocRPo+lFjxrdp0yomKjwmKtzFxWn02AmlDiV+iIEDvC5fvlqFMQNAZfxf 1tWhg+vVS/8YGRne
vn1b9P9u375tZGR49dI/HTq4Sh8xLCzCzMyYz+cTQvjfmJkZh4VFlNqe/suM +RMtPiFh0ODhpmb1
DYzr9OvvzfyRV1RUNHdeUH0r2zr1Gmzd9mfJjoQQPr9gytTpbHNLtrml/7QZ fH4Bc4htf263tW9W
W9eoWw+PUyFnmKMnJSc3tG2cnZ1d7jMHUEVmTJ+2cdNWZnPjpi0zZ/zObEp6 VRcUFEzy9Tc2rWtl
3Wjzlm1Me6FQuGjxMosGdkam5qPHTvjy5Uuxw+Xn8ydO8qM7btq8lXn7SOqo pWO4Z+8++yaOeobs
tu3dwsL+k/REFP7m1dIx3L3n7ybNWuobmTm3dn30+MnhI8eat3ChIw8Pj2Ca bdq81bKhnbFp3cl+
UwsKCqT/BzFevnzdqpWL+OHogqT/C9kVFhbu2LHb328yvXnh/OluXX+rraVl bGy0YvmSJ09C6fqo
qKiAmdN1dXV1dXVnB8yKiooqc2QXF+fnL15WICQA+BGKX2Hs0MH18IF9gwYN vHPnDlN5586dQYMG
Hj6wr8yUixCSlJxCCMnNzaX/pTH1JdF/mTF/og30HubvNzkhLioh9p2tjfWc eUF0sxWr1kREvnv8
4E7k21cpKaklOxJClixbkZb28b/Xz8NePUtKSl66fCVzlOfPXzy6f/tLFm92 4MxVa9ZRFEXXr1q9
borvJB0dnTKfF8AP0s+zD5fLffI0lBBy//7DzMxMz74ezF5Jr+qly1fxeBkR b18/fXT/zt37TPt1
6ze9evX60f3bCbHv1dXVgxYtKXa4pctXZmdnR4a/fvbkwaPHT2XpeP/Bw1s3 rqQkxfX16O0/bYak
J6IMb97bd+5eu3IhJSlu8KCBXgMGX7t+49KFc6nJ8f37eYpH/uDh42dPHka8 fc3hcJetWC3pGRWT
zklns9kl6yX9X0i6YlByhKPHTnTo4FqnjlnJXf/+e6dVK2e63LNH942btmZn Z2dlZW3YtLlH925M
s7oW1nqGbMeWrdZv2CwUCr/Xm5unfUyT8QkCwI+mcv7Ijsa14gkhdXrMZmrv 3384YvTYU6dC3Nzc
ypVyEUJCzl4tKCgwMtLz8vI6e/YsXcnjfWaxWAO9Sl8vL2lRal5+vn0Tx8S4 94QQW/tmly6cs7Fu
KKWjtV2Tq5cvWDdsQAiJjont5eEZ/e4t3SzmfbiZ2dcflx3dfvOfMtl70MCY 2LheHp6vX4ZqamjI
8tQAqhz9Gv5734Gr166fPH7E02tQP88+Y8eMYl7bkl7VNo2aXr18gV5jFB0T 69iyFd3evonjP+dC
bGysCSEcDrd1uw7xMe+I2JvFplHT61cvWllaEkJi4+IdHJ3L7JicGKOvr08I ycvPNzO3zMpMJyVW
UxV7C8vzzcsMpaVjmJQQbWBgQAdgbFpXfFM88rDXz+lTR48TFflfyZBKMjSp 8zElkVkOIX5KS/2/
kBFFUS2d25w4dtjOzrbYrrCw/4YOH33u7Cn67KWmpnXp2vNDUhIhpH69ev/e vMqcGUKIUCiMjHwX
OHe+Q9Oma1avoCuLiorM6lry0kv/oxcAyquwoJAubF27LDB4NSUUUpSQooSU kPpWEIooii5QlJCi
qIN7dw5z0SaERBRZlb6anpnxWrx4cblSLkKIg0Pjt+GRhYWFhBAvLy8vL6/C wsK34ZEODo1l6f7k
aWjX7r1N2PW0dAyNTetmZHz94fXxY7qVpYX0vhwOl2ljZWnB4XCZXeI/m2YH zlq5ep1QKFyxYvWM
6dOQcoHCDR825MXLVydDTr8NDx82dLD4Lkmv6vR0jqVFfaaeaZ+alubo1Jqe U7GybsRc5mOkp3Pq
16tHly3q15OlI51yEUI0NTQEAgFTX2wtkTK8eekciw612KZ45Myps7Son57O kR4bg23KTklNLVkv
6f9CRmfOnrexsSmZct2//9B7yIgD+/cwCauP75RhwwanJsWlJsUNHeo9cbKf ePuaNWs2bdpk/97d
h44cYyqTU1LYpqXMzwGAQkj8DCOdeC1ZsqRcKRchxLZhfU/P3o8ev9i9ex/9 ePT4hadnb+k3j2CM
HD1uks+EmKjwL1m8jykJzGe72GzT+IRE6X1NTIwTEr+u2Y9PSDQ2Niq1We9e PdTU1BYtXvY09Nn4
b8tXARSIxWL5TfaZ6OM3xXcSi8US3yXpVW1qasLUMwVCCJttGv3uLZMMfckq nnWZmBjTkyWEkMQP
SbJ3LNNP9OZljpX4IcnExFjGXi1bOoY+e16yXtL/hYxXGDds2jJzxrRilSGn z46d4HP0yAFnp5ZM
5aNHT2YHzGTWdT169KRkMDVq1GAuwhJCnj177uzcsmQzAFAIaXeO6NDBlfMx qVwpF62Nc7OxY4ax
WKy34ZEsFmvsmGFtnJvJ2Dc/n6+urs5isRISE8XXYQwfNmRWwJyUlNSsrKw5 cxfQlbq6utHRMUyb
gf29Zs+Zz+XyuFxe4Ox5gwb0L/UQKioqswNmbt6ybc7sWWpqauV9dgA/wqyZ 07My08XX0dMkvaoH
DRwwd14Qj8fjcnmz58xn2k8YN9Z/2oy4+HiBQBAeHiH+MbevHQf0nzd/YUZG RkZGxtxvS69k6Vim
n+jNS586Ho83Z+4C70EDZezVq2ePM2fOlayX9H8hPhco6TOG167f0NTQaNum tXjl1j/+Clq4+NI/
Z1u2cBSvb9TIbv3GzVlZWVlZWes2bGxs34iuHzNuYkREpEAgiIuPn+Tr79Wv L9PlzNnzvXp2l/EJ
AsCPVsb9uir8CXbbhvUHevVYtWzuQK8eMs5y0bb/uXXuvCBjdr2evTzbiP0k mj93diM729btOjZu
1rJuvbp05fRp/u07dmb+dgxeFGRiYty0uVPT5k5mZuxFCxdIOkrNmjWtGzYY PmxIhZ4cgPxIelUv
Cpqvp6fbqElzlzauHcX+NJo18/d2bdv09vAyMq07ZrxPH4/idzpYuHC+llZt u8YOTq3atXJxZlYp
ldmxTD/Rm7dd29bOrV3tmzgaGOgvXDBPxl6enn0ePwmNjYsvVi/p/0IW6zds njG9+ETXvPkLk5KT
6du20Y8vubmEkH17dj54+KihbeOGto0fP3m6b+8uun3fPh5jJ0wyNDHv3rOP rY31xvVr6fqY2Lgn
T0P7eHgQAFAOpa+mr/a8hwwfOLC/98ABig4EQJGioqK9BgwO/+9nurNA5d+8 lbmn/PoNm589f37i
2OEKH12eBg8d0crFedbM6YoOBKD6qORq+h/1jUBKi6KoAwcPx8bFD+zvpehY ABRjztwF8+YGFhQU
zp2/sI9HL0WHIytlePMGzPqZMpifJTsE+HX8clmXtp6xRf36Bw/srVFDKb75 G0D+6lvUd2jRqoDP
79WrZ1CQrNfXFA5vXgD42f1yWdcv8tUlAFJM8Z00xXeSoqMot6p68+KHAAAo Cv5kBAAAAJAHZF0A
AAAA8oCsCwAAAEAekHUBAAAAyMP31fS1WBW8ISoAAADAr4C5X1fFYK4LAAAA QB6QdQEAAADIA7Iu
AAAAAHlA1gUAAAAgD8i6AAAAAOThl8661Fgaig4BoHyYFy1evQAAP51fOusC AAAAkJtqm3Vt2LhJ
Q7P2ho2bfsTgmGYAAACA8lK6rOvBoRYPDrWo5CAURe3cuWv9+rW7du2mKKpK AgMAAACoDOXKuh4c
auHuvdrde7UsiZdQKAwKWmhet56unsHwESNzcnKYXVevXdPX15vi52doaHDt +nWmvqCgYMJEHz19
w3r1LMSnweLj4736DzQwNNbW0evTx5PD5dL1aiyN9Rs21q1bX0/fcKLPpIKC AvJtokuNpcHMeEmK
RI2lsWXrNqsG1ix1TToqxxZOWrV1rG3s/v57XyXPFVQPaiyNXbt2W9vYadXW cWnV+s2bMKa+WDNF
RAcAAFVJibKuu/sc3fv6En4i4Se69/G9u99RevvVa9a+fPkq9OmT5KREDQ2N +QuCmF07d+zy8/Ul
hEyeNGnH9p1M/eLFS3hcXlTUu+cvnt25fYep7+c1YNo0/5TkD8lJiba2toGB s5ldD+4/ePnqRVTU
O046Z8mSpYSQwoJ8+l+6ID2S0NDQ0KdPCvh5hJBx48YHL1qYmcG9/e/Np6Gh FT9TUL3cvXfv3t3b
nPQ0T09PXz8/RYcDAAA/isr5Izsa14onhFh4LlZgHP/uatZ9QADhp3yvUje/ enpDF58wSV2sbewu
X7pga2tLCEnncJydXJKSEgkh8fHx7Tt0jI2JVldX5/P5Da1tHj64b2lpSQix tGp46+b1hg0bEkKi
o6ObNHVgMidGXl6eja1dSnISIUSNpREZ8ZZuHxMT07Vbj/i4GLpevKOkSNRY GokJcWZmZnSzBg1t
AgJmevbta25uXvkzBtWDGkvjY1qqgYE+ISQvL8/I2DQvN4eUeI0xmyULAAAg N7nZn+nC1rXLAoNX
U0IhRQkpSkgJqW8FoYii6AJFCSmKOrh35zAXbUJIRJGVssx1FRZSotwvoqxP 3x+5XwSFQildUlNT
mzZrTl/pq1fPgrksuHPnLi6Xp6Orr8bS0NHV53J5O3fuonelp6fT6RchxMrK ihnq8eMn7p276BsY
qbE09PQNebwMZhfT3tLSMj09vVyREEKYlIsQcvLE8Rs3bjm7tLJr1PjSpUuy nhqo7uiUixCiqakp
EAgUGwwAAPw4ypJ19fAPPxfyB9FtzDzOhWzr7h8upQubzU6Ij6Wv9BUW5NNX 8fh8/qHDh6Oj3jH1
Ue8jDx46xOfzCSGmpqYJCQl0d6ZACBk6bLifr29CfGwBP4/HTReJRMwuplli YqKJiYnskZTk7Ox0
9kxIakry5s0bfX2nyHpq4Jekqqqal/f1hZSRkanYYAAAoEooS9ZFCPGYGRly aBVdDjm82mPmO+nt
fXwm+vpOiYuLEwgEb9++HT5iJCHk5MlTrVxaWVhYMM0sLS2dWjqdOhVCCBky 2DsgcA6Xy+NwubNm
BTJt+Px8dXWWurp6QkKCr9//5UN0ey6XFxAwe+iQwXSlnp5uVFSU9EhKGjFy VGRkZFFRkUgkEggx
pQHSODg027hpc15eXmpq6hR/f0WHAwAAVUCJsi5CSL+5USf2rzyxf2W/Oe/L bDw7MKCda7tu3Xvq
6hmMHDWmn6cnIWT7jh2TJvsUazlp0sTtO3YQQhYvDtbX17O2sW3Zwqljp45M g107dwYEztHTN/yt
a/d27dqK93V1bdeiRUsbWzsDQ4Pg4EV05YwZM9q0dWU+VlZqJCX17dNnkPcQ PX3D+fOD9u/HZxhB
mh3bt//zzz9GxqYdO7m7u7kpOhwAAKgCyrKaXjlhwTIAAAAwqslqegAAAIDq DVkXAAAAgDwg65IG
lxcBAACgqiDrAgAAAJAHZF0AAAAA8oCsCwAAAEAekHUBAAAAyIMqUyoqKP17 bAAAAACg8jDXBQAA
ACAPyLoAAAAA5AFZFwAAAIA8IOsCAAAAkAdkXQAAAADygKwL4GeipWNYrAAA AD8LZF3yg1+TAAAA
v7Jqm3Vt3rJNR99k85ZtP2JwGfOnYs1yszN+RDAAAADwU1C6rOvpSdenJ10r OQhFUbv37FuzesWe
vfspiqqSwAAAAAAqQ7myrqcnXd29V7t7r5Yl8RIKhYsWL7NoYGdkaj567IQv X74wu67fuKmnp+s7
aaKBgf6Nm7eYei0dwz1799k3cdQzZLdt7xYW9p/0ej6/YMrU6WxzS7a5pf+0 GXx+Afk2g6WlY8hM
ZcUnJAwaPNzUrL6BcZ1+/b25XF6pzZhCqcNKCeP6jZvOrV31DNn2TRz3HzhU oVMLSkrKa7JYM0VE
BwAAVUmJsq5HR9u59/Ul/ETCT3Tv4/voWDvp7det3/Tq1etH928nxL5XV1cP WrSE2bVr99+TJ00k
hPhMHL9z117xXvcfPLx140pKUlxfj97+02ZIr1+ybEVa2sf/Xj8Pe/UsKSl5 6fKV5NuFwtzsDOaK
4UDvYf5+kxPiohJi39naWM+ZF1RqM0apw0oJY4KPX9D8uempidevXQx99lzm Mwo/B0mvSQAAqGZU
zh/Z0bhWPCGkTo/ZCozj/oHW3QcEEH7K9yp186unN3Qc/URSF/smjv+cC7Gx sSaEcDjc1u06xMe8
I4TEJyS4de7+PiJMXZ3F5xfYNXa4e/u6pYUFIURLxzA5MUZfX58Qkpefb2Zu mZWZLqXe2q7J1csX
rBs2IIREx8T28vCMfveWbi9pkVZefr59E8fEuPclmzGbUoYtNQy7xg4zpk/r 07uXuXmdSpxjUEZS
XpOlvnhKFgAAQG4KCwrpwta1ywKDV1NCIUUJKUpICalvBaGIougCRQkpijq4 d+cwF21CSESRlbLM
dRUWUqLcL6KsT98fuV8EhUIpXVLT0hydWtOX8KysG9HX9Qghu/fs4/F4hiZ1 tHQMDU3q8Hi83Xv2
Mb3oX2+EEE0NDYFAIL2ew+FaWVrQZStLCw6HW2okT56Gdu3e24RdT0vH0Ni0 bkZGGb8LpQxbahhH
Dx+4det2G9dOTR2cLl+5Kn1w+OlIek0CAEA1oyxZV5eJz86F/EF0GzOPcyHb Ok98JqULm20a/e4t
fQkvNzvjSxaPEMLnFxw5ejzi7SumPvy/l4ePHGPWTpWLiYlxQuIHuhyfkGhs bFRqs5Gjx03ymRAT
Ff4li/cxJUEkElXJsAynli1OnTjyIT5qw/rVuAL1i1BVVc3Lz6fLmZmZig0G AACqhLJkXYSQ7lNe
hBxaRZdDDq/uPuWl9PYTxo31nzYjLj5eIBCEh0eMHjuBEBJy+oyLs5NF/fpM M0sLi5YtHE+fOVuB
kAb295o9Zz6Xy+NyeYGz5w0a0J+u19XVjY6OYZrl5/PV1dVZLFZCYqJ4VlSs WZnDSjJm3MR3794X
FRWJRCKBQNr8H1QbzZo22bLlj7z8/NTUtGnTZyk6HAAAqAJKlHURQnrPeH1i /8oT+1f2nv6qzMaz
Zv7erm2b3h5eRqZ1x4z36ePRmxCyc9cen4njirWcOGHszl17KhBP8KIgExPj ps2dmjZ3MjNjL1q4
gK6fPs2/fcfOzMfKtv+5de68IGN2vZ69PNu0ac10L9aszGEl6d2r55Dho4zZ 9YIWLfl7z44KPBH4
6fy5bfOFS5fNzC27dO3ZqWMHRYcDAABVQFlW0wMAAAAouWqymh4AAACgekPW BQAAACAPyLoAAAAA
5AFZFwAAAIA8IOsCAAAAkAdkXQAAAADygKwLAAAAQB5UmVItlqYC4wAAAABQ csz9uioGc10AAAAA
8oCsCwAAAEAekHUBAAAAyAOyLgAAAAB5QNYFAAAAIA/IugB+JmosjWIFAAD4 WSDrqgL4/QcAAABl
qp5Z1/v3770HD2Gb1amtrdu2neuZM2crNo6M6VRhQX7FxgcAAIBfh9JlXQ8O tXhwqEVlRoiJiena
tXuXzp3D377N4HG2bN584uTJqgoPAAAAoGKUK+t6cKiFu/dqd+/VsiReQqEw KGihed16unoGw0eM
zMnJoeuXLlseEBgwaZKPoaEBi8Vq1crlxPFj9C4+nz/Z18/I2NTI2NTXbwqf z6fr1Vgau3bttrax
06qt49Kq9Zs3YeTbRJcaS4OZ8YqPj/fqP9DA0FhbR69PH08Ol8t0lzKOlFDV WBpbtm6zamDNUscX
A/yiJL1mis2z4io2AEA1oERZ1919ju59fQk/kfAT3fv43t3vKL396jVrX758 Ffr0SXJSooaGxvwF
QXT97X9vDxo0sNQuwcGL01LTIiLehof/l/QhafHiJd+Pfu/evbu3Oelpnp6e vn5+5Nt1w8KCfOYC
Yj+vAdOm+ackf0hOSrS1tQ0MnF3KsygxjpRQCSGhoaGhT58U8PNkPEtQ/ZT6 mgEAgOpH5fyRHY1r
xRNCLDwXKzCOf3c16z4ggPBTvlepm189vaGLT5ikLtY2dpcvXbC1tSWEpHM4 zk4uSUmJhBBNLe3s
rE+qqqolu1hYNrh187q1tTUhJDo6umu3HgnxsYQQNZbGx7RUAwN9QkheXp6R sWlebg5dL2nNVl5e
no2tXUpykngzSeNIClWNpZGYEGdmZlahcwbVgYyvPfHXWLECAADITW72Z7qw de2ywODVlFBIUUKK
ElJC6ltBKKIoukBRQoqiDu7dOcxFmxASUWRVSmqiEIWFlCj3C/ny6XuVUFdQ KJTSJTU1tWmz5sym
iooKXTA0MODyeGZsdskuHA7HysqKLltZWXE4HGYX/WuPEKKpqSkQCEo94uPH T+YvWPD69Zvc3Fzx
I4ordRxJoRJCkHKBLK89AACoBpTlCmMP//BzIX8Q3cbM41zItu7+4VK6sNns hPhY+gpgYUE+c5HO
vbP76dOnS+1iYmKSkJBAl+Pj442NjcsV5NBhw/18fRPiYwv4eTxuukgkkrGj pFABJFFVVc3L+/o6
ycjIVGwwAABQJZQl6yKEeMyMDDm0ii6HHF7tMfOd9PY+PhN9fafExcUJBIK3 b98OHzGSrl+0MGjd
2vV79uzNzPxUUFDw7NnzwUOG0ru8Bw2cNSuQw+VyuNyZMwMGew+Sfgg9Pd2o qChmk8/PV1dnqaur
JyQk+PpNkf2pSQoVQBIHh2YbN23Oy8tLTU2d4u+v6HAAAKAKKFHWRQjpNzfq xP6VJ/av7DfnfZmN
ZwcGtHNt1617T109g5GjxvTz9KTrra2tr127cu369Ub29oZGJr9Pnz7Y25ve tXTpEhNTE3v7Jvb2
TczqmC1Zslj6IWbMmNGmrSvz8bFdO3cGBM7R0zf8rWv3du3ayv68JIUKIMmO 7dv/+ecfI2PTjp3c
3d3cFB0OAABUAWVZTQ8AAACg5Cq5ml655roAAAAAqitkXQAAAADygKwLAAAA QB6QdQEAAADIA7Iu
AAAAAHlA1gUAAAAgD8i6AAAAAOTh+/cwFhXga2oAAAAAfhTMdQEAAADIA7Iu AAAAAHlA1gUAAAAg
D8i6AAAAAOQBWRcAAACAPCDrAgAAAJAHZF1VQEvHUNEhAAAAgLKrnllXVFT0 sBGj61pY6xuZdejU
5ey5fyo2jozpVG52RsXGL3k48UfFRqiSSIqNRsdTW9eIbW7Ztr3bwuClPB6v Cg/0KxOJREGLltSp
18C8fsNFi5eJRCJFRwQAAD+K0mVdT0+6Pj3pWpkRYmLjevTu6+7WKexV6MeU hI0b1oacPlNV4f1o
udkZzEPRsfyf3OyML1m82OiIHX9u5efnt3HtlPjhg6KDqg7+3nfg4cNHntal XwAAIABJREFUL0If
PX/68P79BwcOHlZ0RAAA8KMoV9b19KSru/dqd+/VsiReQqFw0eJlFg3sjEzN R4+d8OXLF7p+xco1
M2f8PnHCOAMDAxaL5eLsdOTQfnoXn18wZep0trkl29zSf9oMPr+ArtfSMdyz d599E0c9Q3bb9m5h
Yf+RbzM94tNO8QkJgwYPNzWrb2Bcp19/by6Xx3SXMo6UULV0DLf9ud3Wvllt XaNyPcdSIykZcLGp
L/F68eNKOkpJWpqazZs7rFu7avSoEUuXrZL6/wMyOXL0+OJFQWZmbDMz9uJF QYePHFN0RAAA8KMo
Udb16Gg7976+hJ9I+InufXwfHWsnvf269ZtevXr96P7thNj36urqQYuW0PV3 7t4d2N+r1C5Llq1I
S/v43+vnYa+eJSUlL12+ktl1/8HDWzeupCTF9fXo7T9tBvl23VB82mmg9zB/ v8kJcVEJse9sbazn
zAsqeYiS40gJlRDy/PmLR/dvf8kq/YKdpI6lRlIyYCnEjyslPEnGjB55+86d MptBmSIi37Vs6UiX
W7RoHhH5TrHxAADAj6Ny/siOxrXiCSF1esxWYBz3D7TuPiCA8FO+V6mbXz29 oePoJ5K62Ddx/Odc
iI2NNSGEw+G2btchPuYdIUTXwDSDk6Kqqlqyi7Vdk6uXL1g3bEAIiY6J7eXh Gf3uLSFES8cwOTFG
X1+fEJKXn29mbpmVmU7XS8pg8vLz7Zs4Jsa9F28maRxJoWrpGMa8DzczY9Nj Fpuays3OkNSxzEiY
AUvdLHZcKeEx7Yudh6KiImN2vc8ZH0v/vwGZaesZZ2Wm16hRgxBCUZSeITv7 E0fRQQEAQOkKCwrp
wta1ywKDV1NCIUUJKUpICalvBaGIougCRQkpijq4d+cwF21CSESRVSmpiUIU FlKi3C/ky6fvVUJd
QaFQSpfUtDRHp9bMpoqKCl0wMNDn8TLYbNOSXTgcrpWlBV22srTgcLjMLjpV IoRoamgIBIJSj/jk
aejCRUvevAnLzcsTP6K4UseRFCohhEl9aMWSG0kdZYlEOvHjSglPko/p6YaG BuU9KJSkpaWVm5ur
ra1NCPny5Uvt2rUVHREAAPwoynKFscvEZ+dC/iC6jZnHuZBtnSc+k9KFzTaN fveWWXvOXKRz69Tp
zLnzpXYxMTFOSPy6Bjw+IdHYuPTVVJKMHD1uks+EmKjwL1m8jykJsn/cTFKo Fe4oYySqqqp5+fl0
OTMzswrD23/gkLtbJxmfBUjR2L7Ry5ev6fKrV28a2zdSbDwAAPDjKEvWRQjp PuVFyKGvC7RDDq/u
PuWl9PYTxo31nzYjLj5eIBCEh0eMHjuBrl8wf86GjZv/3nfg06dPBQUFz1+8 HD5yDL1rYH+v2XPm
c7k8LpcXOHveoAH9pR9CV1c3OjqG2czP56urq7NYrITERGbNliwkhVrhjpIi KRZws6ZNtmz5Iy8/
PzU1bdr0WZUPLy8//82bsNlz5h84eHhh0DwZnwVIMWzoYHq5YVraxyXLVowY PlTREQEAwI+iRFkX
IaT3jNcn9q88sX9l7+mvymw8a+bv7dq26e3hZWRad8x4nz4evel664YNLl84 d+PmrSYOTmxzy5mz
Zg/8ll0FLwoyMTFu2typaXMnMzP2ooULpB9i+jT/9h07M8uttv+5de68IGN2 vZ69PNu0aS29ryyh
VrijpEiKBfznts0XLl02M7fs0rVnp44dKhMefb8uqwZ2k/ymqrFYjx/ctahf X9bnD5KNHzembds2
Tq3aObVq5+rabvSoEYqOCAAAfhRlWU0PAAAAoOQquZpeuea6AAAAAKorZF0A AAAA8oCsCwAAAEAe
kHUBAAAAyAOyLgAAAAB5QNYFAAAAIA/IugAAAADk4fv3MNZiaSowDgAAAAAl x9yvq2Iw1wUAAAAg
D8i6AAAAAOQBWRcAAACAPCDrAgAAAJAHZF0AAAAA8oCsC+BnosbSKFYAAICf BbIuAAAAAHmonlnX
+/fvvQcPYZvVqa2t27ad65kzZys2TiWnEzAbAQAAAAyly7oeHGrx4FCLyowQ ExPTtWv3Lp07h799
m8HjbNm8+cTJk1UVHgAAAEDFKFfW9eBQC3fv1e7eq2VJvIRCYVDQQvO69XT1 DIaPGJmTk0PXL122
PCAwYNIkH0NDAxaL1aqVy4njx+hdfD5/sq+fkbGpkbGpr98UPp9P16uxNHbt 2m1tY6dVW8elVes3
b8LIt5kqNZYGM2Ul6Ygydr967ZpjCyet2jrWNnZ//72vis4Z/NxKffGQEhOl mDcFAKgGlCjrurvP
0b2vL+EnEn6iex/fu/sdpbdfvWbty5evQp8+SU5K1NDQmL8giK6//e/tQYMG ltolOHhxWmpaRMTb
8PD/kj4kLV685PvR7927d/c2Jz3N09PT18+PEFJYkE//SxekHFHG7uPGjQ9e tDAzg3v735tPQ0Mr
dJKgGir54gEAgGpJ5fyRHY1rxRNCLDwXKzCOf3c16z4ggPBTvlepm189vaGL T5ikLtY2dpcvXbC1
tSWEpHM4zk4uSUmJhBBNLe3srE+qqqolu1hYNrh187q1tTUhJDo6umu3Hgnx sYQQNZbGx7RUAwN9
QkheXp6RsWlebg5dz+RMUo4oY/cGDW0CAmZ69u1rbm5emXMF1YmMLx5ms2QB AADkJjf7M13YunZZ
YPBqSiikKCFFCSkh9a0gFFEUXaAoIUVRB/fuHOaiTQiJKLJSlrmuwkJKlPtF lPXp+yP3i6BQKKVL
ampq02bN6Ut49epZcLhcut7QwIDL45XahcPhWFlZ0WUrKysOh8Pson/tEUI0 NTUFAkG5jihj95Mn
jt+4ccvZpZVdo8aXLl2S8tTglyLLiwcAAKoBZcm6eviHnwv5g+g2Zh7nQrZ1 9w+X0oXNZifEx9KX
8AoL8gv4eXS9e2f306dPl9rFxMQkISGBLsfHxxsbG5crSElHlJGzs9PZMyGp KcmbN2/09Z1Srr7w
q1FVVc3L+/oCy8jIVGwwAABQJZQl6yKEeMyMDDm0ii6HHF7tMfOd9PY+PhN9 fafExcUJBIK3b98O
HzGSrl+0MGjd2vV79uzNzPxUUFDw7NnzwUOG0ru8Bw2cNSuQw+VyuNyZMwMG ew+Sfgg9Pd2oqKgy
jyhj9xEjR0VGRhYVFYlEIoEQUxogjYNDs42bNufl5aWmpk7x91d0OAAAUAWU KOsihPSbG3Vi/8oT
+1f2m/O+zMazAwPaubbr1r2nrp7ByFFj+nl60vXW1tbXrl25dv16I3t7QyOT 36dPH+ztTe9aunSJ
iamJvX0Te/smZnXMlixZLP0QM2bMaNPWlfn4mKQjyti9b58+g7yH6Okbzp8f tH8/PsMI0uzYvv2f
f/4xMjbt2Mnd3c1N0eEAAEAVUJbV9AAAAABKrpqspgcAAACo3pB1AQAAAMgD si4AAAAAeUDWBQAA
ACAPyLoAAAAA5AFZFwAAAMD/2LvzuJj2/w/gH98fLaJtWiahokKFUomyZQtJ 1izJdpVvimupbFFZ
k13cKxdFuBe5l+va910hW9KmRZtmprK0TMvM/P4419F3qjEtZqa8no/z8PjM ez6fz3nPmU96d86p
kQRUXQAAAACS8PUjoivK6vb5NgAAAAAgPpzrAgAAAJAEVF0AAAAAkoCqCwAA AEASUHUBAAAASAKq
LgAAAABJQNUlOXfv3e9i0kNJmSHtRKAJo9cPFhIAQJODqktyVvkHbN8aUvwp X9qJAAAAgBQ0z6or
KSl52vSZ7fUM1TR0+g8c8teZv7/TjpSUGdSmrdPRccy4hIREEZ1fx8cPHzbk O2UCAAAAMk7mqq7o
k3bRJ+0aMkPK29QRjmPsBw18+SzmfXb69m0hUaf/bKz0qiv+lF/8KT85MW5A v36z584T0ZPLLWvV
qtX3ywQAAABkmWxVXdEn7exdgu1dgsUpvHg83prAdXqdumho686cPbeoqIiK b9i4ecnin93nzlFX
V5eXl7e2sjwWGUE9paTMCN37q3G37m1UNAghXG6Z14JFTF19pq6+98LFXG4Z 1e3K1WtWNnaqDGY3
U/OIw5EigjRlZeVFP3tT57pqTIy6C4c6MSYieaEMRXQ7cDC8m6m5KoPZt9+g ly9fUfGKiorlK/w7
Ghi369Bpd+he0QdK9CsCyajtrRS6bQt3cQEANAMyVHU9OG5rP8aTcDMIN8Pe yfPB77ai+2/ZuuPZ
s+cP7t5Mf5uooKDgvyaIit+6fXvi+HG1jXry5OmDuzeLPnIIIUHrNuTmvn/1 /MnLZ48zM7PWrt9I
9ZnrMd9/5fK8nIwrl/+JefxERJD26dOnHTtDe3Q3qy0x6nYu6sSYiOSFMhTR 7e69+9evXszOTB0z
2tF74WIquGHT5vg3CQ/v3XoT9yw7O0f0gRL9ikBianwrAQCg+Wlx9tg+k1Zp hJB2I/ykmMfdwzYO
E3wIN/trSEH30ultA2Y+qm1IN1Pzv89EGRkZEkJYLLaNbf+0lARCiIq6dj4r u2XLltWHKCkzUhJf
6+gwqYeGXUwvXThn2LkTISQ55e2o0c7JCXGEkC4mPRYvWujkOEpXtx09tsZg 1TMQTKb2jWuX9Dp2
rC0xJWUGfSu9iD5VMxTRLSsjRU1NjRBSUlqqo6v/sSCPEGLcrfv5c2eMDDuL c6BqfEUgYbW9lVVX
S9WH1RsAACAx5WXlVGN3yDrfgGA+j8fn8/h8Hp/H/9LgCfh8qsHn8/h8/pGD YdOs2xJC4isMZOVc
V3k5X1BcJPhY+HUrLqos54kYkpOba25pQ12zMzDsymZzqLi6uhqHU+t3I7qg IYSwWGwDfT2qbaCv
x2Kxqfbxo4evX7/Zx26gWQ/LCxcviQgSQoo/5Rd95Lx68cTI0PDFi1ciEhMn eaEMRXSjvk8TQlor
KlZWVlLt9+/z6Ff0zUlqe0UgYTW+lQAA0PzIStU1xP3xmag9RMWE3s5EhQ52 fyxiCJOpnZwQR12z
o0ofKj5o4MA/z5wVZ6daWprpGe+odlp6hqamBtW27GVx6sSxd2lJ27YG01d8 agxSWrRo0cnA4OBv
+xYt8SkqKqotMXGSr1+3qv3T0jPEnETEKwKpa9myZUlpKdUuKCiQbjIAANAo ZKXqIoQ4eD2NitxE
taOOBjt4xYruP3fObO+Fi1PT0iorK1+/jp85ey4VX7Vy2bbtOw+FHy4sLCwr K3vyNNbVbVaNM0wc
P85v2Uo2m8Nmc3z9VkyaMJ6Kz5rjnpCQWFFRIRAIKit5IoJV6eq2s+3b58TJ qNoSEyf5+nWjuU6b
stRnWXZ2zsePH5ctXyV6km++IpCi7mamu3btKSktzcnJXbhoqbTTAQCARiBD VRchxHHx8xMRG09E
bHRc9OybnZcu+dm2bx/H0eM0tNvP+snDabQjFTfs3OnCuTNXr1037WHJ1NVf stRv4pdySkjAGn8t
LU2znpZmPS11dJhrVv9bpjiOGjnFdYYms4P/mqBDB/aJCAqZPWvGofDDtSUm TvL160Zbudyvaxdj
G9sBJt17te/QXvQk4rwikJa9oTvPnb+go6s/ZNjIgQP6SzsdAABoBLJyNz0A AACAjGsmd9MDAAAA
NG+ougAAAAAkAVUXAAAAgCSg6gIAAACQBFRdAAAAAJKAqgsAAABAElB1AQAA AEjC14+IbiXfWop5
AAAAAMg4+u911Q/OdQEAAABIAqouAAAAAElA1QUAAAAgCai6AAAAACQBVRcA AACAJPwoVZecvKK0
UwAAAIAf2o9SdQEAAABIV8tvd2lqhE5rlZeVSisTAHHQKxZrFQCgeZO5qute pAUhpJ/bs4ZMgu9e
0IRQyxUXwQEAmj3ZusJ4L9LC3iXY3iWYqr1E4/F4/v6rddt3UFFVd53u9vnz Z3F2kZaWNm78RHWG
ZltlVScnZxabTcVLS0vn/DRXVY3RoYPe1m3b6W+Bly5fNrewVGqjbGjU5dCh 8Hq/NAAAAPjByVDV
dTvc3H6MJ+FmEG6GvZPn7Qhz0f2DN4fExj6LiX6UlZmhqKi4cpW/OHsZO27C woXe2VnvsjIzjI2N
fX39qHhgYNDHj5+SkxJjY5/ev/+A7j9nzk8Ba1YX5LNv3rgWHRNT71cHAAAA PzhZucJ4Y393hwk+
pCibjgx1Wnxpf48hHi9rGxIeHnHh/DldXV1CyIYN660srUN376KeqnqxRuhq 44vnsVRDUVFx3bog
I+Mu1MMTJ0/dvHFNU1ODELJ1y+bz589TcQUFxdz3uRwOp0OHDmH7fm34KwUA AIAfk6yc6yov5wuK
iwQfC79uxUWV5TwRQ3Jycsy695STV5STV+zQQY++VkgIKS8rpTehUQ8fPrIf PERNXUNOXlFVjcHh
5FPxvLy8jh07Um09PT26/8kTf1y9et3KuneXriZ0KQYAAABQV7JSdY3wfn0m ag9RMaG3M1GhDt6v
RQxhMpnpaW/p6qqMWyLOjqZOc53v6Zme9raMW8Jh5wkEAiqupaX17t07qp2R kUH3t7Ky/OvPqJzs
rJ07t3t6etX39QEAAMCPTlaqLkLI6CVvoiI3Ue2oo8GjlySI7u/h4e7p6ZWa mlpZWRkXF+c63U2c
vXC5pQoK8goKCunp6Z7zv1ZRk10m+fot43DyOZx8H99ldHy624w3b95UVFQI BIJKXmXdXxYAAAAA
ITJVdRFCxi5POhGx8UTExrHLEr/Z2c/Xx9bOdrjDSBVVdbcZs8Y6O4uzi/1h YT6+y1TVGEOHOdja
9qXjgYEBbdq06WxoZG5uYWPTu1WrVlR8jJPTJJcpqmqMlSv9IyLwO4zQ+Kir 5FUbAADQLLU4e2yf
Sas0Qoiec6C0k5EViYmJTmPGJiW+kXYiAAAAIEOKP32gGrtD1vkGBPN5PD6f x+fz+Dz+lwZPwOdT
DT6fx+fzjxwMm2bdlhASX2EgW+e6pMvHx7ew8MP7vDw/v+XOY5yknQ4AAAA0 K6i6vtLT1zM1MzMx
MVNRVQkIWCPtdAAAAKBZkZW/1yULFnh7L/D2lnYWAAAA0DzhXBcAAACAJKDq AgAAAJAEVF0AAAAA
koCqCwAAAEASvt5Nn5EUJ8U8AAAAAGScBrN9Q4Z/rbroTyQEAAAAgEZX5S9H oOoCAAAA+G6qnOsi
qLoAAAAAvhdcYQQAAACQBFRdAAAAAJKA+7pq0K2X3ZvY+9LOAqAOvrloZXBV y2BK1YmTZNjBI69e
x+/ZHtzAeSTDa/Ey8x5m7rPdpJ0IwI8I57pqhqMBTc43F60MrmoZTKk60UmW lnKPHD9xPHwf1c3E
sl/803v1mEdifH72mv6T5/SpkxTk5aWdC8APpxlWXaZW/alG2zZKtn16+y9b oq6mWtdJms3RABln
atW/cyf9syeOtGjRgooIBIIxLm6paRmvn9yt01QSrrpk/AvN1Kp/XQ9gbUQn eeXGrV4WPTu016W6
vX5yt7b+Yr5Y+sBS2igpRd++RAi5c//hwcPHXsbFKyoo2vax9l3kpa2lSQhJ f5e5YfOO2BevCCG9
enZftWyxfscO1O52hO47+dffLVq0cBnvvMjLg1pjeh3b9+xudv3W3VHDh4iT DwA0okb4HcbHT5/N
nvez6D7hYbusLS3EmY313IMQomW+v37JUOKe3CGEFBQUhuzcG7hxy64t6+s6 A36jEyRGQUHh1t37
gwbYUQ+v376rqKhI6r4Iv9m/0Ve1jH+hNdbkoue5def+0EH9xdmXmPlQR5Vy 5PhJFptDDQyP/N1t
6qTeVr3+83//OfZH1OLlq48d+pUQsnR5wOBB/bYFBxFCIo7+4bMi4NSxg4SQ k3/9/eTZi7MnjxBC
FvutPn32nwljR1PTjhg++ObteyOHDxYnHwBoRFU+EUggqN9m3cs8fN8udXX1 mzdvCv7XzZs31dXV
w/ftsu5lLs5UrOce9i7B9i7BrOce3+zM5/F27gnrP2yMpd0wnxWBxcXF/z71 5bWoq6muWLrgwaMY
IhBkZWV7L17ee4BDL9sh/13gU5BfQPWprKgI2b6n/1CnvvajIiL/EJohPj5h 8MjxR38/JWKGMi53
ZcAGq37DBw53PnT4uJnVgG+khw1b1Y2QOW5TDx4+TkcOHT7204yp9CIsKysL WB/SZ+DIPgNHBq4P
KSsro+LlZeX+gZuohRd+5Djd/5tfF3VdsWZWA05GnRnuNMm8z+CJ0+YkJCYL TSijX2hfJjezGnDi
1BmHMZMt+g4Z6zIz9tnLM+cuOI6bRr2c5OS3dLdDh48PGO5s1W/46qBN5WXl QvPUtsXFJ/Tsbko/
pBOr7Q0Sf6soLz/2x+kZUyf9uzB+3TlkUP+2bZSUFBVnTnNJSPj3jUhLz5g7 07VtG6W2bZQ85ril
pr+j4mfPXVzoOVdLg6GlwVjoOffMuYv0zD3Nur16HS/9xY8NW1PcGqvqakgO Vr3Mt24ImDBhwq1b
t+gJb926NWHChK0bAqx6mYszSe5TD/sxnoSbQbgZ9k6eubEeovvvPxT5+k3i qcgDty+fkZeX3757
H31AqhycFtRDr8Urpk+ddOfy2duXzurrddy8Yy/VYe/+8JS3aaeOHrx85kQe i111hlt3H8z72c/f
b7Hr5IkiZtj968HPRcVXzp746/eI2Ocv6b3Xlh42bFU3QshQ+wEFBYXPXsQJ BCTmyfMPHz4NGTSA
Xki7fznAYnPO/3nsn9PHct+zQn89SMVD9x0sKPxw+ewfp4+HRz+O/ebC+zph HVcsISTm6fPIA788
uH5+yKABgRu2CE0okMkvNLoDIeRhzJOIsN0Prv8zymGo589+d+49OvDLjoc3 zg8fYh9Q5eU8iX3x
1+/hl8/+wSko3LPvYPWXWePGyS/QYKhXfUNFv0Fm1gNr3KrPfPb8ZWtLc00N DaF4SQk38vcoG2tL
6uHA/rYHDx///Ln406eiAxHHBvbrS8XfpqabdOtCtU26dklJTaNn0NbSYrHz pb74sWFrilsjVl0N
YtWr55YNa+jCiyq5tmxYY9WrpzjDs6N/GjpmMSni/LsVc4Y6Lc6KnitiyJ9n z69Y+rOWpkZrRcVF
Xh7Xbt6h4vRrKSgo3Lx9Tx9rS4FA8NcfEb0tLeTk5JSUWi/0nPsg+jHV5+/z l1f4LNTS1GjTRsl3
kRc9w++n/gzauG3v9uBBA+yoYG0zXLxy3ffn+aqqKqqqKj6LvOi915YeQFWE kBYtWsycPvng4eMC
geDA4aOzpk+h7r+hOly4fH3ZYm81VVV1NdXlSxdcuHyNip+/dNVvyb/xZUsW fHPh0R3qumIJIav8
FmlqMBQU5Ge6urxJTBaaUDa/0OgOhJA1K3y0tTTl5eWnT5lQXFLiv2wJ/fD1 m0S6G3U81VRVly32
Pv/lOH/z/0YiEPzf//1f1TdU9Bv0MvpWjZvQtDweLzzyj1nTpwjFu/ce2Hug w5Hjp/yWeFMRv8Xe
f/19vu/gUbZDHM+eu7jsS7yktFRRQYFqKyoqFBeX/E/aDf4/H+DH1MCqqzHv prey6Lll/eoJE8Yv
WLAwNHT3lvVrrCx6ijlteTlfUFxEigq/hngqleU8EcPz2JwxLl9/+blFixZ0 5x42gwghSkqt+/a2
Wr18qUAgeP7q9c49YQlJyaWl3Kqd2Zx83XY61fdy5PjJMY4Opt260E/VNgMn v0CHqU212zG1yZcj
KSI9gKoEAoHTKIdffou4cPlaUkrqri0bBF++KRJC8gsK6CWq204nv6CQXni6 Okw6TsRYePVescpt
21JteXl5Ho/XVL7Q6IaK8tf8hR5WfTn08Wynw+TkF1Sfp0YaGoz3eSzqLRA6 zjW+QWK6cv2WfscO
BnodhUa9eHSzuLjk2ImoNes2h+/bRQjxD9o0epTDjKmTqOO5KnBTWOhWQoii okJJSamSUmtCSHFx
iZJSa3qq3PcsTQ11/I8EIHmN/Pe6rCx6blm3+ievJQf3brey6Cn+nHq2B85E zRk72Y+OnDmxufOg
cBEzaGowIn/bo6Wp8TX0pfOLhzf+p6tA4Lsy0Gehp12f3kpKrUtKSm2HjqY6 a2owsrJy9DoKf4T4
oV92zvVaoty2zSzXyVSkthkY6mo5Oe/b6+oQQrJzcuk0RKQH8D8EArmWLV0n jV8VtMl73hy5li3/
XSpfFlhWdk7H9rqEkKysHHU1VSquwVCn49nZOXR/UQuv3itWaOk2lS+0b+Uv 9PDr8czJZair1Tr8
f5l2NX4ZF6+rwxSasLY3qGffmu9hFzqYBw8fX+mzsMZdK7VWnDnN5bfwo9Sz sc9fhm7bKNeqFSFk
7izXfkOdqLihgf7r+ITeVhaEkPg3iZ0N9OnZXsa9NjPphv+RACSvyhXGRrro admr58Mb5y179azr
wE6DDkZFbqKSiToa3GnQIdH9J45zCgre9i47u5JXmfQ21W/1Wipe42spKyuT k2vVSr5VVk5uUPBW
uo/TqOGbtu9+z2J9KvocsnMvPYOWlsbBX3f89feFA0eOiZ5hxDD7rbt/Kfzw ofDDh627fqHjtaWH
DVvVjV4ws2dMeXrvymy3KUJxh6H2ITv35hcW5hcWbt4ROmKYPRUfOXzw1l2/ FHwozC8s3Lxz7zcX
Xr1XrNAXFP1Qxr/QastTxMuhjmfBh8ItO/eOHD5ExMusug3o1/fytZvVJ6zt DXr+8HqNW9U57z54
pKAg37OHadXgqrWbUtMzKnmV7Pz80LCDPbqbUPFOBnoHDx/7VPT5U9HnA4eP GXY2oOKjRw0PDTvI
4nBYHE5o2MExjg70VFeu3xrYr4+01jw2bE16a7yqq/EoKMjXb6DRsPATERtP RGw0Gnrom51nT59i
0cPMw9vHZtCoFWvW2w/oR8VrfC0BK3227vq1z6BRc72WmHc3o/vMm+NmoNfR xc3dcfx0prZW1Rk0
GOoH9m775+LV/YciRcww3312a0VFh7FTx0+b0920W8uWLUWnB1BVbV96dNx7 3myGmqrTRDeniW6a
Ggwvj9lUfL77rLZt24wYO22i609WFj3p/t/8uqjrihXKkH4o41/UB2APAAAg AElEQVRoteUp4uWY
9zCbMO2nkeOmqagoz3efKeJlVjVkUP/nL+PeZWYJ9a/tDRLHwSPHZ02fLBQc YNfHb/W63gNHuri5
f/j4afO61VR8Y+DKp89eDh3tMnS0y/MXcRsCV1DxCc6OFj3Nxk+dM37qHEvz HmNHj6DiGe+yXrx6
bf/lRjoAqJMGVl0tzh7bZ9IqjRDyWXdoA+eC9IzMBUtXnouKlHYiAGJpoiv2 e6Tdy25Y7P2r9Rt7
KPL3uNcJ24ODGjGf72fJ8oDupt1mu02RdiIATZKBsRnV2B2yzjcgmM/j8fk8 Pp/H5/G/NHgCPp9q
8Pk8Pp9/5GDYNOu2hJD4CoNm+LfpJW/b7n3us6dXVJRvC903aIAtjiTIuCa6 Yr932vWecPb0KQ0Z
LmHbNgWSppMtQDODT79uBO2YWuOmzC4vLx/Qr89/f5qBIwkyromu2O+edhM5 DgDQdOFcVyOYMnHs
lIlj6Yc4kiDjmuiK/a5pP7lzqakcBwBoulB1AQAAAEhCI3z6NQAAAAB8E+7r AgAAAJAEXGEEAAAA
kISvVZeF7RAp5gEAAAAg44o/fWjI8P98uwsAAAAANBiqLgAAAABJQNUFAAAA IAmougAAAAAkAVUX
AAAAgCSg6qozOXlFaacAPy56+WEdAgA0Oai6AAAAACQBVdf/2LZ9h2LrNtu2 7/gek+PkBAAAwI+s
mVRd9yIt7kVaNHASPp8fFrZ/69aQ/ft/4/P5jZIYAAAAAKU5VF33Ii3sXYLt XYLFKbx4PJ6//2rd
9h1UVNVdp7t9/vyZfurS5ctqaqpe8+czGOqXr1yh42VlZXPdPVTVGB066FU9 DZaWljZu/ER1hmZb
ZVUnJ2cWm03F5eQVt27b3r59R1U1hrvHvLKyMvLlRJecvCJ9xqu2TOTkFXft DjXoZCiv0JrKytzC
UqmNsqFRl0OHwht4rEDWyMkr7t//m6FRF6U2yta9bV68eEnHhbpJIzsAAGhM Tb7quh1ubj/Gk3Az
CDfD3snzdoS56P7Bm0NiY5/FRD/KysxQVFRcucqffips3/75np6EkP/Om7fv 1zA6HhgYxGFzkpIS
njx9fOvmLTo+dtyEhQu9s7PeZWVmGBsb+/r60U/du3sv9tnTpKQEVh4rKGgt IaS8rJT6l2qIziQm
JiYm+lEZt4QQMmfOTwFrVhfks2/euBYdE1P/IwWy6vadO3du32Tl5To7O3vO ny/tdAAA4HtpcfbY
PpNWaYQQPedAaSdTZzf2d3eY4EO42V9DCrqXTm8b4vGytiGGRl0unD9nbGxM CMljsawsrTMzMwgh
aWlp/foPeJuSrKCgwOVyOxsa3b93V19fnxCib9D5+rUrnTt3JoQkJyebmvWg KydaSUmJkXGX7KxM
QoicvOKb+Diqf0pKyrDhI9JSU6h41YG1ZSInr5iRnqqjo0N169TZyMdnifOY Mbq6ug0/YiBr5OQV
3+fmqKurEUJKSko0NLVLij+TaquFfli9AQAAEkN/DuPukHW+AcF8Ho/P5/H5 PD6P/6XBE/D5VIPP
5/H5/CMHw6ZZtyWExFcYNO1zXeXlfEFxkeBj4detuKiynCdiSE5Ojln3ntSV vg4d9OjLgmFh+9ls
jrKKmpy8orKKGpvNCQvbTz2Vl5dHlV+EEAMDA3qqhw8f2Q8eoqauISevqKrG 4HDy6afo/vr6+nl5
eXXKhBBCl1yEkJMn/rh69bqVde8uXU3Onz8v7qGBpoMquQghrVu3rqyslG4y AADw/TTtqmuE9+sz
UXuIigm9nYkKdfB+LWIIk8lMT3tLXekrLyulruJxudzIo0eTkxLoeFLimyOR kVwulxCira2dnp5O
DacbhJCp01zne3qmp70t45Zw2HkCgYB+iu6WkZGhpaUlfibVWVlZ/vVnVE52 1s6d2z09vcQ9NNDE
tWzZsqTk3yWRn18g3WQAAKBRNO2qixAyesmbqMhNVDvqaPDoJQmi+3t4uHt6 eqWmplZWVsbFxblO
dyOEnDx5qrd1bz09Pbqbvr6+ZS/LU6eiCCFTJrv4+C5jszksNnvpUl+6D5db qqAgr6CgkJ6e7jn/
f+ohqj+bzfHx8Zs6ZTIVVFVVSUpKEp1JddPdZrx586aiokIgEFTycCLkR9Gj R/ftO3aWlJTk5OR4
eXtLOx0AAGgETb7qIoSMXZ50ImLjiYiNY5clfrOzn6+PrZ3tcIeRKqrqbjNm jXV2JoT8um/fvP96
CPWcN8/91337CCGBgQFqaqqGRsa9LCwHDBxAd9gfFubju0xVjTF0mIOtbd+q Y+3sbC0sehkZd1Fn
qAcErKGCixcv7tPXjv5ltBozqW6Mk9MklymqaoyVK/0jIvA7jD+Kfb/++vff f2toag8YaG8/aJC0
0wEAgEbQtO+ml024zRkAAKBZ+qHvpgcAAABoKlB1AQAAAEgCqq7Gh8uLAAAA UB2qLgAAAABJQNUF
AAAAIAmougAAAAAkAVUXAAAAgCS0pFsVZTV/Ig0AAAAANBzOdQEAAABIAqou AAAAAElA1QUAAAAg
Cai6AAAAACQBVRcAAACAJKDqAmhKlJQZQg0AAGgqUHXJOnxzBQAAaB5Qdf2P nbtCldW0du4K/R6T
i1k/CXUr/pT/PZIBAAAACWsmVVf0Sbvok3YNnITP5/92IHxz8IYDByP4fH6j JAYAAABAaQ5VV/RJ
O3uXYHuXYHEKLx6PtyZwnV6nLhraujNnzy0qKqKfunL1mqqqiuc8d3V1tavX rtNxJWXGgYPh3UzN
VRnMvv0GvXz5SnScyy3zWrCIqavP1NX3XriYyy0jX85gKSkz6FNZaenpkya7 aut0VNdsN3a8C5vN
qbEb3ahxWhFpXLl6zcrGTpXB7GZqHnE4sl6HFr47EatLqJs0sgMAgMbU5Kuu B8dt7cd4Em4G4WbY
O3k++N1WdP8tW3c8e/b8wd2b6W8TFRQU/NcE0U/t/+3Qf+e5E0I83H8K23+w 6qi79+5fv3oxOzN1
zGhH74WLRceD1m3IzX3/6vmTl88eZ2ZmrV2/kXy5UFj8KZ++YjjRZZr3/P+m pyalv00wNjJctsK/
xm60GqcVkcZcj/n+K5fn5WRcufxPzOMnYh9RkLTaVhcAADQzLc4e22fSKo0Q 0m6En7STqbO7h20c
JvgQbvbXkILupdPbBsx8VNuQbqbmf5+JMjIyJISwWGwb2/5pKQmEkLT09EGD HRLjXyooyHO5ZV1M
ety+eUVfT48QoqTMyMpIUVNTI4SUlJbq6Op/LMgTETfsYnrpwjnDzp0IIckp b0eNdk5OiKP613aT
VklpaTdT84zUxOrd6Icipq0xjS4mPRYvWujkOEpXt10DjjF8XyJWV43LoHoD AAAkprysnGrsDlnn
GxDM5/H4fB6fz+Pz+F8aPAGfTzX4fB6fzz9yMGyadVtCSHyFQdM+11VezhcU Fwk+Fn7diosqy3ki
huTk5ppb2lCX8AwMu1LX9Qghvx0I53A4DK12SsoMhlY7Dofz24FwehT1TZEQ 0lpRsbKyUnScxWIb
6OtRbQN9PRaLXWMmj6Jjhjk4ajE7KCkzNLXb5+d/4zuoiGlrTOP40cPXr9/s YzfQrIflhYuXRE8O
UlTb6gIAgGamaVddQ9wfn4naQ1RM6O1MVOhg98cihjCZ2skJcdQlvOJP+UUf OYQQLrfs2PE/4uOe
0fHXr2KPHvudvneqTrS0NNMz3lHttPQMTU2NGru5zZwzz2NuStLroo+c99np AoGgUaalWfayOHXi
2Lu0pG1bg3Hdqslp2bJlSWkp1S4oKJBuMgAA0CiadtVFCHHwehoVuYlqRx0N dvCKFd1/7pzZ3gsX
p6alVVZWvn4dP3P2XEJI1Ok/ra0s9Tp2pLvp6+n1sjA//edf9Uhp4vhxfstW stkcNpvj67di0oTx
VFxFRSU5OYXuVlrKVVBQkJeXT8/IqFoVCXX75rS1mTXHPSEhsaKiQiAQVFaK Ov8HMqi7memuXXtK
SktzcnIXLloq7XQAAKARNPmqixDiuPj5iYiNJyI2Oi569s3OS5f8bNu3j+Po cRra7Wf95OE02pEQ
Erb/gIf7HKGe7nNnh+0/UI98Atb4a2lpmvW0NOtpqaPDXLN6FRVftNC734DB 9C+j/bp39/IV/prM
DiNHOffpY0MPF+r2zWlr4zhq5BTXGZrMDv5rgg4d2FePFwJStDd057nzF3R0 9YcMGzlwQH9ppwMA
ALVKSklXVmMQQtLe5ahrtSOEZL7P12pvUL1n076bHgAAAEBiarybvnUbFUIE H/JZKuqaAoEg/30W
g6krEJD371KE7qZvKdXkAQAAAJq2ok8FfB6fEFLIec/n8Qgh7Jx3VENIc7jC CAAAACAt4l9hRNUF
AAAAUH+9evWq1rCosSeuMAIAAADUH64wAgAAAEgCrjACAAAASAKuMAIAAABI gvhXGL9WXa3kW0ss
PwAAAIAmh/57XVUlpaT36mXxIZ+V9i7HwsIi/31W5vt8CwuL9++EP2kGVxgB AAAA6g9XGAEAAAAk
Ab/DCAAAACAJ+B1GAAAAAEkQ/wqjTFRdt+/ckXYKAAAAAPVR9KngU2E+IaSQ 876AlUMIYee8Y2Wl
Ve8pE1XXsGEOslx4yckrNvqcN2/dat++4/eYGYDSJFbX90gyeHPIhIkukt9v /YyfMClky1ZpZwEA
DdL0rjBOneoqy4VXY6H/r1+9es3hIxHlZaVElr4BADR1xcXFoaGhIZs3UQ9l /4trc/DGXbt2lZaW
SjsRAKi/JnaFkRBy6tQpqRdeiYmJLpOnMHXatWmr0tfW7s8//2r0XVBlFiHk 1au4gQMGCAXhByQn
ryi0CcU1tZhTp7my2Rzp5ink+1Uz1WcWOjKinTlz1s7OrnPnztTDhn9xCb07 GpraVPzipUtDhg5r
q6zK1GnnOt0tJyeHiicnJ48aNVpVjaGqxhg1anRycjI91cuXLx0dndTUNfT0 O0VEHKaCRkZGNjZ9
zp37p4F5AoAUNbErjISQQYMGNaTwuhdpcS+y5rpSTCkpKcOGOQwZPPh1XFw+ h7Vr584TJ082ZELR
SktLW7bEn+0AQggpLyutugnFX79+JS8vP9/LS4oZSkBiYuLYseOpmkZTiznJ ZfKNmzepp4QOi2j/
nD8/1tm5EROr+tZs2RIy96c5VHzH9p0LFyzIfJeRnJRoZmY2Zeo0Kj7NdXrf vn3S01LT01J797Z2
ne5GxZOTk8eNn+juPjc76929u7fvP3hA78Jl0sRz/6DqAvghyErVRRpQeN2L tLB3CbZ3CRan8OLx
eP7+q3Xbd1BRVXed7vb582cqvnbdeh9fn3nzPBgMdXl5+d69rU/88bvQ2LS0 tHHjJ6ozNNsqqzo5
ObPYbCp+6fJlcwtLpTbKhkZdDh0KFxGkfl6n/xX6Ib623OTkFXftDjXoZCiv 0Lq2maG50tLU3L5t
27Vr10ntK7CiosLX16+dbnstbeaOnbuEZoiNfaZv0Dl0zx4RM5SWls75aa6q GqNDB72t27Z/c03S
5OQVw8L2G3fp1qatirmF5f37D44ciTQx7a7URtm6t01cXBzdbeu27e3bd1RV Y7h7zCsrKxOaZ+Kk
yZaWveLj4wghSYkJM2e4bd2yrR6H6+nT2D59bKqmRzXKysrmuntQL3Db9h31 mLm8vHzv3l8WLlxA
Pbxy5ZKz8xhVVZW2bdsu+nnh8+cvqHhiYpKfn6+qqoqqqsry5csSE5Oo+Np1 61f7rxo71rl169Yd
OnT4bX8YPbONTe/Hj5/UIyUAaHJkqOoihAwaNMjLy2vYMIeSkhIxh9wON7cf 40m4GYSbYe/keTvC
XHT/4M0hsbHPYqIfZWVmKCoqrlzlT8Vv3rg5adJE0WPHjpuwcKF3dta7rMwM Y2NjX18/Kj5nzk8B
a1YX5LNv3rgWHRMjIkihfnCvfm6jttwIITExMTHRj8q4JaJnhmapRYsWVKO2 Fbhu3fr4+DePY6KT
EhOzs7Kqjj1/4YLTmDGhu3cu8PYWMUNgYNDHj5+SkxJjY5/ev//1NIyINUm7 fuPGjetXWXm5U6ZM
HuM89uKlS1cuX2Sz3k+YMMFz/tdTdPfu3ot99jQpKYGVxwoKWis0SVZW1vTp rlqamoQQNTXV0aNH
X7hQn9M/eXl5TCazejwwMIjD5iQlJTx5+vjWzVt0vPoV3tquZh49emzgwAHt 2rUTihcXF4fu2TvY
3p56OGrUyC1bt338+PHDh49btmwdOXIEFb9189b7vDw9PQNVNcbUaa6FhR/o Gdq3b5+bm1uPFwsA
TU6Ls8f2mbRKI4ToOQdKKwk5eUWBQEAIuXXr1qRJk37//Rh9z5NoN/Z3d5jg Q7jZX0MKupdObxvi
8bK2IYZGXS6cP2dsbEwIyWOxrCytMzMzCCGtldp++lhY41U/OXnF6hc4SkpK jIy7ZGdlEkI6dTby
8VniPGaMrq4u3aHGID1V1Tnpdm25yckrZqSn6ujoiJgZmqjq3+CFVgibzVm+ YsWHDx9PR/3PJe+q
K9Cgk+GVyxeNjIyEZt61a0dwcMjpqFNWVpbVd111Bn2DzjdvXDMwMCCEvH37 tpuJ2TfXJJ1nbk42
g6FOTaiqxqj6UENTu6T4M9XtTXwcdbtVSkrKsOEj0lJTqs4TumfP+vUbbPva nr9wIXT3rmHDhnbq
1Eno5YhznVFZRS2fw2rVqpXQKH2DztevXaESSE5ONjXrUadbvvh8fvce5qej Tnbt2lUoK0KIlqbm
7ds3qclzcnIGDLR/9+4dIaRjx45379yivnIVW7eZMcNt08aNhBBfPz8ul3vs aCQ1SUVFhYam9scP
BeLnAwDSUvzp3x+Zdoes8w0I5vN4fD6Pz+fxefwvDZ6Az6cafD6Pz+cfORg2 zbotISS+wkCGznXV
teQihJSX8wXFRYKPhV+34qLK8hr+Bj8tJyfHrHtP6sfZDh306CssDHV1Nucb Nyw/fPjIfvAQNXUN
OXlFVTUGh5NPxU+e+OPq1etW1r27dDU5f/68iKBoteVGCKFLrvrNDLKstvu6 qJXQzcS0qKjol1/2
kNpX4Pv376mCScjOnbtnuE2vWnLVNkNeXl7Hjh2ptp6eHt1fxJqkUTUWIaR1 69ZCDysrK+lu+vr6
dCMvL09okgXe3vGvX8+aNZMQ8vLVK/vBQzeHbBF52GrGZDKzs7Orx/Py8ugE ajxWop0+/aexsbFQ
yUUIKS8r5bDzPOd7unvMoyI//eTuNt2Vlfeelfd+uuu0OXPmUnElJaXgTZvU 1dXU1dW2hIRcvHiJ
niQrK6vG83MA0PzIStVVj5KLEDLC+/WZqD1ExYTezkSFOni/FjGEyWSmp72l v8NR1+wIIfaD7U+f
Pi16d1Onuc739ExPe1vGLeGw86jzc4QQKyvLv/6MysnO2rlzu6enl4igaLXl JqQeM0NTRC0DDjvv
xB+/a2tpkdpXIJPJTEur4Zdlbly/GnX6z6q3MdU2g5aWFnV6hhCSkZFB9xdz TYojPT2dnl9LS6t6
BwZD3dl5DCFk757QB/fvBgdvrsdeLC17RUfXcNldW1ubToBuELGvMIZs2eLr s7TGPSorKy9ZvIi+
Meve/fvLly+j7+u6d/8+Fe/e3azqKPqqMSEkOjrG2tqqLq8SAJoqW
|
|
|
| Re: EPackage.Registry concurrency issues [message #428901 is a reply to message #428898] |
Wed, 01 April 2009 14:50 |
Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
Ed,
One more insight: The first thread that creates the PackageImpl instance
registers the not-yet-fully initialized package in the registry by way of
the superclass EPackage constructor:
protected EPackageImpl(String packageURI, EFactory factory)
{
super();
Registry.INSTANCE.put(packageURI, this);
setEFactoryInstance(factory);
...
}
Any subsequent thread that calls Registry.getEPackage(...) will find this
instance instead of the EPackageDescriptor. Thus the differences between my
first debugging foray and the second one. The other threads don't call
field.get(null) for the eINSTANCE field in this case.
Cheers,
Achim
Achim Demelt wrote:
> Ed,
>
> This is wacky stuff! The key to the problem is concurrent access to the
> package registry during the initialization of the generated EPackage. In
> the debugger, it's very easy to produce the PackageNotFoundExceptions you
> predicted. In some cases, I've actually seen those in my logs, too, so
> you're not far off the mark. Here's how to do it:
>
> 1) Have two or more threads execute EPackage.Registry.getEPackage(...) for
> a generated package registered via the extension point.
> 2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
> 3) Have another breakpoint in the generated PackageImpl's static init()
> method.
> 4) With the first thread, step into the call to field.get(null). This
> actually initializes the package class and thus calls the static init()
> method. Execution should halt at the breakpoint.
> 5) With the second thread, step _over_ the call to field.get(null). This
> time, the classloader apparently considers the class already initialized.
> However, the eINSTANCE field is still null. So, yes, getEPackage() _will_
> receive a null value for eINSTANCE.
>
> When reading from an XMI file, this obviously produces a
> PackageNotFoundException. But it gets even better!
>
> Now I _don't_ halt execution at EPackageDescriptor.getEPackage(). Instead,
> I
> only have a breakpoint in the package's static init() method directly
> _after_ the the line "isInited = true;". I also have a conditional
> breakpoint in XMLHandler.createObjectFromFactory(), with the condition
> "factory != null && newObject == null".
>
> Again, two or more threads reading from XMI. One thread halts at the
> breakpoint in init(). The other thread halts at my conditional breakpoint.
> Taking a look at the variables here, I see that I have indeed the factory
> I want (my ModelgenFactoryImpl). It refers to the correct package (my
> ModelgenPackageImpl). This package, however, has no name, no URI, and no
> contents. In fact, its "isCreated" and "isInitalized" fields are false.
>
> Both threads, the one the halted at init() and the one in
> createObjectFromFactory() refer to the exact same ModelgenPackageImpl
> instance. So it seems that the classloader does returns a non-null
> eINSTANCE field if the thread is not halted in the debugger. But the
> package simply hasn't been fully initialized yet. See attached screenshot.
>
> Now, what can we do?
>
> For the record, I'm using EMF 2.4. The generated code is up-to-date,
> NS_URIs match.
>
> Cheers,
> Achim
>
> Ed Merks wrote:
>
>> Achim,
>>
>> Comments below.
>>
>> Achim Demelt wrote:
>>> Ed,
>>>
>>> Thanks for your comments. Below you'll find and exception stack trace
>>> that occurred when loading a file ending with "*.modelgen". It's an XMI
>>> ClassNotFoundException, not a java.lang one.
>> That's what I would have assumed....
>>> In the particular log file I'm
>>> looking at right now, I see this stack trace three times logged within
>>> one second. However, there are 42 such "*.modelgen" files in the
>>> workspace, all of which are loaded sooner or later, so the other 39 are
>>> ok. The files themselves are ok. I can open them in the editor without
>>> any problems. Very strange...
>>>
>> Indeed.
>>> I don't think classloading is the issue here, since I would suspect
>>> other exceptions (java.lang ones) popping up somewhere. I'll try out
>>> your suggestion for debugging to find out more.
>>>
>> Well, as I said, it would be very odd to see a null coming back for the
>> eINSTANCE. That should be impossible. But if you look at the rest of
>> the logic for getEPackage, I don't see how multiple threads executing
>> that same code, could ever have one of them returning null...
>>>
>>>> Another perhaps more likely cause is that there are
>>>> packages that aren't properly registered and hence when first accessed
>>>> add an entry to the global registry invalidating my comment/assumption
>>>> about read-only thread safety.
>>>>
>>>
>>> How would a package not be properly registered?
>> If the uri in the plugin.xml doesn't actually match
>> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
>> it's possible for it to be out of date.
>>
>> Now that I think of it some more, if the package was null, you ought to
>> see a package not found exception. You've not indicated which version
>> of EMF you are using, but the problem sounds like this one which was
>> fixed a very long time ago.
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>>
>>> All I can do is declare
>>> these extensions, right?
>>>
>>> <extension point="org.eclipse.emf.ecore.generated_package">
>>> <package
>>> uri =
>>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>>> class =
>>>
>
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>>> genModel = "model/modelgen.genmodel" />
>>> </extension>
>>> <extension
>>> point="org.eclipse.emf.ecore.extension_parser">
>>> <parser
>>>
>
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>>> type="modelgen">
>>> </parser>
>>> </extension>
>>>
>>>
>>> Thanks,
>>> Achim
>>>
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>> 'GeneratorConfiguration' not found.
>>>
>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>> 2, 276)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>>> at
>>>
>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>>> at
>>>
>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>>> at
>>>
>
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>>> at
>>>
>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>>> at
>>>
>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>> at
>>>
>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>> at
>>>
>
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>>> at
>>> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>>> at
>>>
>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>> 'GeneratorConfiguration' not found.
>>>
>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>> 2, 276)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>>> at
>>>
>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>>> at
>>> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>>> at
>>>
>
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>>> at
>>>
>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>>> ... 11 more
>>>
>>>
>>> Ed Merks wrote:
>>>
>>>
>>>> Achim,
>>>>
>>>> Comments below.
>>>>
>>>> Achim Demelt wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've been experiencing concurrency issues lately when loading models
>>>>> from XMI resources in multiple threads (Eclipse Jobs) at the same
>>>>> time.
>>>>>
>>>> There is nothing in EMF to make things that affect common data
>>>> structures be thread safe.
>>>>
>>>>> The
>>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>>> classes.
>>>>>
>>>>>
>>>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>>>
>>>>> There is no other error message. It's just that someone down the call
>>>>> stack simply returns null.
>>>>>
>>>>>
>>>> Again, stacks are helpful to see.
>>>>
>>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>>> behaves differently under observation. So if I debug the code,
>>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>>> does the trick. So I've settled with that as a workaround.
>>>>>
>>>>> Out of curiosity, I'd still like to know the underlying cause of the
>>>>> problem and I investigated a bit into the EMF codebase. So here's the
>>>>> situation: My code is running in an Equinox runtime, all bundles are
>>>>> installed properly, all EPackages are registered via extensions.
>>>>> Technically, everything is fine.
>>>>>
>>>>>
>>>> That sounds good, but I've also never heard of a problem like this...
>>>>
>>>>> Now I start a couple of Jobs that each reads one model that resides in
>>>>> the workspace. Each thread uses its own resource set. All models are
>>>>> based on a small number (3 or 4) of metamodels. So several Jobs ask
>>>>> the EPackage.Registry for EPackages of the same URIs, basically at the
>>>>> same time. Being registered via extensions, the Registry contains
>>>>> EPackage.Descriptors. Upon first access, the Descriptors lazily
>>>>> resolve the actual EPackages and EFactories. Since the default
>>>>> EPackage.Registry implementation is not thread-safe, I'm guessing that
>>>>> something goes wrong here and some threads receive
>>>>> not-fully-initialized EPackages.
>>>>>
>>>>>
>>>> I'm not sure how that's possible though. It should be thread safe from
>>>> a read-only point of view...
>>>>
>>>>> _If_ that is the case, overriding the default EPackage.Registry with
>>>>> one that is synchronized should solve these problems, right? Before I
>>>>> do that, though, I'd like to hear your opinion if my logic is flawed
>>>>> or if there are other possible concurrency hotspots in this scenario.
>>>>>
>>>>>
>>>> I know EMF has been in use for a great many years and I've never heard
>>>> of a problem that was attributed to this cause. If there is an
>>>> underlying thread safety problem here, I'd want to fix it in EMF
>>>> itself. I know it's very hard to debug threading problems but here is
>>>> a
>>>> very good trick. Set breakpoints at the place you suspect has the
>>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>>> until both threads are at this breakpoint. Then step each thread
>>>> through the logic. If there is a problem, it's most likely to be here
>>>> in the resolution of the package descriptor.
>>>>
>>>> public EPackage getEPackage()
>>>> {
>>>> // First try to see if this class has an eInstance
>>>> //
>>>> try
>>>> {
>>>> Class<?> javaClass =
>>>>
>>>>
>>>
>
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>>>
>>>> Field field = javaClass.getField("eINSTANCE");
>>>> Object result = field.get(null);
>>>> return (EPackage)result;
>>>> }
>>>>
>>>> I don't think it should be possible to get back a class that hasn't
>>>> been
>>>> initialized. I think a class loader that returns a class that isn't
>>>> completely initialized is doing something bad. After all, how is one
>>>> supposed to use a class that might not be in a good state to be used
>>>> yet? Yet from your description, this sound likely to be at the root
>>>> of
>>>> the problem. Another perhaps more likely cause is that there are
>>>> packages that aren't properly registered and hence when first accessed
>>>> add an entry to the global registry invalidating my comment/assumption
>>>> about read-only thread safety.
>>>>
>>>> I do highly recommend you find the cause rather than just try to fix
>>>> the symptom.
>>>>
>>>>
>>>>> Thanks,
>>>>> Achim
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
|
|
|
| Re: EPackage.Registry concurrency issues [message #428909 is a reply to message #428898] |
Wed, 01 April 2009 16:27 |
Ed Merks
Messages: 33139
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
--------------090409000103050608080504
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Achim,
Comments below.
Achim Demelt wrote:
> Ed,
>
> This is wacky stuff!
No kidding.
> The key to the problem is concurrent access to the
> package registry during the initialization of the generated EPackage. In the
> debugger, it's very easy to produce the PackageNotFoundExceptions you
> predicted. In some cases, I've actually seen those in my logs, too, so
> you're not far off the mark. Here's how to do it:
>
> 1) Have two or more threads execute EPackage.Registry.getEPackage(...) for a
> generated package registered via the extension point.
> 2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
> 3) Have another breakpoint in the generated PackageImpl's static init()
> method.
> 4) With the first thread, step into the call to field.get(null). This
> actually initializes the package class and thus calls the static init()
> method. Execution should halt at the breakpoint.
>
The package class should be properly loaded before it's even returned,
or so I would have thought
> 5) With the second thread, step _over_ the call to field.get(null). This
> time, the classloader apparently considers the class already initialized.
> However, the eINSTANCE field is still null. So, yes, getEPackage() _will_
> receive a null value for eINSTANCE.
>
I've tried something even simpler that just tests the class loader
behavior itself. I just add this to the "standard" LibraryExample.java
public static void main(String[] args)
{
final Bundle bundle = Platform.getBundle("library.tests");
for (int i = 0; i < 2; ++i)
{
new Thread(new Runnable()
{
public void run()
{
try
{
Class<?> theClass =
Class.forName("com.example.library.LibraryPackage");
// Class theClass =
bundle.loadClass("com.example.library.LibraryPackage");
Field field = theClass.getField("eINSTANCE");
Object value = field.get(null);
System.err.println(value);
}
catch (Throwable throwable)
{
}
}
}).start();
}
if (true)
for (;;)
{
}
I see significantly different behavior depending on whether I use the
OSGi or the standard Java class loader. Here's the standard Java
behavior using Class.forName.
Note the class is initialized during the forName call, and the other
thread is stepping and remains in that state until the initialization is
done.
With OSGi, the init doesn't happen until the value is fetched.
But note that in this case too, the step over is stepping indefinitely.
I was told that the Equinox runtime will time out after 5 seconds to
prevent deadlock, but to me that sound fundamentally unsafe. It means I
can never be sure the class I'm working with is properly initialized and
I have no way of ensuring that it is.
> When reading from an XMI file, this obviously produces a
> PackageNotFoundException. But it gets even better!
>
But wait, there's more! :-P
> Now I _don't_ halt execution at EPackageDescriptor.getEPackage(). Instead, I
> only have a breakpoint in the package's static init() method directly
> _after_ the the line "isInited = true;". I also have a conditional
> breakpoint in XMLHandler.createObjectFromFactory(), with the condition
> "factory != null && newObject == null".
>
> Again, two or more threads reading from XMI. One thread halts at the
> breakpoint in init(). The other thread halts at my conditional breakpoint.
> Taking a look at the variables here, I see that I have indeed the factory I
> want (my ModelgenFactoryImpl). It refers to the correct package (my
> ModelgenPackageImpl). This package, however, has no name, no URI, and no
> contents. In fact, its "isCreated" and "isInitalized" fields are false.
>
> Both threads, the one the halted at init() and the one in
> createObjectFromFactory() refer to the exact same ModelgenPackageImpl
> instance. So it seems that the classloader does returns a non-null eINSTANCE
> field if the thread is not halted in the debugger. But the package simply
> hasn't been fully initialized yet. See attached screenshot.
>
> Now, what can we do?
>
The issue in your other note (what the EPackageImpl constructor does)
seems more of an EMF problem though rather than, what's in my opinion a
misbehaving class loader. We could add a guard like this so we only put
it in the registry if it (or its descriptor) is not in the registry:
protected EPackageImpl(String packageURI, EFactory factory)
{
super();
if (!Registry.INSTANCE.containsKey(packageURI))
{
Registry.INSTANCE.put(packageURI, this);
}
It's possible that this change could cause a second call to init() to
return null though, but clients shouldn't be calling this method anyway,
so that doesn't seem bad.
Please open a bugzilla and I'll try to consult with some OSGi experts
about this questionable class loader behavior and look into how to plug
this other hole too.
A workaround might well be as simple as calling
ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
constructor.
> For the record, I'm using EMF 2.4. The generated code is up-to-date, NS_URIs
> match.
>
> Cheers,
> Achim
>
> Ed Merks wrote:
>
>
>> Achim,
>>
>> Comments below.
>>
>> Achim Demelt wrote:
>>
>>> Ed,
>>>
>>> Thanks for your comments. Below you'll find and exception stack trace
>>> that occurred when loading a file ending with "*.modelgen". It's an XMI
>>> ClassNotFoundException, not a java.lang one.
>>>
>> That's what I would have assumed....
>>
>>> In the particular log file I'm
>>> looking at right now, I see this stack trace three times logged within
>>> one second. However, there are 42 such "*.modelgen" files in the
>>> workspace, all of which are loaded sooner or later, so the other 39 are
>>> ok. The files themselves are ok. I can open them in the editor without
>>> any problems. Very strange...
>>>
>>>
>> Indeed.
>>
>>> I don't think classloading is the issue here, since I would suspect other
>>> exceptions (java.lang ones) popping up somewhere. I'll try out your
>>> suggestion for debugging to find out more.
>>>
>>>
>> Well, as I said, it would be very odd to see a null coming back for the
>> eINSTANCE. That should be impossible. But if you look at the rest of
>> the logic for getEPackage, I don't see how multiple threads executing
>> that same code, could ever have one of them returning null...
>>
>>>
>>>
>>>> Another perhaps more likely cause is that there are
>>>> packages that aren't properly registered and hence when first accessed
>>>> add an entry to the global registry invalidating my comment/assumption
>>>> about read-only thread safety.
>>>>
>>>>
>>> How would a package not be properly registered?
>>>
>> If the uri in the plugin.xml doesn't actually match
>> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
>> it's possible for it to be out of date.
>>
>> Now that I think of it some more, if the package was null, you ought to
>> see a package not found exception. You've not indicated which version
>> of EMF you are using, but the problem sounds like this one which was
>> fixed a very long time ago.
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>>
>>
>>> All I can do is declare
>>> these extensions, right?
>>>
>>> <extension point="org.eclipse.emf.ecore.generated_package">
>>> <package
>>> uri =
>>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>>> class =
>>>
>>>
> " de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>
>>> genModel = "model/modelgen.genmodel" />
>>> </extension>
>>> <extension
>>> point="org.eclipse.emf.ecore.extension_parser">
>>> <parser
>>>
>>>
> class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>
>>> type="modelgen">
>>> </parser>
>>> </extension>
>>>
>>>
>>> Thanks,
>>> Achim
>>>
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>
>>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>> 'GeneratorConfiguration' not found.
>>>
>>>
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>
>>> 2, 276)
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>
>>> at
>>>
>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>
>>> at
>>>
>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>
>>> at
>>>
>>>
> org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>
>>> at
>>>
>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>
>>> at
>>>
>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>
>>> at
>>>
>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>
>>> at
>>>
>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>
>>> at
>>> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>>> at
>>>
>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>
>>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>> 'GeneratorConfiguration' not found.
>>>
>>>
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>
>>> 2, 276)
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>
>>> at
>>>
>>>
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>
>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>>> at
>>> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>>> at
>>>
>>>
> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>
>>> at
>>>
>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>
>>> ... 11 more
>>>
>>>
>>> Ed Merks wrote:
>>>
>>>
>>>
>>>> Achim,
>>>>
>>>> Comments below.
>>>>
>>>> Achim Demelt wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> I've been experiencing concurrency issues lately when loading models
>>>>> from XMI resources in multiple threads (Eclipse Jobs) at the same time.
>>>>>
>>>>>
>>>> There is nothing in EMF to make things that affect common data
>>>> structures be thread safe.
>>>>
>>>>
>>>>> The
>>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>>> classes.
>>>>>
>>>>>
>>>>>
>>>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>>>
>>>>
>>>>> There is no other error message. It's just that someone down the call
>>>>> stack simply returns null.
>>>>>
>>>>>
>>>>>
>>>> Again, stacks are helpful to see.
>>>>
>>>>
>>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>>> behaves differently under observation. So if I debug the code,
>>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>>> does the trick. So I've settled with that as a workaround.
>>>>>
>>>>> Out of curiosity, I'd still like to know the underlying cause of the
>>>>> problem and I investigated a bit into the EMF codebase. So here's the
>>>>> situation: My code is running in an Equinox runtime, all bundles are
>>>>> installed properly, all EPackages are registered via extensions.
>>>>> Technically, everything is fine.
>>>>>
>>>>>
>>>>>
>>>> That sounds good, but I've also never heard of a problem like this...
>>>>
>>>>
>>>>> Now I start a couple of Jobs that each reads one model that resides in
>>>>> the workspace. Each thread uses its own resource set. All models are
>>>>> based on a small number (3 or 4) of metamodels. So several Jobs ask the
>>>>> EPackage.Registry for EPackages of the same URIs, basically at the same
>>>>> time. Being registered via extensions, the Registry contains
>>>>> EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
>>>>> the actual EPackages and EFactories. Since the default
>>>>> EPackage.Registry implementation is not thread-safe, I'm guessing that
>>>>> something goes wrong here and some threads receive
>>>>> not-fully-initialized EPackages.
>>>>>
>>>>>
>>>>>
>>>> I'm not sure how that's possible though. It should be thread safe from
>>>> a read-only point of view...
>>>>
>>>>
>>>>> _If_ that is the case, overriding the default EPackage.Registry with
>>>>> one that is synchronized should solve these problems, right? Before I
>>>>> do that, though, I'd like to hear your opinion if my logic is flawed or
>>>>> if there are other possible concurrency hotspots in this scenario.
>>>>>
>>>>>
>>>>>
>>>> I know EMF has been in use for a great many years and I've never heard
>>>> of a problem that was attributed to this cause. If there is an
>>>> underlying thread safety problem here, I'd want to fix it in EMF
>>>> itself. I know it's very hard to debug threading problems but here is a
>>>> very good trick. Set breakpoints at the place you suspect has the
>>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>>> until both threads are at this breakpoint. Then step each thread
>>>> through the logic. If there is a problem, it's most likely to be here
>>>> in the resolution of the package descriptor.
>>>>
>>>> public EPackage getEPackage()
>>>> {
>>>> // First try to see if this class has an eInstance
>>>> //
>>>> try
>>>> {
>>>> Class<?> javaClass =
>>>>
>>>>
>>>>
> Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>
>>>
>>>
>>>> Field field = javaClass.getField("eINSTANCE");
>>>> Object result = field.get(null);
>>>> return (EPackage)result;
>>>> }
>>>>
>>>> I don't think it should be possible to get back a class that hasn't been
>>>> initialized. I think a class loader that returns a class that isn't
>>>> completely initialized is doing something bad. After all, how is one
>>>> supposed to use a class that might not be in a good state to be used
>>>> yet? Yet from your description, this sound likely to be at the root of
>>>> the problem. Another perhaps more likely cause is that there are
>>>> packages that aren't properly registered and hence when first accessed
>>>> add an entry to the global registry invalidating my comment/assumption
>>>> about read-only thread safety.
>>>>
>>>> I do highly recommend you find the cause rather than just try to fix the
>>>> symptom.
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>> Achim
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
>
> ------------------------------------------------------------ ------------
>
--------------090409000103050608080504
Content-Type: multipart/related;
boundary="------------010902010403060807080503"
--------------010902010403060807080503
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Achim,<br>
<br>
Comments below.<br>
<br>
Achim Demelt wrote:
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">Ed,
This is wacky stuff! </pre>
</blockquote>
No kidding.<br>
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">The key to the problem is concurrent access to the
package registry during the initialization of the generated EPackage. In the
debugger, it's very easy to produce the PackageNotFoundExceptions you
predicted. In some cases, I've actually seen those in my logs, too, so
you're not far off the mark. Here's how to do it:
1) Have two or more threads execute EPackage.Registry.getEPackage(...) for a
generated package registered via the extension point.
2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
3) Have another breakpoint in the generated PackageImpl's static init()
method.
4) With the first thread, step into the call to field.get(null). This
actually initializes the package class and thus calls the static init()
method. Execution should halt at the breakpoint.
</pre>
</blockquote>
The package class should be properly loaded before it's even returned,
or so I would have thought<br>
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">5) With the second thread, step _over_ the call to field.get(null). This
time, the classloader apparently considers the class already initialized.
However, the eINSTANCE field is still null. So, yes, getEPackage() _will_
receive a null value for eINSTANCE.
</pre>
</blockquote>
<small><big>I've tried something even simpler that just tests the class
loader behavior itself. I just add this to the "standard"
LibraryExample.java</big></small><br>
<blockquote><small> public static void main(String[] args)<br>
{<br>
final Bundle bundle = Platform.getBundle("library.tests");<br>
<br>
for (int i = 0; i < 2; ++i)<br>
{<br>
new Thread(new Runnable()<br>
{<br>
public void run()<br>
{<br>
try<br>
{<br>
Class<?> theClass =
Class.forName("com.example.library.LibraryPackage");<br>
// Class theClass =
bundle.loadClass("com.example.library.LibraryPackage");<br >
Field field = theClass.getField("eINSTANCE");<br>
Object value = field.get(null);<br>
System.err.println(value);<br>
}<br>
catch (Throwable throwable)<br>
{<br>
<br>
}<br>
}<br>
}).start();<br>
}<br>
<br>
if (true)<br>
for (;;)<br>
{<br>
<br>
}</small><br>
</blockquote>
I see significantly different behavior depending on whether I use the
OSGi or the standard Java class loader. Here's the standard Java
behavior using Class.forName.<br>
<br>
<img src="cid:part1.01070702.00080502@gmail.com" alt=""><br>
Note the class is initialized during the forName call, and the other
thread is stepping and remains in that state until the initialization
is done. <br>
<br>
With OSGi, the init doesn't happen until the value is fetched.<br>
<br>
<img src="cid:part2.03090304.02020909@gmail.com" alt=""><br>
But note that in this case too, the step over is stepping indefinitely.<br>
<br>
I was told that the Equinox runtime will time out after 5 seconds to
prevent deadlock, but to me that sound fundamentally unsafe. It means
I can never be sure the class I'm working with is properly initialized
and I have no way of ensuring that it is.<br>
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">
When reading from an XMI file, this obviously produces a
PackageNotFoundException. But it gets even better!
</pre>
</blockquote>
But wait, there's more! :-P<br>
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">
Now I _don't_ halt execution at EPackageDescriptor.getEPackage(). Instead, I
only have a breakpoint in the package's static init() method directly
_after_ the the line "isInited = true;". I also have a conditional
breakpoint in XMLHandler.createObjectFromFactory(), with the condition
"factory != null && newObject == null".
Again, two or more threads reading from XMI. One thread halts at the
breakpoint in init(). The other thread halts at my conditional breakpoint.
Taking a look at the variables here, I see that I have indeed the factory I
want (my ModelgenFactoryImpl). It refers to the correct package (my
ModelgenPackageImpl). This package, however, has no name, no URI, and no
contents. In fact, its "isCreated" and "isInitalized" fields are false.
Both threads, the one the halted at init() and the one in
createObjectFromFactory() refer to the exact same ModelgenPackageImpl
instance. So it seems that the classloader does returns a non-null eINSTANCE
field if the thread is not halted in the debugger. But the package simply
hasn't been fully initialized yet. See attached screenshot.
Now, what can we do?
</pre>
</blockquote>
The issue in your other note (what the EPackageImpl constructor does)
seems more of an EMF problem though rather than, what's in my opinion a
misbehaving class loader. We could add a guard like this so we only
put it in the registry if it (or its descriptor) is not in the registry:<small><br>
</small>
<blockquote><small> protected EPackageImpl(String packageURI, EFactory
factory)</small><br>
<small> {</small><br>
<small> super();</small><br>
<br>
<small> if (!Registry.INSTANCE.containsKey(packageURI))</small><br>
<small> {</small><br>
<small> Registry.INSTANCE.put(packageURI, this);</small><br>
<small> }</small><br>
</blockquote>
It's possible that this change could cause a second call to init() to
return null though, but clients shouldn't be calling this method
anyway, so that doesn't seem bad.<br>
<br>
Please open a bugzilla and I'll try to consult with some OSGi experts
about this questionable class loader behavior and look into how to plug
this other hole too.<br>
<br>
A workaround might well be as simple as calling
ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
constructor. <br>
<blockquote cite="mid:gqvssu$gjm$1@build.eclipse.org" type="cite">
<pre wrap="">
For the record, I'm using EMF 2.4. The generated code is up-to-date, NS_URIs
match.
Cheers,
Achim
Ed Merks wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Achim,
Comments below.
Achim Demelt wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Ed,
Thanks for your comments. Below you'll find and exception stack trace
that occurred when loading a file ending with "*.modelgen". It's an XMI
ClassNotFoundException, not a java.lang one.
</pre>
</blockquote>
<pre wrap="">That's what I would have assumed....
</pre>
<blockquote type="cite">
<pre wrap="">In the particular log file I'm
looking at right now, I see this stack trace three times logged within
one second. However, there are 42 such "*.modelgen" files in the
workspace, all of which are loaded sooner or later, so the other 39 are
ok. The files themselves are ok. I can open them in the editor without
any problems. Very strange...
</pre>
</blockquote>
<pre wrap="">Indeed.
</pre>
<blockquote type="cite">
<pre wrap="">I don't think classloading is the issue here, since I would suspect other
exceptions (java.lang ones) popping up somewhere. I'll try out your
suggestion for debugging to find out more.
</pre>
</blockquote>
<pre wrap="">Well, as I said, it would be very odd to see a null coming back for the
eINSTANCE. That should be impossible. But if you look at the rest of
the logic for getEPackage, I don't see how multiple threads executing
that same code, could ever have one of them returning null...
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> Another perhaps more likely cause is that there are
packages that aren't properly registered and hence when first accessed
add an entry to the global registry invalidating my comment/assumption
about read-only thread safety.
</pre>
</blockquote>
<pre wrap="">How would a package not be properly registered?
</pre>
</blockquote>
<pre wrap="">If the uri in the plugin.xml doesn't actually match
ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
it's possible for it to be out of date.
Now that I think of it some more, if the package was null, you ought to
see a package not found exception. You've not indicated which version
of EMF you are using, but the problem sounds like this one which was
fixed a very long time ago.
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325">https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325</a>
</pre>
<blockquote type="cite">
<pre wrap="">All I can do is declare
these extensions, right?
<extension point="org.eclipse.emf.ecore.generated_package">
<package
uri =
<a class="moz-txt-link-rfc2396E" href="http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore">"http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"</a>
class =
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> genModel = "model/modelgen.genmodel" />
</extension>
<extension
point="org.eclipse.emf.ecore.extension_parser">
<parser
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> type="modelgen">
</parser>
</extension>
Thanks,
Achim
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">2, 276)
at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">2, 276)
at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at
org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> ... 11 more
Ed Merks wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Achim,
Comments below.
Achim Demelt wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I've been experiencing concurrency issues lately when loading models
from XMI resources in multiple threads (Eclipse Jobs) at the same time.
</pre>
</blockquote>
<pre wrap="">There is nothing in EMF to make things that affect common data
structures be thread safe.
</pre>
<blockquote type="cite">
<pre wrap=""> The
XMLHandler produces ClassNotFoundExceptions when instantiating the
classes.
</pre>
</blockquote>
<pre wrap="">Stack traces are helpful. You mean the XMI one of the java.lang one?
</pre>
<blockquote type="cite">
<pre wrap="">There is no other error message. It's just that someone down the call
stack simply returns null.
</pre>
</blockquote>
<pre wrap="">Again, stacks are helpful to see.
</pre>
<blockquote type="cite">
<pre wrap="">In accordance with Heisenberg's uncertainty principle, the system
behaves differently under observation. So if I debug the code,
everything works fine. Scheduling the Jobs to run sequentially also
does the trick. So I've settled with that as a workaround.
Out of curiosity, I'd still like to know the underlying cause of the
problem and I investigated a bit into the EMF codebase. So here's the
situation: My code is running in an Equinox runtime, all bundles are
installed properly, all EPackages are registered via extensions.
Technically, everything is fine.
</pre>
</blockquote>
<pre wrap="">That sounds good, but I've also never heard of a problem like this...
</pre>
<blockquote type="cite">
<pre wrap="">Now I start a couple of Jobs that each reads one model that resides in
the workspace. Each thread uses its own resource set. All models are
based on a small number (3 or 4) of metamodels. So several Jobs ask the
EPackage.Registry for EPackages of the same URIs, basically at the same
time. Being registered via extensions, the Registry contains
EPackage.Descriptors. Upon first access, the Descriptors lazily resolve
the actual EPackages and EFactories. Since the default
EPackage.Registry implementation is not thread-safe, I'm guessing that
something goes wrong here and some threads receive
not-fully-initialized EPackages.
</pre>
</blockquote>
<pre wrap="">I'm not sure how that's possible though. It should be thread safe from
a read-only point of view...
</pre>
<blockquote type="cite">
<pre wrap="">_If_ that is the case, overriding the default EPackage.Registry with
one that is synchronized should solve these problems, right? Before I
do that, though, I'd like to hear your opinion if my logic is flawed or
if there are other possible concurrency hotspots in this scenario.
</pre>
</blockquote>
<pre wrap="">I know EMF has been in use for a great many years and I've never heard
of a problem that was attributed to this cause. If there is an
underlying thread safety problem here, I'd want to fix it in EMF
itself. I know it's very hard to debug threading problems but here is a
very good trick. Set breakpoints at the place you suspect has the
problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
until both threads are at this breakpoint. Then step each thread
through the logic. If there is a pr
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EPackage.Registry concurrency issues [message #428918 is a reply to message #428909] |
Wed, 01 April 2009 19:53 |
Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
Ed,
My comments inline.
Ed Merks wrote:
>> 1) Have two or more threads execute EPackage.Registry.getEPackage(...)
>> for a generated package registered via the extension point.
>> 2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
>> 3) Have another breakpoint in the generated PackageImpl's static init()
>> method.
>> 4) With the first thread, step into the call to field.get(null). This
>> actually initializes the package class and thus calls the static init()
>> method. Execution should halt at the breakpoint.
>>
> The package class should be properly loaded before it's even returned,
> or so I would have thought
I used to think so, too. Until today...
> I see significantly different behavior depending on whether I use the
> OSGi or the standard Java class loader. Here's the standard Java
> behavior using Class.forName.
>
Apparently, the output was lost in the newsgroup. Or my newsreader swallowed
it. But I would guess that everything works fine in this case.
>
> Note the class is initialized during the forName call, and the other
> thread is stepping and remains in that state until the initialization is
> done.
>
> With OSGi, the init doesn't happen until the value is fetched.
>
>
> But note that in this case too, the step over is stepping indefinitely.
>
> I was told that the Equinox runtime will time out after 5 seconds to
> prevent deadlock, but to me that sound fundamentally unsafe. It means I
> can never be sure the class I'm working with is properly initialized and
> I have no way of ensuring that it is.
I think I heard about such a 5 second timeout in the context of classloading
during bundle activation. If a bundle *activator* loads a class from another
bundle, and this class cannot be loaded within 5 seconds, the activator is
interrupted. But I may be wrong and there's a general 5 second timeout.
Bundle activation may in fact be another part of this puzzle. My guess is
that in your example, the "library.test" bundle is being lazily activated
during the call to bundle.loadClass(). This may lead to your second thread
being blocked until activation is done. What happens if you add an explicit
call to bundle.start() in your example?
> The issue in your other note (what the EPackageImpl constructor does)
> seems more of an EMF problem though rather than, what's in my opinion a
> misbehaving class loader. We could add a guard like this so we only put
> it in the registry if it (or its descriptor) is not in the registry:
>
> protected EPackageImpl(String packageURI, EFactory factory)
> {
> super();
>
> if (!Registry.INSTANCE.containsKey(packageURI))
> {
> Registry.INSTANCE.put(packageURI, this);
> }
>
This sounds like a good solution.
> It's possible that this change could cause a second call to init() to
> return null though, but clients shouldn't be calling this method anyway,
> so that doesn't seem bad.
>
> Please open a bugzilla and I'll try to consult with some OSGi experts
> about this questionable class loader behavior and look into how to plug
> this other hole too.
>
This one is for the EPackageImpl:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=270865
And this one for the classloading issue:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=270870
> A workaround might well be as simple as calling
> ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
> constructor.
I'm not sure this would work. If the package hasn't been fully initialized
yet and a second thread has a reference to the factory, how would it help if
the factory called ModelgenPackage.eINSTANCE.eClass()? The result would be
yet another reference to the incomplete EPackage, wouldn't it?
For the moment, I can live with my old workaround of serializing the Jobs to
avoid these concurrency issues. Still, I'm interested if the Equinox guys
can shed some light on this.
Achim
>> For the record, I'm using EMF 2.4. The generated code is up-to-date,
>> NS_URIs match.
>>
>> Cheers,
>> Achim
>>
>> Ed Merks wrote:
>>
>>
>>> Achim,
>>>
>>> Comments below.
>>>
>>> Achim Demelt wrote:
>>>
>>>> Ed,
>>>>
>>>> Thanks for your comments. Below you'll find and exception stack trace
>>>> that occurred when loading a file ending with "*.modelgen". It's an XMI
>>>> ClassNotFoundException, not a java.lang one.
>>>>
>>> That's what I would have assumed....
>>>
>>>> In the particular log file I'm
>>>> looking at right now, I see this stack trace three times logged within
>>>> one second. However, there are 42 such "*.modelgen" files in the
>>>> workspace, all of which are loaded sooner or later, so the other 39 are
>>>> ok. The files themselves are ok. I can open them in the editor without
>>>> any problems. Very strange...
>>>>
>>>>
>>> Indeed.
>>>
>>>> I don't think classloading is the issue here, since I would suspect
>>>> other exceptions (java.lang ones) popping up somewhere. I'll try out
>>>> your suggestion for debugging to find out more.
>>>>
>>>>
>>> Well, as I said, it would be very odd to see a null coming back for the
>>> eINSTANCE. That should be impossible. But if you look at the rest of
>>> the logic for getEPackage, I don't see how multiple threads executing
>>> that same code, could ever have one of them returning null...
>>>
>>>>
>>>>
>>>>> Another perhaps more likely cause is that there are
>>>>> packages that aren't properly registered and hence when first accessed
>>>>> add an entry to the global registry invalidating my comment/assumption
>>>>> about read-only thread safety.
>>>>>
>>>>>
>>>> How would a package not be properly registered?
>>>>
>>> If the uri in the plugin.xml doesn't actually match
>>> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
>>> it's possible for it to be out of date.
>>>
>>> Now that I think of it some more, if the package was null, you ought to
>>> see a package not found exception. You've not indicated which version
>>> of EMF you are using, but the problem sounds like this one which was
>>> fixed a very long time ago.
>>>
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>>>
>>>
>>>> All I can do is declare
>>>> these extensions, right?
>>>>
>>>> <extension point="org.eclipse.emf.ecore.generated_package">
>>>> <package
>>>> uri =
>>>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>>>> class =
>>>>
>>>>
>>
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>>
>>>> genModel = "model/modelgen.genmodel" />
>>>> </extension>
>>>> <extension
>>>> point="org.eclipse.emf.ecore.extension_parser">
>>>> <parser
>>>>
>>>>
>>
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>>
>>>> type="modelgen">
>>>> </parser>
>>>> </extension>
>>>>
>>>>
>>>> Thanks,
>>>> Achim
>>>>
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>>
>>>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>> 'GeneratorConfiguration' not found.
>>>>
>>>>
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>
>>>> 2, 276)
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>>
>>>> at
>>>>
>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>>
>>>> at
>>>>
>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>>
>>>> at
>>>>
org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>>>> at
>>>>
>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>>
>>>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>>>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>> 'GeneratorConfiguration' not found.
>>>>
>>>>
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>
>>>> 2, 276)
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>>
>>>> at
>>>>
>>>>
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>>
>>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>>>> at
>>>>
org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>>
>>>> at
>>>>
>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>>
>>>> ... 11 more
>>>>
>>>>
>>>> Ed Merks wrote:
>>>>
>>>>
>>>>
>>>>> Achim,
>>>>>
>>>>> Comments below.
>>>>>
>>>>> Achim Demelt wrote:
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I've been experiencing concurrency issues lately when loading models
>>>>>> from XMI resources in multiple threads (Eclipse Jobs) at the same
>>>>>> time.
>>>>>>
>>>>>>
>>>>> There is nothing in EMF to make things that affect common data
>>>>> structures be thread safe.
>>>>>
>>>>>
>>>>>> The
>>>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>>>> classes.
>>>>>>
>>>>>>
>>>>>>
>>>>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>>>>
>>>>>
>>>>>> There is no other error message. It's just that someone down the call
>>>>>> stack simply returns null.
>>>>>>
>>>>>>
>>>>>>
>>>>> Again, stacks are helpful to see.
>>>>>
>>>>>
>>>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>>>> behaves differently under observation. So if I debug the code,
>>>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>>>> does the trick. So I've settled with that as a workaround.
>>>>>>
>>>>>> Out of curiosity, I'd still like to know the underlying cause of the
>>>>>> problem and I investigated a bit into the EMF codebase. So here's the
>>>>>> situation: My code is running in an Equinox runtime, all bundles are
>>>>>> installed properly, all EPackages are registered via extensions.
>>>>>> Technically, everything is fine.
>>>>>>
>>>>>>
>>>>>>
>>>>> That sounds good, but I've also never heard of a problem like this...
>>>>>
>>>>>
>>>>>> Now I start a couple of Jobs that each reads one model that resides
>>>>>> in the workspace. Each thread uses its own resource set. All models
>>>>>> are based on a small number (3 or 4) of metamodels. So several Jobs
>>>>>> ask the EPackage.Registry for EPackages of the same URIs, basically
>>>>>> at the same time. Being registered via extensions, the Registry
>>>>>> contains EPackage.Descriptors. Upon first access, the Descriptors
>>>>>> lazily resolve the actual EPackages and EFactories. Since the default
>>>>>> EPackage.Registry implementation is not thread-safe, I'm guessing
>>>>>> that something goes wrong here and some threads receive
>>>>>> not-fully-initialized EPackages.
>>>>>>
>>>>>>
>>>>>>
>>>>> I'm not sure how that's possible though. It should be thread safe
>>>>> from a read-only point of view...
>>>>>
>>>>>
>>>>>> _If_ that is the case, overriding the default EPackage.Registry with
>>>>>> one that is synchronized should solve these problems, right? Before I
>>>>>> do that, though, I'd like to hear your opinion if my logic is flawed
>>>>>> or if there are other possible concurrency hotspots in this scenario.
>>>>>>
>>>>>>
>>>>>>
>>>>> I know EMF has been in use for a great many years and I've never heard
>>>>> of a problem that was attributed to this cause. If there is an
>>>>> underlying thread safety problem here, I'd want to fix it in EMF
>>>>> itself. I know it's very hard to debug threading problems but here is
>>>>> a
>>>>> very good trick. Set breakpoints at the place you suspect has the
>>>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>>>> until both threads are at this breakpoint. Then step each thread
>>>>> through the logic. If there is a problem, it's most likely to be here
>>>>> in the resolution of the package descriptor.
>>>>>
>>>>> public EPackage getEPackage()
>>>>> {
>>>>> // First try to see if this class has an eInstance
>>>>> //
>>>>> try
>>>>> {
>>>>> Class<?> javaClass =
>>>>>
>>>>>
>>>>>
>>
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>>
>>>>
>>>>
>>>>> Field field = javaClass.getField("eINSTANCE");
>>>>> Object result = field.get(null);
>>>>> return (EPackage)result;
>>>>> }
>>>>>
>>>>> I don't think it should be possible to get back a class that hasn't
>>>>> been
>>>>> initialized. I think a class loader that returns a class that isn't
>>>>> completely initialized is doing something bad. After all, how is one
>>>>> supposed to use a class that might not be in a good state to be used
>>>>> yet? Yet from your description, this sound likely to be at the root
>>>>> of
>>>>> the problem. Another perhaps more likely cause is that there are
>>>>> packages that aren't properly registered and hence when first accessed
>>>>> add an entry to the global registry invalidating my comment/assumption
>>>>> about read-only thread safety.
>>>>>
>>>>> I do highly recommend you find the cause rather than just try to fix
>>>>> the symptom.
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Achim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
>>
>> ------------------------------------------------------------ ------------
>>
|
|
|
| Re: EPackage.Registry concurrency issues [message #428919 is a reply to message #428918] |
Wed, 01 April 2009 22:06 |
Ed Merks
Messages: 33139
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
--------------020606090006000408050406
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Achim,
Comments below.
Achim Demelt wrote:
> Ed,
>
> My comments inline.
>
> Ed Merks wrote:
>
>
>>> 1) Have two or more threads execute EPackage.Registry.getEPackage(...)
>>> for a generated package registered via the extension point.
>>> 2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
>>> 3) Have another breakpoint in the generated PackageImpl's static init()
>>> method.
>>> 4) With the first thread, step into the call to field.get(null). This
>>> actually initializes the package class and thus calls the static init()
>>> method. Execution should halt at the breakpoint.
>>>
>>>
>> The package class should be properly loaded before it's even returned,
>> or so I would have thought
>>
>
> I used to think so, too. Until today...
>
It seems to be more lazy with OSGi...
>
>> I see significantly different behavior depending on whether I use the
>> OSGi or the standard Java class loader. Here's the standard Java
>> behavior using Class.forName.
>>
>>
>
> Apparently, the output was lost in the newsgroup. Or my newsreader swallowed
> it. But I would guess that everything works fine in this case.
>
As expected.
>
>> Note the class is initialized during the forName call, and the other
>> thread is stepping and remains in that state until the initialization is
>> done.
>>
>> With OSGi, the init doesn't happen until the value is fetched.
>>
>>
>> But note that in this case too, the step over is stepping indefinitely.
>>
>> I was told that the Equinox runtime will time out after 5 seconds to
>> prevent deadlock, but to me that sound fundamentally unsafe. It means I
>> can never be sure the class I'm working with is properly initialized and
>> I have no way of ensuring that it is.
>>
>
> I think I heard about such a 5 second timeout in the context of classloading
> during bundle activation. If a bundle *activator* loads a class from another
> bundle, and this class cannot be loaded within 5 seconds, the activator is
> interrupted. But I may be wrong and there's a general 5 second timeout.
>
> Bundle activation may in fact be another part of this puzzle. My guess is
> that in your example, the "library.test" bundle is being lazily activated
> during the call to bundle.loadClass(). This may lead to your second thread
> being blocked until activation is done. What happens if you add an explicit
> call to bundle.start() in your example?
>
I could try...
>
>> The issue in your other note (what the EPackageImpl constructor does)
>> seems more of an EMF problem though rather than, what's in my opinion a
>> misbehaving class loader. We could add a guard like this so we only put
>> it in the registry if it (or its descriptor) is not in the registry:
>>
>> protected EPackageImpl(String packageURI, EFactory factory)
>> {
>> super();
>>
>> if (!Registry.INSTANCE.containsKey(packageURI))
>> {
>> Registry.INSTANCE.put(packageURI, this);
>> }
>>
>>
>
> This sounds like a good solution.
>
>
>> It's possible that this change could cause a second call to init() to
>> return null though, but clients shouldn't be calling this method anyway,
>> so that doesn't seem bad.
>>
>> Please open a bugzilla and I'll try to consult with some OSGi experts
>> about this questionable class loader behavior and look into how to plug
>> this other hole too.
>>
>>
>
> This one is for the EPackageImpl:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=270865
>
> And this one for the classloading issue:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=270870
>
I've not been able to reproduce that one... I'll try some more
tomorrow, though I'm not sure what else to try. I'm pretty sure it's
the Sun 6.0 JVM like you tried. I'll look at the details of all the
options. Somehow I doubt it's taking 5 seconds to initialize your
package...
>
>> A workaround might well be as simple as calling
>> ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
>> constructor.
>>
>
> I'm not sure this would work. If the package hasn't been fully initialized
> yet and a second thread has a reference to the factory, how would it help if
> the factory called ModelgenPackage.eINSTANCE.eClass()?
It's worth trying. It ought to be fundamentally impossible to see an
uninitialized constant because the class didn't quite finish loading on
your thread yet...
> The result would be
> yet another reference to the incomplete EPackage, wouldn't it?
>
Perhaps not...
> For the moment, I can live with my old workaround of serializing the Jobs to
> avoid these concurrency issues. Still, I'm interested if the Equinox guys
> can shed some light on this.
>
Me too. The said you can adjust the 5 second limit. Not sure how yet.
But I'll seek knowledge and find it I'm sure!
> Achim
>
>
>>> For the record, I'm using EMF 2.4. The generated code is up-to-date,
>>> NS_URIs match.
>>>
>>> Cheers,
>>> Achim
>>>
>>> Ed Merks wrote:
>>>
>>>
>>>
>>>> Achim,
>>>>
>>>> Comments below.
>>>>
>>>> Achim Demelt wrote:
>>>>
>>>>
>>>>> Ed,
>>>>>
>>>>> Thanks for your comments. Below you'll find and exception stack trace
>>>>> that occurred when loading a file ending with "*.modelgen". It's an XMI
>>>>> ClassNotFoundException, not a java.lang one.
>>>>>
>>>>>
>>>> That's what I would have assumed....
>>>>
>>>>
>>>>> In the particular log file I'm
>>>>> looking at right now, I see this stack trace three times logged within
>>>>> one second. However, there are 42 such "*.modelgen" files in the
>>>>> workspace, all of which are loaded sooner or later, so the other 39 are
>>>>> ok. The files themselves are ok. I can open them in the editor without
>>>>> any problems. Very strange...
>>>>>
>>>>>
>>>>>
>>>> Indeed.
>>>>
>>>>
>>>>> I don't think classloading is the issue here, since I would suspect
>>>>> other exceptions (java.lang ones) popping up somewhere. I'll try out
>>>>> your suggestion for debugging to find out more.
>>>>>
>>>>>
>>>>>
>>>> Well, as I said, it would be very odd to see a null coming back for the
>>>> eINSTANCE. That should be impossible. But if you look at the rest of
>>>> the logic for getEPackage, I don't see how multiple threads executing
>>>> that same code, could ever have one of them returning null...
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Another perhaps more likely cause is that there are
>>>>>> packages that aren't properly registered and hence when first accessed
>>>>>> add an entry to the global registry invalidating my comment/assumption
>>>>>> about read-only thread safety.
>>>>>>
>>>>>>
>>>>>>
>>>>> How would a package not be properly registered?
>>>>>
>>>>>
>>>> If the uri in the plugin.xml doesn't actually match
>>>> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
>>>> it's possible for it to be out of date.
>>>>
>>>> Now that I think of it some more, if the package was null, you ought to
>>>> see a package not found exception. You've not indicated which version
>>>> of EMF you are using, but the problem sounds like this one which was
>>>> fixed a very long time ago.
>>>>
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>>>>
>>>>
>>>>
>>>>> All I can do is declare
>>>>> these extensions, right?
>>>>>
>>>>> <extension point="org.eclipse.emf.ecore.generated_package">
>>>>> <package
>>>>> uri =
>>>>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>>>>> class =
>>>>>
>>>>>
>>>>>
> " de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>
>>>
>>>
>>>>> genModel = "model/modelgen.genmodel" />
>>>>> </extension>
>>>>> <extension
>>>>> point="org.eclipse.emf.ecore.extension_parser">
>>>>> <parser
>>>>>
>>>>>
>>>>>
> class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>
>>>
>>>
>>>>> type="modelgen">
>>>>> </parser>
>>>>> </extension>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Achim
>>>>>
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>
>>>
>>>
>>>>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>>> 'GeneratorConfiguration' not found.
>>>>>
>>>>>
>>>>>
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>
>>>
>>>
>>>>> 2, 276)
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>
>>>>> at
>>>>>
>>>>>
>>>>>
> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>
>>>
>>>
>>>>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>>>>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>>> 'GeneratorConfiguration' not found.
>>>>>
>>>>>
>>>>>
> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>
>>>
>>>
>>>>> 2, 276)
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>
>>>
>>>
>>>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>>>>> at
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>
>>>
>>>
>>>>> at
>>>>>
>>>>>
>>>>>
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>
>>>
>>>
>>>>> ... 11 more
>>>>>
>>>>>
>>>>> Ed Merks wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Achim,
>>>>>>
>>>>>> Comments below.
>>>>>>
>>>>>> Achim Demelt wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've been experiencing concurrency issues lately when loading models
>>>>>>> from XMI resources in multiple threads (Eclipse Jobs) at the same
>>>>>>> time.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> There is nothing in EMF to make things that affect common data
>>>>>> structures be thread safe.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> The
>>>>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>>>>> classes.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Stack traces are helpful. You mean the XMI one of the java.lang one?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> There is no other error message. It's just that someone down the call
>>>>>>> stack simply returns null.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Again, stacks are helpful to see.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>>>>> behaves differently under observation. So if I debug the code,
>>>>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>>>>> does the trick. So I've settled with that as a workaround.
>>>>>>>
>>>>>>> Out of curiosity, I'd still like to know the underlying cause of the
>>>>>>> problem and I investigated a bit into the EMF codebase. So here's the
>>>>>>> situation: My code is running in an Equinox runtime, all bundles are
>>>>>>> installed properly, all EPackages are registered via extensions.
>>>>>>> Technically, everything is fine.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> That sounds good, but I've also never heard of a problem like this...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Now I start a couple of Jobs that each reads one model that resides
>>>>>>> in the workspace. Each thread uses its own resource set. All models
>>>>>>> are based on a small number (3 or 4) of metamodels. So several Jobs
>>>>>>> ask the EPackage.Registry for EPackages of the same URIs, basically
>>>>>>> at the same time. Being registered via extensions, the Registry
>>>>>>> contains EPackage.Descriptors. Upon first access, the Descriptors
>>>>>>> lazily resolve the actual EPackages and EFactories. Since the default
>>>>>>> EPackage.Registry implementation is not thread-safe, I'm guessing
>>>>>>> that something goes wrong here and some threads receive
>>>>>>> not-fully-initialized EPackages.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I'm not sure how that's possible though. It should be thread safe
>>>>>> from a read-only point of view...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> _If_ that is the case, overriding the default EPackage.Registry with
>>>>>>> one that is synchronized should solve these problems, right? Before I
>>>>>>> do that, though, I'd like to hear your opinion if my logic is flawed
>>>>>>> or if there are other possible concurrency hotspots in this scenario.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I know EMF has been in use for a great many years and I've never heard
>>>>>> of a problem that was attributed to this cause. If there is an
>>>>>> underlying thread safety problem here, I'd want to fix it in EMF
>>>>>> itself. I know it's very hard to debug threading problems but here is
>>>>>> a
>>>>>> very good trick. Set breakpoints at the place you suspect has the
>>>>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>>>>> until both threads are at this breakpoint. Then step each thread
>>>>>> through the logic. If there is a problem, it's most likely to be here
>>>>>> in the resolution of the package descriptor.
>>>>>>
>>>>>> public EPackage getEPackage()
>>>>>> {
>>>>>> // First try to see if this class has an eInstance
>>>>>> //
>>>>>> try
>>>>>> {
>>>>>> Class<?> javaClass =
>>>>>>
>>>>>>
>>>>>>
>>>>>>
> Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>
>>>
>>>
>>>>>
>>>>>
>>>>>
>>>>>> Field field = javaClass.getField("eINSTANCE");
>>>>>> Object result = field.get(null);
>>>>>> return (EPackage)result;
>>>>>> }
>>>>>>
>>>>>> I don't think it should be possible to get back a class that hasn't
>>>>>> been
>>>>>> initialized. I think a class loader that returns a class that isn't
>>>>>> completely initialized is doing something bad. After all, how is one
>>>>>> supposed to use a class that might not be in a good state to be used
>>>>>> yet? Yet from your description, this sound likely to be at the root
>>>>>> of
>>>>>> the problem. Another perhaps more likely cause is that there are
>>>>>> packages that aren't properly registered and hence when first accessed
>>>>>> add an entry to the global registry invalidating my comment/assumption
>>>>>> about read-only thread safety.
>>>>>>
>>>>>> I do highly recommend you find the cause rather than just try to fix
>>>>>> the symptom.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Achim
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>> ------------------------------------------------------------ ------------
>>>
>>>
>
>
>
--------------020606090006000408050406
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Achim,<br>
<br>
Comments below.<br>
<br>
Achim Demelt wrote:
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">Ed,
My comments inline.
Ed Merks wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">1) Have two or more threads execute EPackage.Registry.getEPackage(...)
for a generated package registered via the extension point.
2) Halt both threads at the beginning of EPackageDescriptor.getEPackage()
3) Have another breakpoint in the generated PackageImpl's static init()
method.
4) With the first thread, step into the call to field.get(null). This
actually initializes the package class and thus calls the static init()
method. Execution should halt at the breakpoint.
</pre>
</blockquote>
<pre wrap="">The package class should be properly loaded before it's even returned,
or so I would have thought
</pre>
</blockquote>
<pre wrap=""><!---->
I used to think so, too. Until today...
</pre>
</blockquote>
It seems to be more lazy with OSGi...<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">I see significantly different behavior depending on whether I use the
OSGi or the standard Java class loader. Here's the standard Java
behavior using Class.forName.
</pre>
</blockquote>
<pre wrap=""><!---->
Apparently, the output was lost in the newsgroup. Or my newsreader swallowed
it. But I would guess that everything works fine in this case.
</pre>
</blockquote>
As expected.<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Note the class is initialized during the forName call, and the other
thread is stepping and remains in that state until the initialization is
done.
With OSGi, the init doesn't happen until the value is fetched.
But note that in this case too, the step over is stepping indefinitely.
I was told that the Equinox runtime will time out after 5 seconds to
prevent deadlock, but to me that sound fundamentally unsafe. It means I
can never be sure the class I'm working with is properly initialized and
I have no way of ensuring that it is.
</pre>
</blockquote>
<pre wrap=""><!---->
I think I heard about such a 5 second timeout in the context of classloading
during bundle activation. If a bundle *activator* loads a class from another
bundle, and this class cannot be loaded within 5 seconds, the activator is
interrupted. But I may be wrong and there's a general 5 second timeout.
Bundle activation may in fact be another part of this puzzle. My guess is
that in your example, the "library.test" bundle is being lazily activated
during the call to bundle.loadClass(). This may lead to your second thread
being blocked until activation is done. What happens if you add an explicit
call to bundle.start() in your example?
</pre>
</blockquote>
I could try...<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">The issue in your other note (what the EPackageImpl constructor does)
seems more of an EMF problem though rather than, what's in my opinion a
misbehaving class loader. We could add a guard like this so we only put
it in the registry if it (or its descriptor) is not in the registry:
protected EPackageImpl(String packageURI, EFactory factory)
{
super();
if (!Registry.INSTANCE.containsKey(packageURI))
{
Registry.INSTANCE.put(packageURI, this);
}
</pre>
</blockquote>
<pre wrap=""><!---->
This sounds like a good solution.
</pre>
<blockquote type="cite">
<pre wrap="">It's possible that this change could cause a second call to init() to
return null though, but clients shouldn't be calling this method anyway,
so that doesn't seem bad.
Please open a bugzilla and I'll try to consult with some OSGi experts
about this questionable class loader behavior and look into how to plug
this other hole too.
</pre>
</blockquote>
<pre wrap=""><!---->
This one is for the EPackageImpl:
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=270865">https://bugs.eclipse.org/bugs/show_bug.cgi?id=270865</a>
And this one for the classloading issue:
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=270870">https://bugs.eclipse.org/bugs/show_bug.cgi?id=270870</a>
</pre>
</blockquote>
I've not been able to reproduce that one... I'll try some more
tomorrow, though I'm not sure what else to try. I'm pretty sure it's
the Sun 6.0 JVM like you tried. I'll look at the details of all the
options. Somehow I doubt it's taking 5 seconds to initialize your
package...<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">A workaround might well be as simple as calling
ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
constructor.
</pre>
</blockquote>
<pre wrap=""><!---->
I'm not sure this would work. If the package hasn't been fully initialized
yet and a second thread has a reference to the factory, how would it help if
the factory called ModelgenPackage.eINSTANCE.eClass()? </pre>
</blockquote>
It's worth trying. It ought to be fundamentally impossible to see an
uninitialized constant because the class didn't quite finish loading on
your thread yet...<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">The result would be
yet another reference to the incomplete EPackage, wouldn't it?
</pre>
</blockquote>
Perhaps not...<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
For the moment, I can live with my old workaround of serializing the Jobs to
avoid these concurrency issues. Still, I'm interested if the Equinox guys
can shed some light on this.
</pre>
</blockquote>
Me too. The said you can adjust the 5 second limit. Not sure how
yet. But I'll seek knowledge and find it I'm sure!<br>
<blockquote cite="mid:gr0gn7$91m$1@build.eclipse.org" type="cite">
<pre wrap="">
Achim
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">For the record, I'm using EMF 2.4. The generated code is up-to-date,
NS_URIs match.
Cheers,
Achim
Ed Merks wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Achim,
Comments below.
Achim Demelt wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Ed,
Thanks for your comments. Below you'll find and exception stack trace
that occurred when loading a file ending with "*.modelgen". It's an XMI
ClassNotFoundException, not a java.lang one.
</pre>
</blockquote>
<pre wrap="">That's what I would have assumed....
</pre>
<blockquote type="cite">
<pre wrap="">In the particular log file I'm
looking at right now, I see this stack trace three times logged within
one second. However, there are 42 such "*.modelgen" files in the
workspace, all of which are loaded sooner or later, so the other 39 are
ok. The files themselves are ok. I can open them in the editor without
any problems. Very strange...
</pre>
</blockquote>
<pre wrap="">Indeed.
</pre>
<blockquote type="cite">
<pre wrap="">I don't think classloading is the issue here, since I would suspect
other exceptions (java.lang ones) popping up somewhere. I'll try out
your suggestion for debugging to find out more.
</pre>
</blockquote>
<pre wrap="">Well, as I said, it would be very odd to see a null coming back for the
eINSTANCE. That should be impossible. But if you look at the rest of
the logic for getEPackage, I don't see how multiple threads executing
that same code, could ever have one of them returning null...
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap=""> Another perhaps more likely cause is that there are
packages that aren't properly registered and hence when first accessed
add an entry to the global registry invalidating my comment/assumption
about read-only thread safety.
</pre>
</blockquote>
<pre wrap="">How would a package not be properly registered?
</pre>
</blockquote>
<pre wrap="">If the uri in the plugin.xml doesn't actually match
ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
it's possible for it to be out of date.
Now that I think of it some more, if the package was null, you ought to
see a package not found exception. You've not indicated which version
of EMF you are using, but the problem sounds like this one which was
fixed a very long time ago.
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325">https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325</a>
</pre>
<blockquote type="cite">
<pre wrap="">All I can do is declare
these extensions, right?
<extension point="org.eclipse.emf.ecore.generated_package">
<package
uri =
<a class="moz-txt-link-rfc2396E" href="http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore">"http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"</a>
class =
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!---->" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> genModel = "model/modelgen.genmodel" />
</extension>
<extension
point="org.eclipse.emf.ecore.extension_parser">
<parser
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!---->class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> type="modelgen">
</parser>
</extension>
Thanks,
Achim
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">2, 276)
at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
'GeneratorConfiguration' not found.
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> (platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">2, 276)
at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""> at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----> org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
</pre>
<blockquote type="cite"
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EPackage.Registry concurrency issues [message #428930 is a reply to message #428919] |
Thu, 02 April 2009 08:20 |
Achim Demelt
Messages: 160
Registered: July 2009 |
Senior Member |
|
|
Let's continue with the details of this discussion in the Bugzillas I've
opened. We've produced enough chatter already for Google to point anyone to
this thread...
Achim
Ed Merks wrote:
> Achim,
>
> Comments below.
>
> Achim Demelt wrote:
>> Ed,
>>
>> My comments inline.
>>
>> Ed Merks wrote:
>>
>>
>>>> 1) Have two or more threads execute EPackage.Registry.getEPackage(...)
>>>> for a generated package registered via the extension point.
>>>> 2) Halt both threads at the beginning of
>>>> EPackageDescriptor.getEPackage() 3) Have another breakpoint in the
>>>> generated PackageImpl's static init() method.
>>>> 4) With the first thread, step into the call to field.get(null). This
>>>> actually initializes the package class and thus calls the static init()
>>>> method. Execution should halt at the breakpoint.
>>>>
>>>>
>>> The package class should be properly loaded before it's even returned,
>>> or so I would have thought
>>>
>>
>> I used to think so, too. Until today...
>>
> It seems to be more lazy with OSGi...
>>
>>> I see significantly different behavior depending on whether I use the
>>> OSGi or the standard Java class loader. Here's the standard Java
>>> behavior using Class.forName.
>>>
>>>
>>
>> Apparently, the output was lost in the newsgroup. Or my newsreader
>> swallowed it. But I would guess that everything works fine in this case.
>>
> As expected.
>>
>>> Note the class is initialized during the forName call, and the other
>>> thread is stepping and remains in that state until the initialization is
>>> done.
>>>
>>> With OSGi, the init doesn't happen until the value is fetched.
>>>
>>>
>>> But note that in this case too, the step over is stepping indefinitely.
>>>
>>> I was told that the Equinox runtime will time out after 5 seconds to
>>> prevent deadlock, but to me that sound fundamentally unsafe. It means I
>>> can never be sure the class I'm working with is properly initialized and
>>> I have no way of ensuring that it is.
>>>
>>
>> I think I heard about such a 5 second timeout in the context of
>> classloading during bundle activation. If a bundle *activator* loads a
>> class from another bundle, and this class cannot be loaded within 5
>> seconds, the activator is interrupted. But I may be wrong and there's a
>> general 5 second timeout.
>>
>> Bundle activation may in fact be another part of this puzzle. My guess is
>> that in your example, the "library.test" bundle is being lazily activated
>> during the call to bundle.loadClass(). This may lead to your second
>> thread being blocked until activation is done. What happens if you add an
>> explicit call to bundle.start() in your example?
>>
> I could try...
>>
>>> The issue in your other note (what the EPackageImpl constructor does)
>>> seems more of an EMF problem though rather than, what's in my opinion a
>>> misbehaving class loader. We could add a guard like this so we only put
>>> it in the registry if it (or its descriptor) is not in the registry:
>>>
>>> protected EPackageImpl(String packageURI, EFactory factory)
>>> {
>>> super();
>>>
>>> if (!Registry.INSTANCE.containsKey(packageURI))
>>> {
>>> Registry.INSTANCE.put(packageURI, this);
>>> }
>>>
>>>
>>
>> This sounds like a good solution.
>>
>>
>>> It's possible that this change could cause a second call to init() to
>>> return null though, but clients shouldn't be calling this method anyway,
>>> so that doesn't seem bad.
>>>
>>> Please open a bugzilla and I'll try to consult with some OSGi experts
>>> about this questionable class loader behavior and look into how to plug
>>> this other hole too.
>>>
>>>
>>
>> This one is for the EPackageImpl:
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=270865
>>
>> And this one for the classloading issue:
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=270870
>>
> I've not been able to reproduce that one... I'll try some more
> tomorrow, though I'm not sure what else to try. I'm pretty sure it's
> the Sun 6.0 JVM like you tried. I'll look at the details of all the
> options. Somehow I doubt it's taking 5 seconds to initialize your
> package...
>>
>>> A workaround might well be as simple as calling
>>> ModelgenPackage.eINSTANCE.eClass() in your resource factory impl's
>>> constructor.
>>>
>>
>> I'm not sure this would work. If the package hasn't been fully
>> initialized yet and a second thread has a reference to the factory, how
>> would it help if the factory called ModelgenPackage.eINSTANCE.eClass()?
> It's worth trying. It ought to be fundamentally impossible to see an
> uninitialized constant because the class didn't quite finish loading on
> your thread yet...
>> The result would be
>> yet another reference to the incomplete EPackage, wouldn't it?
>>
> Perhaps not...
>> For the moment, I can live with my old workaround of serializing the Jobs
>> to avoid these concurrency issues. Still, I'm interested if the Equinox
>> guys can shed some light on this.
>>
> Me too. The said you can adjust the 5 second limit. Not sure how yet.
> But I'll seek knowledge and find it I'm sure!
>> Achim
>>
>>
>>>> For the record, I'm using EMF 2.4. The generated code is up-to-date,
>>>> NS_URIs match.
>>>>
>>>> Cheers,
>>>> Achim
>>>>
>>>> Ed Merks wrote:
>>>>
>>>>
>>>>
>>>>> Achim,
>>>>>
>>>>> Comments below.
>>>>>
>>>>> Achim Demelt wrote:
>>>>>
>>>>>
>>>>>> Ed,
>>>>>>
>>>>>> Thanks for your comments. Below you'll find and exception stack trace
>>>>>> that occurred when loading a file ending with "*.modelgen". It's an
>>>>>> XMI ClassNotFoundException, not a java.lang one.
>>>>>>
>>>>>>
>>>>> That's what I would have assumed....
>>>>>
>>>>>
>>>>>> In the particular log file I'm
>>>>>> looking at right now, I see this stack trace three times logged
>>>>>> within one second. However, there are 42 such "*.modelgen" files in
>>>>>> the workspace, all of which are loaded sooner or later, so the other
>>>>>> 39 are ok. The files themselves are ok. I can open them in the editor
>>>>>> without any problems. Very strange...
>>>>>>
>>>>>>
>>>>>>
>>>>> Indeed.
>>>>>
>>>>>
>>>>>> I don't think classloading is the issue here, since I would suspect
>>>>>> other exceptions (java.lang ones) popping up somewhere. I'll try out
>>>>>> your suggestion for debugging to find out more.
>>>>>>
>>>>>>
>>>>>>
>>>>> Well, as I said, it would be very odd to see a null coming back for
>>>>> the
>>>>> eINSTANCE. That should be impossible. But if you look at the rest of
>>>>> the logic for getEPackage, I don't see how multiple threads executing
>>>>> that same code, could ever have one of them returning null...
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Another perhaps more likely cause is that there are
>>>>>>> packages that aren't properly registered and hence when first
>>>>>>> accessed add an entry to the global registry invalidating my
>>>>>>> comment/assumption about read-only thread safety.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> How would a package not be properly registered?
>>>>>>
>>>>>>
>>>>> If the uri in the plugin.xml doesn't actually match
>>>>> ModelgenPackage.eNS_URI. Given that plugin.xml doesn't regenerate,
>>>>> it's possible for it to be out of date.
>>>>>
>>>>> Now that I think of it some more, if the package was null, you ought
>>>>> to
>>>>> see a package not found exception. You've not indicated which version
>>>>> of EMF you are using, but the problem sounds like this one which was
>>>>> fixed a very long time ago.
>>>>>
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=91325
>>>>>
>>>>>
>>>>>
>>>>>> All I can do is declare
>>>>>> these extensions, right?
>>>>>>
>>>>>> <extension point="org.eclipse.emf.ecore.generated_package">
>>>>>> <package
>>>>>> uri =
>>>>>> "http://www.exxcellent.de/orchideo/engine/1.0/modelgen.ecore"
>>>>>> class =
>>>>>>
>>>>>>
>>>>>>
>>
" de.exxcellent.orchideo.engine.generator.configuration.modelg en.ModelgenPackage "
>>
>>>>
>>>>
>>>>>> genModel = "model/modelgen.genmodel" />
>>>>>> </extension>
>>>>>> <extension
>>>>>> point="org.eclipse.emf.ecore.extension_parser">
>>>>>> <parser
>>>>>>
>>>>>>
>>>>>>
>>
class=" de.exxcellent.orchideo.engine.generator.configuration.modelg en.util.ModelgenResourceFactoryImpl "
>>
>>>>
>>>>
>>>>>> type="modelgen">
>>>>>> </parser>
>>>>>> </extension>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Achim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1Diagnos ticWrappedException:
>>
>>>>
>>>>
>>>>>> org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>>>> 'GeneratorConfiguration' not found.
>>>>>>
>>>>>>
>>>>>>
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>
>>>>
>>>>
>>>>>> 2, 276)
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDe mandLoadException(ResourceSetImpl.java:315)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:274)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResou rce(ResourceSetImpl.java:397)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager.isModelFile(DynamicRegistryManager.java:177)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$RegistrationVisitor.visit(DynamicRegistryManager.java: 623)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.core.internal.resources.Resource$1.visitElement( Resource.java:58)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:81)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.doItera tion(ElementTreeIterator.java:85)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.core.internal.watson.ElementTreeIterator.iterate (ElementTreeIterator.java:119)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>> org.eclipse.core.internal.resources.Resource.accept(Resource .java:68)
>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
de.exxcellent.orchideo.engine.registration.DynamicRegistryMa nager$WorkspaceRegistrationJob.run(DynamicRegistryManager.ja va:602)
>>
>>>>
>>>>
>>>>>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
>>>>>> Caused by: org.eclipse.emf.ecore.xmi.ClassNotFoundException: Class
>>>>>> 'GeneratorConfiguration' not found.
>>>>>>
>>>>>>
>>>>>>
>>
(platform:/resource/de.exxcellent.orchideo.objects.aspect.tr aversal/configuration/traversal.modelgen,
>>
>>>>
>>>>
>>>>>> 2, 276)
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2229)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.validateCreateObje ctFromFactory(XMLHandler.java:2220)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createObjectByType (XMLHandler.java:1318)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.createTopObject(XM LHandler.java:1454)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.processElement(XML Handler.java:1019)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.processElement(XMI Handler.java:83)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:1001)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLHandler.startElement(XMLHa ndler.java:712)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMIHandler.startElement(XMIHa ndler.java:167)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .startElement(AbstractSAXParser.java:533)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator. startElement(XMLDTDValidator.java:798)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.j ava:878)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerIm pl$ContentDispatcher.scanRootElementHook(XMLDocumentScannerI mpl.java:1157)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFra gmentScannerImpl.java:1794)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentS cannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java: 368)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:834)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XML11Configuratio n.parse(XML11Configuration.java:764)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(X MLParser.java:148)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser .parse(AbstractSAXParser.java:1242)
>>
>>>>
>>>>
>>>>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
>>>>>> at
>>>>>>
>>>>>>
>> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl. java:181)
>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLRes ourceImpl.java:180)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1445)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(Resour ceImpl.java:1241)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo ad(ResourceSetImpl.java:255)
>>
>>>>
>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>>>
>>
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLo adHelper(ResourceSetImpl.java:270)
>>
>>>>
>>>>
>>>>>> ... 11 more
>>>>>>
>>>>>>
>>>>>> Ed Merks wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Achim,
>>>>>>>
>>>>>>> Comments below.
>>>>>>>
>>>>>>> Achim Demelt wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've been experiencing concurrency issues lately when loading
>>>>>>>> models from XMI resources in multiple threads (Eclipse Jobs) at the
>>>>>>>> same time.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> There is nothing in EMF to make things that affect common data
>>>>>>> structures be thread safe.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> The
>>>>>>>> XMLHandler produces ClassNotFoundExceptions when instantiating the
>>>>>>>> classes.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Stack traces are helpful. You mean the XMI one of the java.lang
>>>>>>> one?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> There is no other error message. It's just that someone down the
>>>>>>>> call stack simply returns null.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Again, stacks are helpful to see.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> In accordance with Heisenberg's uncertainty principle, the system
>>>>>>>> behaves differently under observation. So if I debug the code,
>>>>>>>> everything works fine. Scheduling the Jobs to run sequentially also
>>>>>>>> does the trick. So I've settled with that as a workaround.
>>>>>>>>
>>>>>>>> Out of curiosity, I'd still like to know the underlying cause of
>>>>>>>> the problem and I investigated a bit into the EMF codebase. So
>>>>>>>> here's the situation: My code is running in an Equinox runtime, all
>>>>>>>> bundles are installed properly, all EPackages are registered via
>>>>>>>> extensions. Technically, everything is fine.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> That sounds good, but I've also never heard of a problem like
>>>>>>> this...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Now I start a couple of Jobs that each reads one model that resides
>>>>>>>> in the workspace. Each thread uses its own resource set. All models
>>>>>>>> are based on a small number (3 or 4) of metamodels. So several Jobs
>>>>>>>> ask the EPackage.Registry for EPackages of the same URIs, basically
>>>>>>>> at the same time. Being registered via extensions, the Registry
>>>>>>>> contains EPackage.Descriptors. Upon first access, the Descriptors
>>>>>>>> lazily resolve the actual EPackages and EFactories. Since the
>>>>>>>> default EPackage.Registry implementation is not thread-safe, I'm
>>>>>>>> guessing that something goes wrong here and some threads receive
>>>>>>>> not-fully-initialized EPackages.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> I'm not sure how that's possible though. It should be thread safe
>>>>>>> from a read-only point of view...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> _If_ that is the case, overriding the default EPackage.Registry
>>>>>>>> with one that is synchronized should solve these problems, right?
>>>>>>>> Before I do that, though, I'd like to hear your opinion if my logic
>>>>>>>> is flawed or if there are other possible concurrency hotspots in
>>>>>>>> this scenario.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> I know EMF has been in use for a great many years and I've never
>>>>>>> heard
>>>>>>> of a problem that was attributed to this cause. If there is an
>>>>>>> underlying thread safety problem here, I'd want to fix it in EMF
>>>>>>> itself. I know it's very hard to debug threading problems but here
>>>>>>> is a
>>>>>>> very good trick. Set breakpoints at the place you suspect has the
>>>>>>> problem. I.e., EPackageRegistryImpl.getEPackage(). Run each thread
>>>>>>> until both threads are at this breakpoint. Then step each thread
>>>>>>> through the logic. If there is a problem, it's most likely to be
>>>>>>> here in the resolution of the package descriptor.
>>>>>>>
>>>>>>> public EPackage getEPackage()
>>>>>>> {
>>>>>>> // First try to see if this class has an eInstance
>>>>>>> //
>>>>>>> try
>>>>>>> {
>>>>>>> Class<?> javaClass =
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
Platform.getBundle(element.getDeclaringExtension().getContri butor().getName()).loadClass(element.getAttribute(attributeN ame));
>>
>>>>
>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Field field = javaClass.getField("eINSTANCE");
>>>>>>> Object result = field.get(null);
>>>>>>> return (EPackage)result;
>>>>>>> }
>>>>>>>
>>>>>>> I don't think it should be possible to get back a class that hasn't
>>>>>>> been
>>>>>>> initialized. I think a class loader that returns a class that isn't
>>>>>>> completely initialized is doing something bad. After all, how is
>>>>>>> one supposed to use a class that might not be in a good state to be
>>>>>>> used
>>>>>>> yet? Yet from your description, this sound likely to be at the
>>>>>>> root of
>>>>>>> the problem. Another perhaps more likely cause is that there are
>>>>>>> packages that aren't properly registered and hence when first
>>>>>>> accessed add an entry to the global registry invalidating my
>>>>>>> comment/assumption about read-only thread safety.
>>>>>>>
>>>>>>> I do highly recommend you find the cause rather than just try to fix
>>>>>>> the symptom.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Achim
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
------------------------------------------------------------ ------------
>>>>
>>>>
>>
>>
>>
|
|
|
Goto Forum:
Current Time: Tue Apr 23 07:23:31 GMT 2024
Powered by FUDForum. Page generated in 0.02838 seconds |
] 
Search
Help
Register
Login
Home
