Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [EMF] long loading time(load time / lazy ?)
[EMF] long loading time [message #1403905] Mon, 28 July 2014 14:12 Go to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Hello,

I'm new in this forum.
I took over an ECORE model development and the associated functions, but I'm not an expert.

The Architecture.ecore defines our model and I have several implementations of this module for different blocks (block1.architecture, block2.architecture...). These blockX.architecture are btw 1MB and 4MB.

For a specific function I need to parse all these implementations. So I have created a top.architecture, in which I copy all blocks (no link, no reference, a stupid copy/paste).
ROOT
- Block1
- Block2
- ...
- Block8
The result is a 20MB file.
When I load this file under Eclipse, it takes 25mn.

1/ What can I do to reduce loading time (even a 4MB file is long) ?
Is there a way to have a loading log ? Today I have no information about the ongoing actions during these 25mn.

2/ How could I avoid this stupid copy/paste in top.architecture and link or reference each Block.architecture, keeping the capabitility to parse the associated (XMI) file.
I heard about lazy loading, but I can't make the link with my case...

Thanks in advance for your help,
Regards,
Remi
Re: [EMF] long loading time [message #1403966 is a reply to message #1403905] Mon, 28 July 2014 18:51 Go to previous messageGo to next message
Felix Dorner is currently offline Felix DornerFriend
Messages: 295
Registered: March 2012
Senior Member
On 28/07/2014 19:01, Remi ADAM wrote:
> Hello,
>
> I'm new in this forum.
> I took over an ECORE model development and the associated functions, but
> I'm not an expert.
>
> The Architecture.ecore defines our model and I have several
> implementations of this module for different blocks
> (block1.architecture, block2.architecture...). These blockX.architecture
> are btw 1MB and 4MB.
>
> For a specific function I need to parse all these implementations. So I
> have created a top.architecture, in which I copy all blocks (no link, no
> reference, a stupid copy/paste).
> ROOT
> - Block1
> - Block2
> - ...
> - Block8
> The result is a 20MB file.
> When I load this file under Eclipse, it takes 25mn.

That sounds too long. To get a quick idea of what's going on you could
debug this and hit stop/play a few times in a row, waiting a few seconds
in between. If you see the same stack trace more than twice or even
once, and it contains no sensitive data, you can post it here, or even
maybe find out yourself what's wrong. You could also show us the
contents of ArchitectureResourceImpl to see what load options are specified.
Re: [EMF] long loading time [message #1403995 is a reply to message #1403905] Tue, 29 July 2014 04:22 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 28/07/2014 7:01 PM, Remi ADAM wrote:
> Hello,
>
> I'm new in this forum.
> I took over an ECORE model development and the associated functions,
> but I'm not an expert.
>
> The Architecture.ecore defines our model and I have several
> implementations of this module for different blocks
> (block1.architecture, block2.architecture...). These
> blockX.architecture are btw 1MB and 4MB.
>
> For a specific function I need to parse all these implementations. So
> I have created a top.architecture, in which I copy all blocks (no
> link, no reference, a stupid copy/paste).
> ROOT
> - Block1
> - Block2
> - ...
> - Block8
> The result is a 20MB file.
> When I load this file under Eclipse, it takes 25mn.
>
> 1/ What can I do to reduce loading time (even a 4MB file is long) ?
> Is there a way to have a loading log ? Today I have no information
> about the ongoing actions during these 25mn.
I expect there are some intra-resource references involved, not only
containment references and attributes. Perhaps you've even defined
attributes that are IDs, i.e., (EAttribute.isID is true)? In that case,
it's likely ID resolution that's very slow and then
org.eclipse.emf.ecore.xmi.XMLResource.OPTION_DEFER_IDREF_RESOLUTION
would help a lot, especially in combination with
org.eclipse.emf.ecore.resource.impl.ResourceImpl.setIntrinsicIDToEObjectMap(Map<String,
EObject>) so that all IDs are resolved after the entire resource is
read, and when the first ID is resolve, the entire resource is traversed
to build a map such that each ID resolution takes O(1) time rather than
O(n) where n is the number of contained objects in the resource.
>
> 2/ How could I avoid this stupid copy/paste in top.architecture and
> link or reference each Block.architecture, keeping the capabitility to
> parse the associated (XMI) file.
Ecore supports cross resource containment as well. So if the EReference
has isContainment true and isResolveProxies true, and you set the
GenModel property Containment Proxies to true, you can (in the above
example, put each Blockx in a separate resource from Root.
> I heard about lazy loading, but I can't make the link with my case...
As Felix suggests, you'll have to try a few things on your own. His
suggested "sampling technique" of stopping the debugger occasionally and
see what's going on, should help. If you want more general help, you'll
have to provide a bit more detail about your model, particularly about
your cross reference structure, i.e., do you see a lot of hrefs in your
serialization and if so, what do they look like?
>
> Thanks in advance for your help,
> Regards,
> Remi


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404025 is a reply to message #1403995] Tue, 29 July 2014 08:33 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Hi Felix and Ed and thanks for your support,

I tried to stop during top.architecture loading. Stacks are not exactly the same, but there are only few differences (cf attached T0.txt, T0+5s.txt...). Do you see something strange ?

I use some unique IDs, but no trace of EAttribute.isID, now I have a useUUIDs set at true in the ArchitectureResourceImpl.java. In the instanciation it looks like <documents xmi:id="_vKb95AjeEeSimJP2Do7blw">. There are a lot of intra references, using these IDs (cf top.architecture, which is an abstract of the 20MB file). Would the two options, you mentionned, help in my case?

In the 20MB (111460 lines), I have 686 href looking like:
<feature xsi:type="architecture:Feature" href="features.architecture#_MtnMDGctEeOnxpO1IkhpXA"/>

Remi
  • Attachment: T0.txt
    (Size: 10.11KB, Downloaded 340 times)
  • Attachment: T0+5s.txt
    (Size: 9.77KB, Downloaded 446 times)
  • Attachment: T0+10mn.txt
    (Size: 10.22KB, Downloaded 639 times)
  • Attachment: T0+15s.txt
    (Size: 10.28KB, Downloaded 438 times)
  • Attachment: top.architecture
    (Size: 10.59KB, Downloaded 358 times)
Re: [EMF] long loading time [message #1404026 is a reply to message #1404025] Tue, 29 July 2014 08:34 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
I attach the ArchitectureResourceImpl.java and ArchitectureResourceFactoryImpl.java files.
Re: [EMF] long loading time [message #1404035 is a reply to message #1404026] Tue, 29 July 2014 09:21 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Yes, and I suspected you have ID attributes and are resolving them
immediately as they are encountered in the resource.

FeatureMapUtil.isFeatureMap(EStructuralFeature) line: 150
EClassImpl$1EAllStructuralFeaturesList.init() line: 832
EClassImpl$1EAllStructuralFeaturesList.containments() line: 851
EContentsEList<E>.createEContentsEList(EObject) line: 55
TypedefImpl(MinimalEObjectImpl).eContents() line: 739
EcoreUtil$ProperContentIterator<E>.<init>(EObject, boolean) line: 1247
EcoreUtil$ProperContentIterator<E>.<init>(EObject) line: 1241
ResourceImpl$5.getChildren(Object) line: 549
ResourceImpl$5(AbstractTreeIterator<E>).next() line: 144
ArchitectureResourceImpl(ResourceImpl).getEObjectByID(String) line: 830
ArchitectureResourceImpl(XMLResourceImpl).getEObjectByID(String) line: 566
ArchitectureResourceImpl(ResourceImpl).getEObject(String) line: 768
SAXXMIHandler(XMLHandler).setValueFromId(EObject, EReference, String) line: 2816

This will result in nasty O(n^2) performance. Deferring ID resolution
and setting a map (both best done in your resource factory), should
eliminate that bottleneck. If you don't yet have a generated resource
factory, in the GenPackage (the second level nested object when you open
the *.genmodel), set the "Resource Type property" to "XMI" and generate
one. Make your changes to set options and configure your
XyzResourceImpl in the createResource method of your generated
XyzResourceFactoryImpl (and double check that it's registered in the
generated plugin.xml).


On 29/07/2014 10:34 AM, Remi ADAM wrote:
> I attach the ArchitectureResourceImpl.java and ArchitectureResourceFactoryImpl.java files.


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404066 is a reply to message #1404035] Tue, 29 July 2014 12:17 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Thanks Ed,

It works !! Incredible, it takes 1s vs. 25mn before.
The resource factory was already generated in my case, I have just updated the methos as below:

@Override
public Resource createResource(URI uri) 
   {
   XMLResourceImpl result = new ArchitectureResourceImpl(uri);
   result.getDefaultLoadOptions().put(XMLResource.OPTION_DEFER_IDREF_RESOLUTION, Boolean.TRUE);
   result.setIntrinsicIDToEObjectMap(new HashMap());
   return result;
   }


Now I have to investigate cross resource containment.
Thank you again.
Re: [EMF] long loading time [message #1404074 is a reply to message #1404066] Tue, 29 July 2014 12:37 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Tell your boss that you worked a whole week on performance tuning and
achieved a 2500% improvement. Time for a bonus. :-P


On 29/07/2014 2:17 PM, Remi ADAM wrote:
> Thanks Ed,
>
> It works !! Incredible, it takes 1s vs. 25mn before.
> The resource factory was already generated in my case, I have just
> updated the methos as below:
>
> @Override
> public Resource createResource(URI uri) {
> XMLResourceImpl result = new ArchitectureResourceImpl(uri);
> result.getDefaultLoadOptions().put(XMLResource.OPTION_DEFER_IDREF_RESOLUTION,
> Boolean.TRUE);
> result.setIntrinsicIDToEObjectMap(new HashMap());
> return result;
> }
>
>
> Now I have to investigate cross resource containment. Thank you again.


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404086 is a reply to message #1404074] Tue, 29 July 2014 14:35 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Ed,

Good idea, I will try Cool

I have implemented the cross resource containment, it works.
In my new top.architecture, I add one folder (BlockX_virtual) for each linked block and use right clic to choose "Control..." and I choose the expected BlockX.architecture.
This has 2 consequences :
- In the original BlockX.architecture, it adds a node BlockX_virtual at the same level as Root.
- In my scripts, instead of parsing the top.architecture directly, I will have to load all the references.

This is completely acceptable, I don't see any limitation up to now...

Regards,
Remi
Re: [EMF] long loading time [message #1404114 is a reply to message #1404086] Tue, 29 July 2014 17:09 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 29/07/2014 4:35 PM, Remi ADAM wrote:
> Ed,
>
> Good idea, I will try 8)
>
> I have implemented the cross resource containment, it works.
> In my new top.architecture, I add one folder (BlockX_virtual) for each
> linked block and use right clic to choose "Control..." and I choose
> the expected BlockX.architecture.
> This has 2 consequences :
> - In the original BlockX.architecture, it adds a node BlockX_virtual
> at the same level as Root. - In my scripts, instead of parsing the
> top.architecture directly, I will have to load all the references.
Hmmm. I don't think I completely follow. What I'd expect is that you
can have some resource top.architecture that contains a root object x.
It would have proxy resolving containing references so some other
resource y.architecture could contain an object y and then x can cross
resource contain y. So each resource only has one root object. If you
load top.architecture and use something like resource.getAllContents(),
you would visit all objects, including the ones in all other resources...
>
> This is completely acceptable, I don't see any limitation up to now...
>
> Regards,
> Remi


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404173 is a reply to message #1404114] Wed, 30 July 2014 08:54 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Ed,

Below is the TOP.architecture view.
index.php/fa/18698/0/
I have created one node for each controlled resouce. DIAG_virtual allows to load DIAG.architecture.
The text view of TOP.archi looks like that:
<?xml version="1.0" encoding="UTF-8"?>
<architecture:Root xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:architecture="http://architecture/1.0" xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzDIAG.architecture#_zFWIIRcfEeSvR-VX0bDXYw"/>
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCDD.architecture#_gfBpEBcmEeSvR-VX0bDXYw"/>
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCORE.architecture#_jvCoUBcmEeSvR-VX0bDXYw"/>
</architecture:Root>

Is it ok ?
(on the DIAG view, we can see that DIAG_virtual has been automatically added in // of Root.
index.php/fa/18699/0/)

Now, I had a script running on the former TOP.archi. It uses the following command to list all Modules (in my example, DEM and DET are modules).
List<EObject> referenceList = ModelUtil.getInstancesOfType(root, ArchitecturePackage.eINSTANCE.getwModule(), true);

With this new TOP.archi, DEM and DET are not seen by this command. So I have to find another solution to parse the TOP.archi.

Remi
Re: [EMF] long loading time [message #1404187 is a reply to message #1404173] Wed, 30 July 2014 10:02 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 30/07/2014 10:54 AM, Remi ADAM wrote:
> Ed,
>
> Below is the TOP.architecture view.
>
> I have created one node for each controlled resouce. DIAG_virtual allows to load DIAG.architecture.
> The text view of TOP.archi looks like that:
> <?xml version="1.0" encoding="UTF-8"?>
> <architecture:Root xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:architecture="http://architecture/1.0" xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
> <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzDIAG.architecture#_zFWIIRcfEeSvR-VX0bDXYw"/>
> <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCDD.architecture#_gfBpEBcmEeSvR-VX0bDXYw"/>
> <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCORE.architecture#_jvCoUBcmEeSvR-VX0bDXYw"/>
> </architecture:Root>
> Is it ok ?
Yes, so there's a single root object and it cross-resource contains
three other objects.
> (on the DIAG view, we can see that DIAG_virtual has been automatically added in // of Root.
> )
So you're using GMF and it's storing the diagram and the model in the
same resource. It would seem maybe better to store the diagram in a
separate resource. There's some way to configure that, but I'm not so
familiar with GMF's details.
>
> Now, I had a script running on the former TOP.archi. It uses the following command to list all Modules (in my example, DEM and DET are modules).
> List<EObject> referenceList = ModelUtil.getInstancesOfType(root, ArchitecturePackage.eINSTANCE.getwModule(), true);
Unless your model is very big, there's probably a
ArchitecturePackage.Literals.WMODULE constant that's slightly more
efficient than the method call...
>
> With this new TOP.archi, DEM and DET are not seen by this command. So I have to find another solution to parse the TOP.archi.
I don't really follow, but I don't really need to. :-P
>
> Remi


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404218 is a reply to message #1404187] Wed, 30 July 2014 14:03 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Ed,

I don't use GMF, so I don't know why diagram and model are in the same resource, but having this new node at the same level as Root is not disturbing.

On the Top.archi, when I right clic on Root, I can launch a command to parse the model.
Before having the cross reference implemented, the top.archi looked like this:
<?xml version="1.0" encoding="UTF-8"?>
<architecture:Root xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:architecture="http://architecture/1.0" xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
    <modules xmi:id="_61IaoAwkEeSEiMOfc07hWQ" shortName="CDD" longName="CDD Stack">
      <moduleChildren xmi:id="_61IaoQwkEeSEiMOfc07hWQ" shortName="BOSS" longName="Bulb Optimized Smart Switch"/>
      <moduleChildren xmi:id="_61JrFQwkEeSEiMOfc07hWQ" shortName="SBC" longName="SBC Driver"/>
    </modules>
    <modules xmi:id="_lhQX4AwlEeSEiMOfc07hWQ" shortName="CORE" longName="CORE Stack">
      <moduleChildren xmi:id="_lhRAsAwlEeSEiMOfc07hWQ" shortName="EVTMAN" longName="Event Manager"/>
      <moduleChildren xmi:id="_lhRmwAwlEeSEiMOfc07hWQ" shortName="EVTMAN_FG2" longName="Event Manager FG2"/>
    </modules>
    <modules xmi:id="_mD38wAwlEeSEiMOfc07hWQ" shortName="DIAG" longName="DIAG Stack">
      <moduleChildren xmi:id="_mD38wQwlEeSEiMOfc07hWQ" shortName="DET" longName="Development Error Tracer"/>
      <moduleChildren xmi:id="_mD4kCAwlEeSEiMOfc07hWQ" shortName="DEM" longName="Diagnostic Event Manager"/>
    </modules>
</architecture:Root>


I was able to output module list : CDD, BOSS, SBC, CORE, EVTMAN, EVTMAN_FG2, DIAG, DET, DEM by using the code:
Root root = (Root) strSelect.getFirstElement();
List<EObject> referenceList = ModelUtil.getInstancesOfType(root, ArchitecturePackage.Literals.MODULE, true);


Now the Top.archi look like this :
<?xml version="1.0" encoding="UTF-8"?>
<architecture:Root xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:architecture="http://architecture/1.0" xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCDD.architecture#_-VZgABfjEeSjCfp3BWJx-Q"/>
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCORE.architecture#_Rv_3gBfkEeSjCfp3BWJx-Q"/>
  <modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzDIAG.architecture#_rfFvEBfoEeSjCfp3BWJx-Q"/>
</architecture:Root>

With this, no module is seen, that is why I say I have to find another way to parse de Top.archi.

Do I have to load the 3 href with a specific command?
Or do I have to use another command to parse the referenced .architecture?

Thanks,
Remi

Re: [EMF] long loading time [message #1404229 is a reply to message #1404218] Wed, 30 July 2014 14:50 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 30/07/2014 4:03 PM, Remi ADAM wrote:
> Ed,
>
> I don't use GMF, so I don't know why diagram and model are in the same
> resource, but having this new node at the same level as Root is not
> disturbing.
I just don't follow the terminology of DIAG....
>
> On the Top.archi, when I right clic on Root, I can launch a command to
> parse the model.
> Before having the cross reference implemented, the top.archi looked
> like this:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <architecture:Root xmi:version="2.0"
> xmlns:xmi="http://www.omg.org/XMI"
> xmlns:architecture="http://architecture/1.0"
> xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
> <modules xmi:id="_61IaoAwkEeSEiMOfc07hWQ" shortName="CDD"
> longName="CDD Stack">
> <moduleChildren xmi:id="_61IaoQwkEeSEiMOfc07hWQ" shortName="BOSS"
> longName="Bulb Optimized Smart Switch"/>
> <moduleChildren xmi:id="_61JrFQwkEeSEiMOfc07hWQ" shortName="SBC"
> longName="SBC Driver"/>
> </modules>
> <modules xmi:id="_lhQX4AwlEeSEiMOfc07hWQ" shortName="CORE"
> longName="CORE Stack">
> <moduleChildren xmi:id="_lhRAsAwlEeSEiMOfc07hWQ"
> shortName="EVTMAN" longName="Event Manager"/>
> <moduleChildren xmi:id="_lhRmwAwlEeSEiMOfc07hWQ"
> shortName="EVTMAN_FG2" longName="Event Manager FG2"/>
> </modules>
> <modules xmi:id="_mD38wAwlEeSEiMOfc07hWQ" shortName="DIAG"
> longName="DIAG Stack">
> <moduleChildren xmi:id="_mD38wQwlEeSEiMOfc07hWQ" shortName="DET"
> longName="Development Error Tracer"/>
> <moduleChildren xmi:id="_mD4kCAwlEeSEiMOfc07hWQ" shortName="DEM"
> longName="Diagnostic Event Manager"/>
> </modules>
> </architecture:Root>
>
>
> I was able to output module list : CDD, BOSS, SBC, CORE, EVTMAN,
> EVTMAN_FG2, DIAG, DET, DEM by using the code:
>
> Root root = (Root) strSelect.getFirstElement();
> List<EObject> referenceList = ModelUtil.getInstancesOfType(root,
> ArchitecturePackage.Literals.MODULE, true);
>
>
> Now the Top.archi look like this :
>
> <?xml version="1.0" encoding="UTF-8"?>
> <architecture:Root xmi:version="2.0"
> xmlns:xmi="http://www.omg.org/XMI"
> xmlns:architecture="http://architecture/1.0"
> xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
> <modules
> href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCDD.architecture#_-VZgABfjEeSjCfp3BWJx-Q"/>
> <modules
> href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzCORE.architecture#_Rv_3gBfkEeSjCfp3BWJx-Q"/>
> <modules
> href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzDIAG.architecture#_rfFvEBfoEeSjCfp3BWJx-Q"/>
> </architecture:Root>
>
> With this, no module is seen, that is why I say I have to find another
> way to parse de Top.archi.
They're children of root. I.e., root.getModules() will visit them.
>
> Do I have to load the 3 href with a specific command?
No, that will be done automatically when you iterate over getModules().
> Or do I have to use another command to parse the referenced
> .architecture?
No.
>
> Thanks,
> Remi
>
>


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404236 is a reply to message #1404229] Wed, 30 July 2014 15:18 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
DIAG is for Diagnosys, that the name of a module container. No link with Diagram.

You are right, when I say "no module is seen", I wanted to say "none of the expected module is seen".
The output of the root.getModules() is
[architecture.impl.BswModuleImpl@18144fa (locked: false, shortName: CDD_link, longName: null, description: , createdFromGateway: false) (autosarModuleId: 0, gatewayTypes: null, gatewayTypesChangeReason: null), architecture.impl.BswModuleImpl@1ece4b (locked: false, shortName: CORE_link, longName: null, description: , createdFromGateway: false) (autosarModuleId: 0, gatewayTypes: null, gatewayTypesChangeReason: null), architecture.impl.BswModuleImpl@9ca3e0 (locked: false, shortName: DIAG_link, longName: null, description: , createdFromGateway: false) (autosarModuleId: 0, gatewayTypes: null, gatewayTypesChangeReason: null)]

But I can not parse the references linked to these fake nodes.
Remi
Re: [EMF] long loading time [message #1404241 is a reply to message #1404236] Wed, 30 July 2014 15:26 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
I have seen that the reference to DIAG is:
<modules href="file:/D:/WORKSPACE/ws_ARCHI_MODEL/zzDEBUG/zzDIAG.architecture#_rfFvEBfoEeSjCfp3BWJx-Q"/>

This ID is the one from the fake node (last line) and not from the referenced root.
Is that normal ?

<?xml version="1.0" encoding="UTF-8"?>
<xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:architecture="http://architecture/1.0">
  <architecture:Root xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
      <modules xmi:id="_mD38wAwlEeSEiMOfc07hWQ" shortName="DIAG" longName="DIAG Stack">
      <moduleChildren xmi:id="_mD38wQwlEeSEiMOfc07hWQ" shortName="DET" longName="Development Error Tracer"/>
      <moduleChildren xmi:id="_mD4kCAwlEeSEiMOfc07hWQ" shortName="DEM" longName="Diagnostic Event Manager"/>
	</modules>
  </architecture:Root>
  <architecture:BswModule xmi:id="_rfFvEBfoEeSjCfp3BWJx-Q" shortName="DIAG_link"/>
</xmi:XMI>

Re: [EMF] long loading time [message #1404243 is a reply to message #1404236] Wed, 30 July 2014 15:25 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 30/07/2014 5:18 PM, Remi ADAM wrote:
> DIAG is for Diagnosys, that the name of a module container. No link
> with Diagram.
>
> You are right, when I say "no module is seen", I wanted to say "none
> of the expected module is seen".
> The output of the root.getModules() is
> [architecture.impl.BswModuleImpl@18144fa (locked: false, shortName:
> CDD_link, longName: null, description: , createdFromGateway: false)
> (autosarModuleId: 0, gatewayTypes: null, gatewayTypesChangeReason:
> null), architecture.impl.BswModuleImpl@1ece4b (locked: false,
> shortName: CORE_link, longName: null, description: ,
> createdFromGateway: false) (autosarModuleId: 0, gatewayTypes: null,
> gatewayTypesChangeReason: null),
> architecture.impl.BswModuleImpl@9ca3e0 (locked: false, shortName:
> DIAG_link, longName: null, description: , createdFromGateway: false)
> (autosarModuleId: 0, gatewayTypes: null, gatewayTypesChangeReason: null)]
>
> But I can not parse the references linked to these fake nodes.
What do you mean by "fake nodes"? These look like the properly resolved
objects. These are your modules. They got there by loading the
references resource (parsing it), and linking that resource's root model
object to your "root" in the first resource.
> Remi
>


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404247 is a reply to message #1404243] Wed, 30 July 2014 15:46 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
What I call "fake nodes" are the nodes I had to create (CDD_link, CORE_link and DIAG_link) to be able to control the reference.

Here the reference is mapped on ORANGE, but I would need it to be mapped on GREEN or LIME GREEN.

<architecture:Root xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
<modules xmi:id="_mD38wAwlEeSEiMOfc07hWQ" shortName="DIAG" longName="DIAG Stack">
<moduleChildren xmi:id="_mD38wQwlEeSEiMOfc07hWQ" shortName="DET" longName="Development Error Tracer"/>
<moduleChildren xmi:id="_mD4kCAwlEeSEiMOfc07hWQ" shortName="DEM" longName="Diagnostic Event Manager"/>
</modules>
</architecture:Root>
<architecture:BswModule xmi:id="_rfFvEBfoEeSjCfp3BWJx-Q" shortName="DIAG_link"/>

In another way, getModules is able to catch DIAG_link, is there a way to catch DIAG, DET and DEM ?
Re: [EMF] long loading time [message #1404295 is a reply to message #1404247] Thu, 31 July 2014 04:54 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 30/07/2014 5:46 PM, Remi ADAM wrote:
> What I call "fake nodes" are the nodes I had to create (CDD_link,
> CORE_link and DIAG_link) to be able to control the reference.
Given I don't know your model it's very hard to understand what you mean
or to give advice...
>
> Here the reference is mapped on ORANGE, but I would need it to be
> mapped on GREEN or LIME GREEN.
>
> <architecture:Root xmi:id="_gX9ugGcAEeOnxpO1IkhpXA">
> <modules xmi:id="_mD38wAwlEeSEiMOfc07hWQ" shortName="DIAG"
> longName="DIAG Stack">
> <moduleChildren xmi:id="_mD38wQwlEeSEiMOfc07hWQ" shortName="DET"
> longName="Development Error Tracer"/>
> <moduleChildren xmi:id="_mD4kCAwlEeSEiMOfc07hWQ" shortName="DEM"
> longName="Diagnostic Event Manager"/>
> </modules>
> </architecture:Root>
> <architecture:BswModule xmi:id="_rfFvEBfoEeSjCfp3BWJx-Q"
> shortName="DIAG_link"/>
>
> In another way, getModules is able to catch DIAG_link, is there a way
> to catch DIAG, DET and DEM ?
I see no cross document references here. Of course you can always
define a non-containment EReference that can link to whatever type of
object is allowed by its type.


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404329 is a reply to message #1404295] Thu, 31 July 2014 09:29 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Hello Ed and thanks for your effort !

I'm going to try to give you the overview to understand my need.

I have 3 separated modules, based on the same ecore model.
index.php/fa/18717/0/
I need to parse all these modules in a same time to be able to catch the interactions btw modules (for example sub-module DET declares a function F1, sub-module BOSS calls a F1 function, but currently not linked to DET F1). By parsing the 3 modules, I will be able to fill BOSS/F1 parameters with "declaredIn=DIAG/DET").

Up to now, the only way I found to do this was to create a TOP.archi and copy/past the modules inside:
index.php/fa/18718/0/
It was functional, but I needed to copy back the updated information in the standalone module (because I want to keep this splitted modules). That's why I wanted to find a way to create a TOP.archi with link to the modules to be able to perform the same action but avoiding the copy.paste actions.

To do this I have:
- created a TOP.archi,
- added a "node" (folder of Module type : XXX_link below) for each Module,
- linked the new node by right clic "Control..." and
- chosen the right target to control.
index.php/fa/18719/0/

Then a XXX_link node is created in each standalone MODULE.archi:
index.php/fa/18720/0/

Thanks to this, I have a link btw my TOP.archi and my 3 modules.
Now the link is on the DIAG_link and not the Root or DIAG of DIAG.archi.
In the former TOP.archi, I was able to catch CDD, BOSS, SBC, CORE, EVTMAN, EVTMAN_FG2, DIAG, DET, DEM. With the new way, I only catch CDD_link, CORE_link and DIAG_link.
I hope it clarifies...
Is there a way to have this linked version of TOP.archi but keeping the capability to parse the standard modules?
Remi

[Updated on: Thu, 31 July 2014 09:30]

Report message to a moderator

Re: [EMF] long loading time [message #1404397 is a reply to message #1404329] Fri, 01 August 2014 04:36 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
Remi,

Comments below.

On 31/07/2014 11:29 AM, Remi ADAM wrote:
> Hello Ed and thanks for your effort !
>
> I'm going to try to give you the overview to understand my need.
This is turning into free consulting...
>
> I have 3 separated modules, based on the same ecore model.
>
> I need to parse all these modules in a same time to be able to catch the interactions btw modules (for example sub-module DET declares a function F1, sub-module BOSS calls a F1 function, but currently not linked to DET F1). By parsing the 3 modules, I will be able to fill BOSS/F1 parameters with "declaredIn=DIAG/DET").
In EMF, parsing means loading resources, typically into a resource set
so that their references can resolve...
>
> Up to now, the only way I found to do this was to create a TOP.archi and copy/past the modules inside:
Yes, the default behavior for adding an existing object that's already
in its own resource as a child of a cross resource containment isn't
very direct. The workflow is more focused on moving a contained object
to a new resource (the Control...) action.
>
> It was functional, but I needed to copy back the updated information in the standalone module (because I want to keep this splitted modules).
In Oomph we specialized drag and drop behavior so that for a proxy
resolving containment reference you can drag (with the link modifier
keys) to drag a resource's root object to an object that can cross
contain it.

http://git.eclipse.org/c/oomph/org.eclipse.oomph.git/tree/plugins/org.eclipse.oomph.base.edit/src/org/eclipse/oomph/base/provider/ModelElementItemProvider.java


> That's why I wanted to find a way to create a TOP.archi with link to the modules to be able to perform the same action but avoiding the copy.paste actions.
>
> To do this I have:
> - created a TOP.archi,
> - added a "node" (folder of Module type : XXX_link below) for each Module,
> - linked the new node by right clic "Control..." and
> - chosen the right target to control.
>
> Then a XXX_link node is created in each standalone MODULE.archi:
>
> Thanks to this, I have a link btw my TOP.archi and my 3 modules.
> Now the link is on the DIAG_link and not the Root or DIAG of DIAG.archi.
> In the former TOP.archi, I was able to catch CDD, BOSS, SBC, CORE, EVTMAN, EVTMAN_FG2, DIAG, DET, DEM. With the new way, I only catch CDD_link, CORE_link and DIAG_link.
> I hope it clarifies...
You can only cross contain the root object of some other resource. Any
other linking you'll have to do with a non-containment reference.
> Is there a way to have this linked version of TOP.archi but keeping the capability to parse the standard modules?
> Remi


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1404440 is a reply to message #1404397] Fri, 01 August 2014 12:14 Go to previous messageGo to next message
Remi ADAM is currently offline Remi ADAMFriend
Messages: 14
Registered: July 2014
Junior Member
Quote:
This is turning into free consulting...

You are right, I will have to deeply investigate on my side, but for the time being, my mind will be focused on holidays Cool.
Remi
Re: [EMF] long loading time [message #1707417 is a reply to message #1404440] Fri, 04 September 2015 15:05 Go to previous messageGo to next message
Ameni BF is currently offline Ameni BFFriend
Messages: 8
Registered: November 2014
Junior Member
Hello,

I have the same problem, loading an xmi file is taking a lot of time (1hour). I tried to modify the load options but it did not solve the problem.
The code is below:


static EObject loadXMI(String instancePath, EmfModelReader reader)
{

//load model instance
ResourceSetImpl load_resourceSet = new ResourceSetImpl();
load_resourceSet.getLoadOptions().put(XMIResource.OPTION_DEFER_IDREF_RESOLUTION, Boolean.TRUE);
load_resourceSet.getLoadOptions().put(XMIResource.OPTION_DEFER_ATTACHMENT, Boolean.TRUE);
load_resourceSet.getLoadOptions().put(XMIResource.OPTION_USE_DEPRECATED_METHODS, Boolean.TRUE);
load_resourceSet.getLoadOptions().put(XMIResource.OPTION_USE_PARSER_POOL, new XMLParserPoolImpl());
load_resourceSet.getLoadOptions().put(XMIResource.OPTION_USE_XML_NAME_TO_FEATURE_MAP, new HashMap());
load_resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("xmi", new XMIResourceFactoryImpl());
// reader.getPackages().get(0): returns the EPackage of the core model
load_resourceSet.getPackageRegistry().put(reader.getPackages().get(0).getNsURI(),reader.getPackages().get(0));
Resource load_resource = load_resourceSet.getResource(URI.createURI(instancePath),true);
EList<EObject> elements= load_resource.getContents();
return elements.get(0);
}
Re: [EMF] long loading time [message #1824157 is a reply to message #1403905] Wed, 08 April 2020 19:08 Go to previous messageGo to next message
Christof Tinnes is currently offline Christof TinnesFriend
Messages: 2
Registered: April 2020
Junior Member
Hey,

I know it has been a long time but I stumbled upon this thread and the defer id resolution option actually solved my problem.

Now, I am wondering about the O(n^2) complexity without the defer option. Naively, I would assume, that the algorithm iterated over all nodes and resolves every reference. Of course, worst case would be O(n^2), but in my case there are 4 references on average. So I would assume it should be something like O(4*n). But actually, I experienced a reduction by a factor of 120.

Could someone point me to a description of the algorithm, to understand why it so slow? (I don't want to go through all the code ;-))

Thanks,

Christof
Re: [EMF] long loading time [message #1824174 is a reply to message #1824157] Thu, 09 April 2020 03:38 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
When and ID is resolved, org.eclipse.emf.ecore.resource.impl.ResourceImpl.getEObjectByID(String) is called (or some override of it). In the worst case, this will visit the entire tree (as it exists so far if this happens during loading). So if each EObject in the tree had 1 forward reference to the last EObject in the tree, you would visit the O(n) EObjects in the tree O(n) times. Without an non-null org.eclipse.emf.ecore.resource.impl.ResourceImpl.getIntrinsicIDToEObjectMap() or without an idToEObjectObject in org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.getEObjectByID(String), even the deferred cost is that high. But with a map for intrinsic IDs (modeled as EAttributes with isID true) the ID resolution is a map lookup O(1), with a one time cost to build the map for intrinsic IDs; for extrinsic IDs, the map is built during loading.

A poor-man's performance analyzer is to stop the running processes repeatedly and looking at where the code currently running, i.e., like sampling in a profiler.


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMF] long loading time [message #1824187 is a reply to message #1824174] Thu, 09 April 2020 08:30 Go to previous messageGo to next message
Christof Tinnes is currently offline Christof TinnesFriend
Messages: 2
Registered: April 2020
Junior Member
Thanks for the quick reply. Actually figured it out yesterday, but was not able to withdraw my pending post anymore =/

I somehow thought that a HashMap will be used internally anyway, which obviously is not the case for default settings. As far as I understood from what you wrote above, the default variant is there because of the memory<->performance trade-off.
But since all objects are loaded anyway when the resource is loaded (not talking about cross-resource references now), even for huge objects, the map should not consume to much memory. E.g., it would be approx. 36 bytes per object on a 32-bit JVM (or am I missing here something?).
On the other hand, performance impact of the default approach seems to be huge for large models.
So I am wondering now, why the default is not the defered map lookup for id resolution or is it just for historical reasons?

[Updated on: Thu, 09 April 2020 08:41]

Report message to a moderator

Re: [EMF] long loading time [message #1824192 is a reply to message #1824187] Thu, 09 April 2020 10:20 Go to previous message
Ed Merks is currently offline Ed MerksFriend
Messages: 33264
Registered: July 2009
Senior Member
EMF has a very long history and we strive to make changes that are minimally disruptive including subtle changes in behavior. Mostly everything in EMF is public or protected, so even changes that affect calls to protected methods, e.g, when they are called, can be potentially disruptive. Often this means that new better behavior is driven by an option, while the older behavior remains the default.

Ed Merks
Professional Support: https://www.macromodeling.com/
Previous Topic:XSD to Ecore: Group renamed to Group1
Next Topic:How to 'force' loading of uml resource with incorrect reference
Goto Forum:
  


Current Time: Sat Dec 14 07:25:08 GMT 2024

Powered by FUDForum. Page generated in 0.05759 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top