Home » Modeling » EMF » segmented models and lazy loading
| segmented models and lazy loading [message #502800] |
Wed, 09 December 2009 18:01  |
 gary s thompson
Messages: 92
Registered: July 2009 |
Member |
|
|
I have recently been subdividing a large model (433 classes in 92 packages) from a single monolithic ecore resource into a resource per package setup. After this Odyssey a couple of questions crop up.
The first is about the use of basePackage. base package allowed me to set the parent package hierachy of each class which was neat as I didn't use nested subpackages any more. However, I had to go round and apply base package 98 times! (I wrote a little python script which modifies the genmodel xml file) so here is my question is there a better way? [I think I asked about a basePackage annotation on the ecore model many moons ago would that be an idea?] [in passing is there any way to get code to run before and after model generation within the eclipse gui (obviously i could use an ant script but at development time thats so much more clunky) as it would be nice to run my base package setter before each source code generation cycle]
ok so the second question was about resource loading for ecore classes. i now have a nicely subdivided model. However, you might assume naively that the individual resources containing the ecore metamodel would be lazily loaded but in my hands this doesn't seem to be the case. Thus my model load time is quite long (10s) as the whole set of ecore model classes appears to get loaded into memory at the same time. Is there a way to avoid this? [I should add that whole model is rooted but that i wouldn't expect all packages to be loaded at any one session] I should add further that I am not loading the packages/resources via extension points currently I am just using a standalone application and load the packages by instantiating th package which is at the root of the containment hierarchy (one of the classes in this package acts as a a container for all other containment hierarchies)
any help gratefully recieved!
regards
gary
[Updated on: Wed, 09 December 2009 18:15] Report message to a moderator |
|
| | | |
| Re: segmented models and lazy loading [message #504056 is a reply to message #503984] |
Wed, 16 December 2009 23:08  |
gary s thompson
Messages: 92
Registered: July 2009 |
Member |
|
|
hi Ed
| Ed Merks wrote on Wed, 16 December 2009 12:16 | Gary,
gary s thompson wrote:
> ok ed here goes for hopefully enough detail
>
> the framework is organised as a project class plus a pendant series of
> object trees of different types with each type of the pendant object
> trees and acting as a containment hierachy and exising in its own
> package and being stored as a resource [there maybe 0..n instances for
> each of the pendant trees, so in anyone project many of them could be
> empty].
There's of course no relationship between how classes are divided into
packages and how instances are divided into resources...
|
this is true but in the case of the model i am trying reimplement there is 1:1 relationship between resources and packages and i don't want to break it it will just cause too much more work.
| Quote: |
>
> the projects has a bi directional link to each of the child trees
> (which from my further reading of the emf book seems to be the
> problem) thus while the ecore model can be split on a epackage basis
> the genmodel can't because everything depends on the root and the root
> depends on everything else... (is there any way round this without
> losing the root object as I can't see it... I should note that the
> resource and root object for each of the child trees is individually
> identifiable within the resource set without requiring a containment
> link)
Yes, a bidirectional reference is very intimate and things related in
this way can't be in separate GenModels.
>
> so when I load the root package all the pendant packages get loaded (I
> don't even think they are proxies which suprised me as I would have
> thought they would be)
> here is the relevant line in GenPackage DependencyHelper.DependencyHelper
>
> interDependencies = new ArrayList<GenPackage>();
> collectPackages(interDependencies,
> genPackage.getGenModel().getGenPackages(), -1);
> interDependencies.remove(genPackage);
> addAll(interDependencies);
>
>
> which causes as series of statements of the form
> YYYYPackageImpl theYYYYPackage =
> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY
> YPackage.eNS_URI) instanceof YYYYPackageImpl ?
> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>
> to be generated for each package interdependency
>
> with YYYYPackage.eINSTANCE effectively resulting a statement of the form
>
> YYYYPackage eINSTANCE =
> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>
> to be executed in the package YYYY
>
> which finally leeds to again in package YYYY
>
> if (isInited) return
> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack
> age.eNS_URI);
>
> // Obtain or create and register package
> YYYYPackageImpl theYYYYPackage =
> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI)
> instanceof YYYYPackageImpl ?
> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new YYYYPackageImpl());
>
> isInited = true;
>
> which instansiates the YYYY package if it hasn't been done so already....
And yes, the packages must all be initialized to set up the
bidirectional references. There's no proxy resolution in generated
packages...
|
that confirms what i surmised after a bit of digging...
| Quote: |
>
>
> so by my understanding all packages within the same genmodel are
> interdependant so the real trick here is to split the genmodel...
> (however this seems hard as I guess any cross reference to another
> package even by operation or non containment reference will prevent me
> splitting it because it will introduce an implicit bidrectional link...)
Bidirectional references can't be split across GenModels but any other
references can.
|
I guess this include containment references which are implicitly bidirectional
| Quote: |
>
> As a side note in the test I did I just loaded root package and didn't
> instantiate anything from the child packages. So all I did was
> RootPackage rootPackage = RootPackage.eINSTANCE;
>
>
> again all or any help gratefully recieved
It's just how it works. Sorry.
|
thats fine [I do science and boy do I know that I can't beat nature...]
On an aside I did have one slightly insane idea that might improve things. So here we go, please don't hesitate to shoot me if I have missed the obvious flaw, [apart from the fact that it might be a lot of work].
So my problem isn't that all the models in the genmodel get loaded but that it takes a long time to do it (about 4 seconds). So the idea was to load all the packageImpls together as currently so that loading one packImpl via its init method resisters all the other ones in the package registry at the same time. However the twist is to then to delay the loading of the rest of the meta data in the packages until first use for each package using the Initialization on demand holder idiom [ http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom] to materialise the packages metadata as it is required so giving a lazy loading scheme for the metadat within the genmodel
regards gary
|
|
|
| Re: segmented models and lazy loading [message #504135 is a reply to message #504056] |
Thu, 17 December 2009 07:00 |
Ed Merks
Messages: 33136
Registered: July 2009 |
Senior Member |
|
|
Gary,
Comments below.
gary s thompson wrote:
> hi Ed
> Ed Merks wrote on Wed, 16 December 2009 12:16
>> Gary,
>>
>>
>>
>> gary s thompson wrote:
>> > ok ed here goes for hopefully enough detail
>> >
>> > the framework is organised as a project class plus a pendant series
>> of > object trees of different types with each type of the pendant
>> object > trees and acting as a containment hierachy and exising in
>> its own > package and being stored as a resource [there maybe 0..n
>> instances for > each of the pendant trees, so in anyone project many
>> of them could be > empty].
>> There's of course no relationship between how classes are divided
>> into packages and how instances are divided into resources...
>
> this is true but in the case of the model i am trying reimplement
> there is 1:1 relationship between resources and packages and i don't
> want to break it it will just cause too much more work.
> Quote:
>> >
>> > the projects has a bi directional link to each of the child trees >
>> (which from my further reading of the emf book seems to be the >
>> problem) thus while the ecore model can be split on a epackage basis
>> > the genmodel can't because everything depends on the root and the
>> root > depends on everything else... (is there any way round this
>> without > losing the root object as I can't see it... I should note
>> that the > resource and root object for each of the child trees is
>> individually > identifiable within the resource set without requiring
>> a containment > link)
>> Yes, a bidirectional reference is very intimate and things related in
>> this way can't be in separate GenModels.
>> >
>> > so when I load the root package all the pendant packages get loaded
>> (I > don't even think they are proxies which suprised me as I would
>> have > thought they would be)
>> > here is the relevant line in GenPackage
>> DependencyHelper.DependencyHelper
>> >
>> > interDependencies = new ArrayList<GenPackage>();
>> > collectPackages(interDependencies, >
>> genPackage.getGenModel().getGenPackages(), -1);
>> > interDependencies.remove(genPackage);
>> > addAll(interDependencies);
>> >
>> >
>> > which causes as series of statements of the form
>> > YYYYPackageImpl theYYYYPackage = >
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY >
>> YPackage.eNS_URI) instanceof YYYYPackageImpl ? >
>> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>> >
>> > to be generated for each package interdependency
>> >
>> > with YYYYPackage.eINSTANCE effectively resulting a statement of the
>> form
>> >
>> > YYYYPackage eINSTANCE = >
>> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>> >
>> > to be executed in the package YYYY
>> >
>> > which finally leeds to again in package YYYY
>> >
>> > if (isInited) return >
>> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack >
>> age.eNS_URI);
>> >
>> > // Obtain or create and register package
>> > YYYYPackageImpl theYYYYPackage = >
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI) >
>> instanceof YYYYPackageImpl ? >
>> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new
>> YYYYPackageImpl());
>> >
>> > isInited = true;
>> >
>> > which instansiates the YYYY package if it hasn't been done so
>> already....
>> And yes, the packages must all be initialized to set up the
>> bidirectional references. There's no proxy resolution in generated
>> packages...
>
> that confirms what i surmised after a bit of digging...
> Quote:
>> >
>> >
>> > so by my understanding all packages within the same genmodel are >
>> interdependant so the real trick here is to split the genmodel... >
>> (however this seems hard as I guess any cross reference to another >
>> package even by operation or non containment reference will prevent
>> me > splitting it because it will introduce an implicit bidrectional
>> link...)
>> Bidirectional references can't be split across GenModels but any
>> other references can.
>
> I guess this include containment references which are implicitly
> bidirectional Quote:
No. Only explicitly bidirectional references create a two-way (mutual)
dependency.
>> >
>> > As a side note in the test I did I just loaded root package and
>> didn't > instantiate anything from the child packages. So all I did was
>> > RootPackage rootPackage = RootPackage.eINSTANCE;
>> >
>> >
>> > again all or any help gratefully recieved
>> It's just how it works. Sorry.
>
> thats fine [I do science and boy do I know that I can't beat nature...]
> On an aside I did have one slightly insane idea that might improve
> things. So here we go, please don't hesitate to shoot me if I have
> missed the obvious flaw, [apart from the fact that it might be a lot
> of work].
>
> So my problem isn't that all the models in the genmodel get loaded but
> that it takes a long time to do it (about 4 seconds).
Not even enough time to have a cup of coffee. Are the packages really
big? Are you sure the first one doesn't take 3.95 seconds because of
other initializations and that the other three take no appreciable
amount of time?
> So the idea was to load all the packageImpls together as currently so
> that loading one packImpl via its init method resisters all the other
> ones in the package registry at the same time. However the twist is to
> then to delay the loading of the rest of the meta data in the packages
> until first use for each package using the Initialization on demand
> holder idiom
> [ http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom]
> to materialise the packages metadata as it is required so giving a
> lazy loading scheme for the metadat within the genmodel
And who's going to make that happen? :-P
>
>
>
> Quote:
>> >
>> >
>
> regards gary
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: segmented models and lazy loading [message #504161 is a reply to message #504135] |
Thu, 17 December 2009 14:14 |
gary s thompson
Messages: 92
Registered: July 2009 |
Member |
|
|
| Ed Merks wrote on Thu, 17 December 2009 02:00 | Gary,
Comments below.
gary s thompson wrote:
> hi Ed
> Ed Merks wrote on Wed, 16 December 2009 12:16
>> Gary,
>>
>>
>>
>> gary s thompson wrote:
>> > ok ed here goes for hopefully enough detail
>> >
>> > the framework is organised as a project class plus a pendant series
>> of > object trees of different types with each type of the pendant
>> object > trees and acting as a containment hierachy and exising in
>> its own > package and being stored as a resource [there maybe 0..n
>> instances for > each of the pendant trees, so in anyone project many
>> of them could be > empty].
>> There's of course no relationship between how classes are divided
>> into packages and how instances are divided into resources...
>
> this is true but in the case of the model i am trying reimplement
> there is 1:1 relationship between resources and packages and i don't
> want to break it it will just cause too much more work.
> Quote:
>> >
>> > the projects has a bi directional link to each of the child trees >
>> (which from my further reading of the emf book seems to be the >
>> problem) thus while the ecore model can be split on a epackage basis
>> > the genmodel can't because everything depends on the root and the
>> root > depends on everything else... (is there any way round this
>> without > losing the root object as I can't see it... I should note
>> that the > resource and root object for each of the child trees is
>> individually > identifiable within the resource set without requiring
>> a containment > link)
>> Yes, a bidirectional reference is very intimate and things related in
>> this way can't be in separate GenModels.
>> >
>> > so when I load the root package all the pendant packages get loaded
>> (I > don't even think they are proxies which suprised me as I would
>> have > thought they would be)
>> > here is the relevant line in GenPackage
>> DependencyHelper.DependencyHelper
>> >
>> > interDependencies = new ArrayList<GenPackage>();
>> > collectPackages(interDependencies, >
>> genPackage.getGenModel().getGenPackages(), -1);
>> > interDependencies.remove(genPackage);
>> > addAll(interDependencies);
>> >
>> >
>> > which causes as series of statements of the form
>> > YYYYPackageImpl theYYYYPackage = >
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY >
>> YPackage.eNS_URI) instanceof YYYYPackageImpl ? >
>> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>> >
>> > to be generated for each package interdependency
>> >
>> > with YYYYPackage.eINSTANCE effectively resulting a statement of the
>> form
>> >
>> > YYYYPackage eINSTANCE = >
>> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>> >
>> > to be executed in the package YYYY
>> >
>> > which finally leeds to again in package YYYY
>> >
>> > if (isInited) return >
>> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack >
>> age.eNS_URI);
>> >
>> > // Obtain or create and register package
>> > YYYYPackageImpl theYYYYPackage = >
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI) >
>> instanceof YYYYPackageImpl ? >
>> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new
>> YYYYPackageImpl());
>> >
>> > isInited = true;
>> >
>> > which instansiates the YYYY package if it hasn't been done so
>> already....
>> And yes, the packages must all be initialized to set up the
>> bidirectional references. There's no proxy resolution in generated
>> packages...
>
> that confirms what i surmised after a bit of digging...
> Quote:
>> >
>> >
>> > so by my understanding all packages within the same genmodel are >
>> interdependant so the real trick here is to split the genmodel... >
>> (however this seems hard as I guess any cross reference to another >
>> package even by operation or non containment reference will prevent
>> me > splitting it because it will introduce an implicit bidrectional
>> link...)
>> Bidirectional references can't be split across GenModels but any
>> other references can.
>
> I guess this include containment references which are implicitly
> bidirectional Quote:
No. Only explicitly bidirectional references create a two-way (mutual)
dependency.
|
that may be my get out clause then. I will investigate that next
| Quote: |
>> >
>> > As a side note in the test I did I just loaded root package and
>> didn't > instantiate anything from the child packages. So all I did was
>> > RootPackage rootPackage = RootPackage.eINSTANCE;
>> >
>> >
>> > again all or any help gratefully recieved
>> It's just how it works. Sorry.
>
> thats fine [I do science and boy do I know that I can't beat nature...]
> On an aside I did have one slightly insane idea that might improve
> things. So here we go, please don't hesitate to shoot me if I have
> missed the obvious flaw, [apart from the fact that it might be a lot
> of work].
>
> So my problem isn't that all the models in the genmodel get loaded but
> that it takes a long time to do it (about 4 seconds).
Not even enough time to have a cup of coffee. Are the packages really
big? Are you sure the first one doesn't take 3.95 seconds because of
other initializations and that the other three take no appreciable
amount of time?
|
no I checked the root package is one of the top five but the packages range in size from 5Mb to 84k for the ecore the total size is 27Mb for all packages
| Quote: |
> So the idea was to load all the packageImpls together as currently so
> that loading one packImpl via its init method resisters all the other
> ones in the package registry at the same time. However the twist is to
> then to delay the loading of the rest of the meta data in the packages
> until first use for each package using the Initialization on demand
> holder idiom
> [ http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom]
> to materialise the packages metadata as it is required so giving a
> lazy loading scheme for the metadat within the genmodel
And who's going to make that happen? 
|
I was planning to have a try at making it work to see if it was good. I take it the actual idea isn't insane? I guess I would then have to download the emf develeopment sources and run the unit tests and submit it as a patch on a bugzilla and hope that in the end it would be accepted after modification reveiew etc.
| Quote: |
>
>
>
> Quote:
>> >
>> >
>
> regards gary
|
regards
gary
|
|
|
| Re: segmented models and lazy loading [message #504164 is a reply to message #504161] |
Thu, 17 December 2009 14:18 |
Ed Merks
Messages: 33136
Registered: July 2009 |
Senior Member |
|
|
Gary,
Before you work on changes you assume will improve performance, I'd
suggest carefully measure what's taking so long. You might well find,
as I suggested, that the first package takes a lot of time because
that's the point at which EMF's core runtime is initialized too.
gary s thompson wrote:
> Ed Merks wrote on Thu, 17 December 2009 02:00
>> Gary,
>>
>> Comments below.
>>
>> gary s thompson wrote:
>> > hi Ed
>> > Ed Merks wrote on Wed, 16 December 2009 12:16
>> >> Gary,
>> >>
>> >>
>> >>
>> >> gary s thompson wrote:
>> >> > ok ed here goes for hopefully enough detail
>> >> >
>> >> > the framework is organised as a project class plus a pendant
>> series >> of > object trees of different types with each type of the
>> pendant >> object > trees and acting as a containment hierachy and
>> exising in >> its own > package and being stored as a resource [there
>> maybe 0..n >> instances for > each of the pendant trees, so in anyone
>> project many >> of them could be > empty].
>> >> There's of course no relationship between how classes are divided
>> >> into packages and how instances are divided into resources...
>> >
>> > this is true but in the case of the model i am trying reimplement >
>> there is 1:1 relationship between resources and packages and i don't
>> > want to break it it will just cause too much more work.
>> > Quote:
>> >> >
>> >> > the projects has a bi directional link to each of the child
>> trees > >> (which from my further reading of the emf book seems to be
>> the > >> problem) thus while the ecore model can be split on a
>> epackage basis >> > the genmodel can't because everything depends on
>> the root and the >> root > depends on everything else... (is there
>> any way round this >> without > losing the root object as I can't see
>> it... I should note >> that the > resource and root object for each
>> of the child trees is >> individually > identifiable within the
>> resource set without requiring >> a containment > link)
>> >> Yes, a bidirectional reference is very intimate and things related
>> in >> this way can't be in separate GenModels.
>> >> >
>> >> > so when I load the root package all the pendant packages get
>> loaded >> (I > don't even think they are proxies which suprised me as
>> I would >> have > thought they would be)
>> >> > here is the relevant line in GenPackage >>
>> DependencyHelper.DependencyHelper
>> >> >
>> >> > interDependencies = new ArrayList<GenPackage>();
>> >> > collectPackages(interDependencies, > >>
>> genPackage.getGenModel().getGenPackages(), -1);
>> >> > interDependencies.remove(genPackage);
>> >> > addAll(interDependencies);
>> >> >
>> >> >
>> >> > which causes as series of statements of the form
>> >> > YYYYPackageImpl theYYYYPackage = > >>
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY > >>
>> YPackage.eNS_URI) instanceof YYYYPackageImpl ? > >>
>> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>> >> >
>> >> > to be generated for each package interdependency
>> >> >
>> >> > with YYYYPackage.eINSTANCE effectively resulting a statement of
>> the >> form
>> >> >
>> >> > YYYYPackage eINSTANCE = > >>
>> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>> >> >
>> >> > to be executed in the package YYYY
>> >> >
>> >> > which finally leeds to again in package YYYY
>> >> >
>> >> > if (isInited) return > >>
>> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack > >>
>> age.eNS_URI);
>> >> >
>> >> > // Obtain or create and register package
>> >> > YYYYPackageImpl theYYYYPackage = > >>
>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI) >
>> >> instanceof YYYYPackageImpl ? > >>
>> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new >>
>> YYYYPackageImpl());
>> >> >
>> >> > isInited = true;
>> >> >
>> >> > which instansiates the YYYY package if it hasn't been done so >>
>> already....
>> >> And yes, the packages must all be initialized to set up the >>
>> bidirectional references. There's no proxy resolution in generated
>> >> packages...
>> >
>> > that confirms what i surmised after a bit of digging...
>> > Quote:
>> >> >
>> >> >
>> >> > so by my understanding all packages within the same genmodel are
>> > >> interdependant so the real trick here is to split the
>> genmodel... > >> (however this seems hard as I guess any cross
>> reference to another > >> package even by operation or non
>> containment reference will prevent >> me > splitting it because it
>> will introduce an implicit bidrectional >> link...)
>> >> Bidirectional references can't be split across GenModels but any
>> >> other references can.
>> >
>> > I guess this include containment references which are implicitly >
>> bidirectional Quote:
>>
>> No. Only explicitly bidirectional references create a two-way
>> (mutual) dependency.
>
> that may be my get out clause then. I will investigate that next
> Quote:
>> >> >
>> >> > As a side note in the test I did I just loaded root package and
>> >> didn't > instantiate anything from the child packages. So all I
>> did was
>> >> > RootPackage rootPackage = RootPackage.eINSTANCE;
>> >> >
>> >> >
>> >> > again all or any help gratefully recieved
>> >> It's just how it works. Sorry.
>> >
>> > thats fine [I do science and boy do I know that I can't beat
>> nature...]
>> > On an aside I did have one slightly insane idea that might improve
>> > things. So here we go, please don't hesitate to shoot me if I have
>> > missed the obvious flaw, [apart from the fact that it might be a
>> lot > of work].
>> >
>> > So my problem isn't that all the models in the genmodel get loaded
>> but > that it takes a long time to do it (about 4 seconds). Not even
>> enough time to have a cup of coffee. Are the packages really big?
>> Are you sure the first one doesn't take 3.95 seconds because of other
>> initializations and that the other three take no appreciable amount
>> of time?
>
> no I checked the root package is one of the top five but the packages
> range in size from 5Mb to 84k for the ecore the total size is 27Mb for
> all packages
> Quote:
>> > So the idea was to load all the packageImpls together as currently
>> so > that loading one packImpl via its init method resisters all the
>> other > ones in the package registry at the same time. However the
>> twist is to > then to delay the loading of the rest of the meta data
>> in the packages > until first use for each package using the
>> Initialization on demand > holder idiom > [
>> http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom]
>> > to materialise the packages metadata as it is required so giving a
>> > lazy loading scheme for the metadat within the genmodel
>> And who's going to make that happen? :p
>
> I was planning to have a try at making it work to see if it was good.
> I take it the actual idea isn't insane? I guess I would then have to
> download the emf develeopment sources and run the unit tests and
> submit it as a patch on a bugzilla and hope that in the end it would
> be accepted after modification reveiew etc.
> Quote:
>> >
>> >
>> >
>> > Quote:
>> >> >
>> >> >
>> >
>> > regards gary
>
> regards
> gary
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: segmented models and lazy loading [message #504170 is a reply to message #504164] |
Thu, 17 December 2009 14:46 |
gary s thompson
Messages: 92
Registered: July 2009 |
Member |
|
|
Ed Merks wrote:
> Gary,
>
> Before you work on changes you assume will improve performance, I'd
> suggest carefully measure what's taking so long. You might well find,
> as I suggested, that the first package takes a lot of time because
> that's the point at which EMF's core runtime is initialized too.
>
HI Ed
Thats a fair comment. Premature optimisation is supposed to be the root
of all evil after all. So two further bits of information that I now
have to hand
1. I have run this under tftp and the initialisation code for my
packages were by far the worst offenders.
2. I have tried directly referencing the ecore code before startup and
it seems to make very little difference to load times for the models and
in fact directly timing it gives a time of a few hundred ms.
regards
gary
anyway I will look at quashing the bidirectional references first.
>
> gary s thompson wrote:
>> Ed Merks wrote on Thu, 17 December 2009 02:00
>>> Gary,
>>>
>>> Comments below.
>>>
>>> gary s thompson wrote:
>>> > hi Ed
>>> > Ed Merks wrote on Wed, 16 December 2009 12:16
>>> >> Gary,
>>> >>
>>> >>
>>> >>
>>> >> gary s thompson wrote:
>>> >> > ok ed here goes for hopefully enough detail
>>> >> >
>>> >> > the framework is organised as a project class plus a pendant
>>> series >> of > object trees of different types with each type of the
>>> pendant >> object > trees and acting as a containment hierachy and
>>> exising in >> its own > package and being stored as a resource [there
>>> maybe 0..n >> instances for > each of the pendant trees, so in anyone
>>> project many >> of them could be > empty].
>>> >> There's of course no relationship between how classes are divided
>>> >> into packages and how instances are divided into resources...
>>> >
>>> > this is true but in the case of the model i am trying reimplement >
>>> there is 1:1 relationship between resources and packages and i don't
>>> > want to break it it will just cause too much more work.
>>> > Quote:
>>> >> >
>>> >> > the projects has a bi directional link to each of the child
>>> trees > >> (which from my further reading of the emf book seems to be
>>> the > >> problem) thus while the ecore model can be split on a
>>> epackage basis >> > the genmodel can't because everything depends on
>>> the root and the >> root > depends on everything else... (is there
>>> any way round this >> without > losing the root object as I can't see
>>> it... I should note >> that the > resource and root object for each
>>> of the child trees is >> individually > identifiable within the
>>> resource set without requiring >> a containment > link)
>>> >> Yes, a bidirectional reference is very intimate and things related
>>> in >> this way can't be in separate GenModels.
>>> >> >
>>> >> > so when I load the root package all the pendant packages get
>>> loaded >> (I > don't even think they are proxies which suprised me as
>>> I would >> have > thought they would be)
>>> >> > here is the relevant line in GenPackage >>
>>> DependencyHelper.DependencyHelper
>>> >> >
>>> >> > interDependencies = new ArrayList<GenPackage>();
>>> >> > collectPackages(interDependencies, > >>
>>> genPackage.getGenModel().getGenPackages(), -1);
>>> >> > interDependencies.remove(genPackage);
>>> >> > addAll(interDependencies);
>>> >> >
>>> >> >
>>> >> > which causes as series of statements of the form
>>> >> > YYYYPackageImpl theYYYYPackage = > >>
>>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY > >>
>>> YPackage.eNS_URI) instanceof YYYYPackageImpl ? > >>
>>> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>>> >> >
>>> >> > to be generated for each package interdependency
>>> >> >
>>> >> > with YYYYPackage.eINSTANCE effectively resulting a statement of
>>> the >> form
>>> >> >
>>> >> > YYYYPackage eINSTANCE = > >>
>>> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>>> >> >
>>> >> > to be executed in the package YYYY
>>> >> >
>>> >> > which finally leeds to again in package YYYY
>>> >> >
>>> >> > if (isInited) return > >>
>>> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack > >>
>>> age.eNS_URI);
>>> >> >
>>> >> > // Obtain or create and register package
>>> >> > YYYYPackageImpl theYYYYPackage = > >>
>>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI) >
>>> >> instanceof YYYYPackageImpl ? > >>
>>> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new >>
>>> YYYYPackageImpl());
>>> >> >
>>> >> > isInited = true;
>>> >> >
>>> >> > which instansiates the YYYY package if it hasn't been done so >>
>>> already....
>>> >> And yes, the packages must all be initialized to set up the >>
>>> bidirectional references. There's no proxy resolution in generated
>>> >> packages...
>>> >
>>> > that confirms what i surmised after a bit of digging...
>>> > Quote:
>>> >> >
>>> >> >
>>> >> > so by my understanding all packages within the same genmodel are
>>> > >> interdependant so the real trick here is to split the
>>> genmodel... > >> (however this seems hard as I guess any cross
>>> reference to another > >> package even by operation or non
>>> containment reference will prevent >> me > splitting it because it
>>> will introduce an implicit bidrectional >> link...)
>>> >> Bidirectional references can't be split across GenModels but any
>>> >> other references can.
>>> >
>>> > I guess this include containment references which are implicitly >
>>> bidirectional Quote:
>>>
>>> No. Only explicitly bidirectional references create a two-way
>>> (mutual) dependency.
>>
>> that may be my get out clause then. I will investigate that next
>> Quote:
>>> >> >
>>> >> > As a side note in the test I did I just loaded root package and
>>> >> didn't > instantiate anything from the child packages. So all I
>>> did was
>>> >> > RootPackage rootPackage = RootPackage.eINSTANCE;
>>> >> >
>>> >> >
>>> >> > again all or any help gratefully recieved
>>> >> It's just how it works. Sorry.
>>> >
>>> > thats fine [I do science and boy do I know that I can't beat
>>> nature...]
>>> > On an aside I did have one slightly insane idea that might improve
>>> > things. So here we go, please don't hesitate to shoot me if I have
>>> > missed the obvious flaw, [apart from the fact that it might be a
>>> lot > of work].
>>> >
>>> > So my problem isn't that all the models in the genmodel get loaded
>>> but > that it takes a long time to do it (about 4 seconds). Not even
>>> enough time to have a cup of coffee. Are the packages really big?
>>> Are you sure the first one doesn't take 3.95 seconds because of other
>>> initializations and that the other three take no appreciable amount
>>> of time?
>>
>> no I checked the root package is one of the top five but the packages
>> range in size from 5Mb to 84k for the ecore the total size is 27Mb for
>> all packages
>> Quote:
>>> > So the idea was to load all the packageImpls together as currently
>>> so > that loading one packImpl via its init method resisters all the
>>> other > ones in the package registry at the same time. However the
>>> twist is to > then to delay the loading of the rest of the meta data
>>> in the packages > until first use for each package using the
>>> Initialization on demand > holder idiom > [
>>> http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom]
>>> > to materialise the packages metadata as it is required so giving a
>>> > lazy loading scheme for the metadat within the genmodel
>>> And who's going to make that happen? :p
>>
>> I was planning to have a try at making it work to see if it was good.
>> I take it the actual idea isn't insane? I guess I would then have to
>> download the emf develeopment sources and run the unit tests and
>> submit it as a patch on a bugzilla and hope that in the end it would
>> be accepted after modification reveiew etc.
>> Quote:
>>> >
>>> >
>>> >
>>> > Quote:
>>> >> >
>>> >> >
>>> >
>>> > regards gary
>>
>> regards
>> gary
|
|
|
| Re: segmented models and lazy loading [message #504899 is a reply to message #504170] |
Wed, 23 December 2009 07:55 |
gary s thompson
Messages: 92
Registered: July 2009 |
Member |
|
|
| gary s thompson wrote on Thu, 17 December 2009 09:46 | Ed Merks wrote:
> Gary,
>
> Before you work on changes you assume will improve performance, I'd
> suggest carefully measure what's taking so long. You might well find,
> as I suggested, that the first package takes a lot of time because
> that's the point at which EMF's core runtime is initialized too.
>
|
Hi Ed
just to let you know that I have dones some more work on my problem. I had a had a play with the Package class and now have (a non functional) version that just creates all the PackageImpls but doesn't initialise the contents and I see a reduction in initial load time of ~90% i.e. from 4-9 seconds to 0.4-0.9 seconds depending on how warm the jvm data is.
This seems to agree with the expectation that most of the work is in creating the structures inside the PackageImpls (some of the classes are rather broad in the number of functions attributes and references they provide [they are not my design but a port]) rather than creating the PackageImpls and registering them )
I will now work forward and see if I can get something that runs successfullly
regards
gary
nb if I don't talk again until after the 25th have a good christmas break
| Quote: |
HI Ed
Thats a fair comment. Premature optimisation is supposed to be the root
of all evil after all. So two further bits of information that I now
have to hand
1. I have run this under tftp and the initialisation code for my
packages were by far the worst offenders.
2. I have tried directly referencing the ecore code before startup and
it seems to make very little difference to load times for the models and
in fact directly timing it gives a time of a few hundred ms.
regards
gary
anyway I will look at quashing the bidirectional references first.
>
> gary s thompson wrote:
>> Ed Merks wrote on Thu, 17 December 2009 02:00
>>> Gary,
>>>
>>> Comments below.
>>>
>>> gary s thompson wrote:
>>> > hi Ed
>>> > Ed Merks wrote on Wed, 16 December 2009 12:16
>>> >> Gary,
>>> >>
>>> >>
>>> >>
>>> >> gary s thompson wrote:
>>> >> > ok ed here goes for hopefully enough detail
>>> >> >
>>> >> > the framework is organised as a project class plus a pendant
>>> series >> of > object trees of different types with each type of the
>>> pendant >> object > trees and acting as a containment hierachy and
>>> exising in >> its own > package and being stored as a resource [there
>>> maybe 0..n >> instances for > each of the pendant trees, so in anyone
>>> project many >> of them could be > empty].
>>> >> There's of course no relationship between how classes are divided
>>> >> into packages and how instances are divided into resources...
>>> >
>>> > this is true but in the case of the model i am trying reimplement >
>>> there is 1:1 relationship between resources and packages and i don't
>>> > want to break it it will just cause too much more work.
>>> > Quote:
>>> >> >
>>> >> > the projects has a bi directional link to each of the child
>>> trees > >> (which from my further reading of the emf book seems to be
>>> the > >> problem) thus while the ecore model can be split on a
>>> epackage basis >> > the genmodel can't because everything depends on
>>> the root and the >> root > depends on everything else... (is there
>>> any way round this >> without > losing the root object as I can't see
>>> it... I should note >> that the > resource and root object for each
>>> of the child trees is >> individually > identifiable within the
>>> resource set without requiring >> a containment > link)
>>> >> Yes, a bidirectional reference is very intimate and things related
>>> in >> this way can't be in separate GenModels.
>>> >> >
>>> >> > so when I load the root package all the pendant packages get
>>> loaded >> (I > don't even think they are proxies which suprised me as
>>> I would >> have > thought they would be)
>>> >> > here is the relevant line in GenPackage >>
>>> DependencyHelper.DependencyHelper
>>> >> >
>>> >> > interDependencies = new ArrayList<GenPackage>();
>>> >> > collectPackages(interDependencies, > >>
>>> genPackage.getGenModel().getGenPackages(), -1);
>>> >> > interDependencies.remove(genPackage);
>>> >> > addAll(interDependencies);
>>> >> >
>>> >> >
>>> >> > which causes as series of statements of the form
>>> >> > YYYYPackageImpl theYYYYPackage = > >>
>>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(YYY > >>
>>> YPackage.eNS_URI) instanceof YYYYPackageImpl ? > >>
>>> EPackage.Registry.INSTANCE.getEPackage(YYYYPackage.eNS_URI) : );
>>> >> >
>>> >> > to be generated for each package interdependency
>>> >> >
>>> >> > with YYYYPackage.eINSTANCE effectively resulting a statement of
>>> the >> form
>>> >> >
>>> >> > YYYYPackage eINSTANCE = > >>
>>> ccp.ecore_api.general.YYYY.impl.YYYYPackageImpl.init();
>>> >> >
>>> >> > to be executed in the package YYYY
>>> >> >
>>> >> > which finally leeds to again in package YYYY
>>> >> >
>>> >> > if (isInited) return > >>
>>> (YYYYPackage)EPackage.Registry.INSTANCE.getEPackage(YYYYPack > >>
>>> age.eNS_URI);
>>> >> >
>>> >> > // Obtain or create and register package
>>> >> > YYYYPackageImpl theYYYYPackage = > >>
>>> (YYYYPackageImpl)(EPackage.Registry.INSTANCE.getEPackage(eNS _URI) >
>>> >> instanceof YYYYPackageImpl ? > >>
>>> EPackage.Registry.INSTANCE.getEPackage(eNS_URI) : new >>
>>> YYYYPackageImpl());
>>> >> >
>>> >> > isInited = true;
>>> >> >
>>> >> > which instansiates the YYYY package if it hasn't been done so >>
>>> already....
>>> >> And yes, the packages must all be initialized to set up the >>
>>> bidirectional references. There's no proxy resolution in generated
>>> >> packages...
>>> >
>>> > that confirms what i surmised after a bit of digging...
>>> > Quote:
>>> >> >
>>> >> >
>>> >> > so by my understanding all packages within the same genmodel are
>>> > >> interdependant so the real trick here is to split the
>>> genmodel... > >> (however this seems hard as I guess any cross
>>> reference to another > >> package even by operation or non
>>> containment reference will prevent >> me > splitting it because it
>>> will introduce an implicit bidrectional >> link...)
>>> >> Bidirectional references can't be split across GenModels but any
>>> >> other references can.
>>> >
>>> > I guess this include containment references which are implicitly >
>>> bidirectional Quote:
>>>
>>> No. Only explicitly bidirectional references create a two-way
>>> (mutual) dependency.
>>
>> that may be my get out clause then. I will investigate that next
>> Quote:
>>> >> >
>>> >> > As a side note in the test I did I just loaded root package and
>>> >> didn't > instantiate anything from the child packages. So all I
>>> did was
>>> >> > RootPackage rootPackage = RootPackage.eINSTANCE;
>>> >> >
>>> >> >
>>> >> > again all or any help gratefully recieved
>>> >> It's just how it works. Sorry.
>>> >
>>> > thats fine [I do science and boy do I know that I can't beat
>>> nature...]
>>> > On an aside I did have one slightly insane idea that might improve
>>> > things. So here we go, please don't hesitate to shoot me if I have
>>> > missed the obvious flaw, [apart from the fact that it might be a
>>> lot > of work].
>>> >
>>> > So my problem isn't that all the models in the genmodel get loaded
>>> but > that it takes a long time to do it (about 4 seconds). Not even
>>> enough time to have a cup of coffee. Are the packages really big?
>>> Are you sure the first one doesn't take 3.95 seconds because of other
>>> initializations and that the other three take no appreciable amount
>>> of time?
>>
>> no I checked the root package is one of the top five but the packages
>> range in size from 5Mb to 84k for the ecore the total size is 27Mb for
>> all packages
>> Quote:
>>> > So the idea was to load all the packageImpls together as currently
>>> so > that loading one packImpl via its init method resisters all the
>>> other > ones in the package registry at the same time. However the
>>> twist is to > then to delay the loading of the rest of the meta data
>>> in the packages > until first use for each package using the
>>> Initialization on demand > holder idiom > [
>>> http://en.wikipedia.org/wiki/Initialization_on_demand_holder _idiom]
>>> > to materialise the packages metadata as it is required so giving a
>>> > lazy loading scheme for the metadat within the genmodel
>>> And who's going to make that happen?
>>
>> I was planning to have a try at making it work to see if it was good.
>> I take it the actual idea isn't insane? I guess I would then have to
>> download the emf develeopment sources and run the unit tests and
>> submit it as a patch on a bugzilla and hope that in the end it would
>> be accepted after modification reveiew etc.
>> Quote:
>>> >
>>> >
>>> >
>>> > Quote:
>>> >> >
>>> >> >
>>> >
>>> > regards gary
>>
>> regards
>> gary
|
|
|
|
Goto Forum:
Current Time: Fri Apr 19 04:58:32 GMT 2024
Powered by FUDForum. Page generated in 0.03841 seconds |
] 
Search
Help
Register
Login
Home
