Home » Modeling » EMF » [CDO] Low performance of CDOResource.getURI()
| [CDO] Low performance of CDOResource.getURI() [message #1411639] |
Sun, 24 August 2014 00:14  |
 Leonid Ripeynih
Messages: 150
Registered: February 2012 |
Senior Member |
|
|
Hello!
I have encoutered some problems with CDOResource.getURI() performance, and subsequent problems with AbstractCDOView.newResourceInstance().
First, each call to getURI is getting about 150ms on my machine (Core i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in memory (revisions are cached). The resource im testing on is level 10 nested resource (common situation for our repository layout). This 150ms is constant, they do not depends on any possible cache, and is entirely made by recursive getPath() calls.
With AbstractCDOView.newResourceInstance() things gets much worse. It seems to call getURI() on each resource in current transaction three times - first in
// Bug 334995: Check if locally there is already a resource with the same URI
CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri, false);
Then in AbstractCDOView.getResource(String, boolean)
and
(CDOResource)resourceSet.getResource(uri, loadOnDemand);
For example, for relatively small repository of 30 000 resources, it takes about an hour for transfer framework to map (just map!) resources to their URIs. That seems pretty long to me. Transfer framework seems to exposure the problem in a most severe way - for each resource in repository (n) it checks uri of every other resource (n) which in order acceses all of it's parents (log (n)). It seems like mapping process takes n^2*log (n) accesses to getName() which is quite a large number.
Disabling the ensure URI check, and setting a resource to URI cache in resourceSet seems to help, as mapping time is reduced to abot one minute.
Still, opening a transaction with a dozen of resources under editing leads to a noticable lag to a user (few seconds) which is not desirable.
Is there anything that could be done about it, or am I doing something horribly wrong? 
Many thanks in advance!
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1411830 is a reply to message #1411639] |
Sun, 24 August 2014 15:17  |
Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
Leonid,
You might try creating a
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
Just creating one adds it as an adapter of the ResourceSet (and
contained resources), and then it's used to preemptively maintain a
perfect map from URI to resource, including being responsive addition,
removeals, direct changes to a Resource's URI, and even changes to the
URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
log(n)). Of course reducing the cost of the Resource.getURI would help
further in combination.
On 24/08/2014 2:14 AM, Leonid Ripeynih wrote:
> Hello!
>
> I have encoutered some problems with CDOResource.getURI() performance,
> and subsequent problems with AbstractCDOView.newResourceInstance().
>
> First, each call to getURI is getting about 150ms on my machine (Core
> i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in
> memory (revisions are cached). The resource im testing on is level 10
> nested resource (common situation for our repository layout). This
> 150ms is constant, they do not depends on any possible cache, and is
> entirely made by recursive getPath() calls.
> With AbstractCDOView.newResourceInstance() things gets much worse. It
> seems to call getURI() on each resource in current transaction three
> times - first in
>
> // Bug 334995: Check if locally there is already a resource with the
> same URI
> CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri,
> false);
>
>
> Then in AbstractCDOView.getResource(String, boolean)
>
> ensureURIs(resourceSet);
>
> and
>
> (CDOResource)resourceSet.getResource(uri, loadOnDemand);
>
>
> For example, for relatively small repository of 30 000 resources, it
> takes about an hour for transfer framework to map (just map!)
> resources to their URIs. That seems pretty long to me. Transfer
> framework seems to exposure the problem in a most severe way - for
> each resource in repository (n) it checks uri of every other resource
> (n) which in order acceses all of it's parents (log (n)). It seems
> like mapping process takes n^2*log (n) accesses to getName() which is
> quite a large number.
> Disabling the ensure URI check, and setting a resource to URI cache in
> resourceSet seems to help, as mapping time is reduced to abot one minute.
> Still, opening a transaction with a dozen of resources under editing
> leads to a noticable lag to a user (few seconds) which is not desirable.
> Is there anything that could be done about it, or am I doing something
> horribly wrong? :)
>
> Many thanks in advance!
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1411863 is a reply to message #1411830] |
Sun, 24 August 2014 17:22 |
Leonid Ripeynih
Messages: 150
Registered: February 2012 |
Senior Member |
|
|
Ed Merks wrote on Sun, 24 August 2014 11:17Leonid,
You might try creating a
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
Just creating one adds it as an adapter of the ResourceSet (and
contained resources), and then it's used to preemptively maintain a
perfect map from URI to resource, including being responsive addition,
removeals, direct changes to a Resource's URI, and even changes to the
URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
log(n)). Of course reducing the cost of the Resource.getURI would help
further in combination.
Ed, thanks for you answer!
I'm afraid that this will not solve the problem completely.
There is more cases in CDO when resource can change it's URI/Path:
1. Setting the resource's 'name' feature to another value.
2. Setting one of the resource's containers (CDOResourceFolders) 'name' feature to another value.
3. Changing the container (CDOResourceFolder) of the resource to another one.
I have also looked at it's implementation, and it seems to use EContentAdapter internally, which can trigger loading of the entire repository in CDO which is a very long process.
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1411867 is a reply to message #1411863] |
Sun, 24 August 2014 17:35 |
Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
Leonid,
Comments below.
On 24/08/2014 7:22 PM, Leonid Ripeynih wrote:
> Ed Merks wrote on Sun, 24 August 2014 11:17
>> Leonid,
>>
>> You might try creating a
>> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
>> Just creating one adds it as an adapter of the ResourceSet (and
>> contained resources), and then it's used to preemptively maintain a
>> perfect map from URI to resource, including being responsive
>> addition, removeals, direct changes to a Resource's URI, and even
>> changes to the URIConverter's URI map. This should reduce the O(n^2
>> log(n)) to O(n log(n)). Of course reducing the cost of the
>> Resource.getURI would help further in combination.
>
>
> Ed, thanks for you answer!
>
> I'm afraid that this will not solve the problem completely.
> There is more cases in CDO when resource can change it's URI/Path:
>
> 1. Setting the resource's 'name' feature to another value.
That might just be a matter of specializing
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.ResourceAdapter.notifyChanged(Notification)
to deal with more than just the Resource.RESOURCE__URI notification...
> 2. Setting one of the resource's containers (CDOResourceFolders)
> 'name' feature to another value.
Is the container also a resource? It's certainly an object that could
be listened to I assume...
> 3. Changing the container (CDOResourceFolder) of the resource to
> another one.
So mostly this comes down to notification handling...
>
> I have also looked at it's implementation, and it seems to use
> EContentAdapter internally, which can trigger loading of the entire
> repository in CDO which is a very long process.
The adapter doesn't visit Resource.contents itself so should only end up
attached to the resource set and any resource added to the resource set,
but I don't know about this container thing or the rest of the CDO
details...
I expect Eike will have some more to say, you might you want to explore
further given your insights....
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
| |
| Re: [CDO] Low performance of CDOResource.getURI() [message #1412431 is a reply to message #1412423] |
Tue, 26 August 2014 06:23 |
Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
Leonid,
Eike will need to comment. Even without a mapped locator, one should
still get into these same stack frames:
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandCreateResource(ResourceSetImpl.java:243)
After all, from the stack trace you show, it's trying to demand load a
resource into the resource set, and one way or another, that needs to
create a resource and load it, which should not be considered a
non-read-only update...
Maybe you compare what's happening in
at
org.eclipse.emf.internal.cdo.view.AbstractCDOView.attachResource(AbstractCDOView.java:1278)
in both the case there is a mapped locator and in the case there isn't;
in either case, I would expect demandCreateResource followed by
createResource to be called...
On 26/08/2014 7:50 AM, Leonid Ripeynih wrote:
> An attempt to use out-of-the-box MappedResourceLocator results in the
> following exception:
>
>
> [ERROR] CDO view is read-only: View 3 [MAIN, 2014-08-18 16:57:39.278]
> org.eclipse.emf.cdo.util.ReadOnlyException: CDO view is read-only:
> View 3 [MAIN, 2014-08-18 16:57:39.278]
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.attachResource(AbstractCDOView.java:1278)
> at
> org.eclipse.emf.cdo.eresource.impl.CDOResourceImpl.basicSetResourceSet(CDOResourceImpl.java:1534)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourcesEList.inverseAdd(ResourceSetImpl.java:586)
> at
> org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUnique(NotifyingListImpl.java:286)
> at
> org.eclipse.emf.common.util.AbstractEList.add(AbstractEList.java:303)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandCreateResource(ResourceSetImpl.java:243)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourceLocator.demandCreateResource(ResourceSetImpl.java:663)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$MappedResourceLocator.getResource(ResourceSetImpl.java:1057)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:352)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.getResource(AbstractCDOView.java:654)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.newResourceInstance(AbstractCDOView.java:1050)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.createObject(AbstractCDOView.java:1020)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.getObject(AbstractCDOView.java:914)
>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1412484 is a reply to message #1411830] |
Tue, 26 August 2014 09:22 |
Esteban Dugueperoux
Messages: 472
Registered: July 2009 |
Senior Member |
|
|
Hi Leonid and Ed,
Note that MappedResourceLocator could have issue with CDOResource in
case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
call cdo doesn't trigger EMF notification as it is done in
ResourceImpl.setURI().
Leonid, could you raise a bugzilla about this issue?
Best Regards.
Le 24/08/2014 17:17, Ed Merks a écrit :
> Leonid,
>
> You might try creating a
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
> Just creating one adds it as an adapter of the ResourceSet (and
> contained resources), and then it's used to preemptively maintain a
> perfect map from URI to resource, including being responsive addition,
> removeals, direct changes to a Resource's URI, and even changes to the
> URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
> log(n)). Of course reducing the cost of the Resource.getURI would help
> further in combination.
>
> On 24/08/2014 2:14 AM, Leonid Ripeynih wrote:
>> Hello!
>>
>> I have encoutered some problems with CDOResource.getURI() performance,
>> and subsequent problems with AbstractCDOView.newResourceInstance().
>>
>> First, each call to getURI is getting about 150ms on my machine (Core
>> i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in
>> memory (revisions are cached). The resource im testing on is level 10
>> nested resource (common situation for our repository layout). This
>> 150ms is constant, they do not depends on any possible cache, and is
>> entirely made by recursive getPath() calls.
>> With AbstractCDOView.newResourceInstance() things gets much worse. It
>> seems to call getURI() on each resource in current transaction three
>> times - first in
>>
>> // Bug 334995: Check if locally there is already a resource with the
>> same URI
>> CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri,
>> false);
>>
>>
>> Then in AbstractCDOView.getResource(String, boolean)
>>
>> ensureURIs(resourceSet);
>>
>> and
>>
>> (CDOResource)resourceSet.getResource(uri, loadOnDemand);
>>
>>
>> For example, for relatively small repository of 30 000 resources, it
>> takes about an hour for transfer framework to map (just map!)
>> resources to their URIs. That seems pretty long to me. Transfer
>> framework seems to exposure the problem in a most severe way - for
>> each resource in repository (n) it checks uri of every other resource
>> (n) which in order acceses all of it's parents (log (n)). It seems
>> like mapping process takes n^2*log (n) accesses to getName() which is
>> quite a large number.
>> Disabling the ensure URI check, and setting a resource to URI cache in
>> resourceSet seems to help, as mapping time is reduced to abot one minute.
>> Still, opening a transaction with a dozen of resources under editing
>> leads to a noticable lag to a user (few seconds) which is not desirable.
>> Is there anything that could be done about it, or am I doing something
>> horribly wrong? :)
>>
>> Many thanks in advance!
>
--
Esteban Dugueperoux - Obeo
Need professional services for Sirius?
http://www.obeodesigner.com/sirius
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1412492 is a reply to message #1412484] |
Tue, 26 August 2014 09:45 |
Leonid Ripeynih
Messages: 150
Registered: February 2012 |
Senior Member |
|
|
Esteban Dugueperoux wrote on Tue, 26 August 2014 05:22Hi Leonid and Ed,
Note that MappedResourceLocator could have issue with CDOResource in
case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
call cdo doesn't trigger EMF notification as it is done in
ResourceImpl.setURI().
Leonid, could you raise a bugzilla about this issue?
Hello, Esteban.
This could sertanly be an issue, but I think that the real issue (and it's fix) is withing CDO. If a measure it right, perfomance of AbstractCDOView.newResourceInstance() is pretty horrible, both because of large overhead CDOResourceNode.getURI() introduce and becouse of it's time constraints (n^2 for the cases, when major parts of repository is displayed).
Unfortunately theese cases are very real, for example for us, when opening EMF Compare window, 70% of times (that's minutes) is taken by an AbstractCDOView.newResourceInstance(). The rest 30% is both creating of CDOComparisonScope requesting the MergeData, which both requires round trips to server. And iterating over a local cache (!) takes much longer.
Here is an idea, on optimizing it:
As CDO controls each change to every CDOObject, it seems possible to see, of it changes CDOResourceNode.name or CDOResourceFolder.nodes feature, and, in simplest case, set an 'invalidateURICache' flag on CDOView to true.
getURI() implementation should be rewritten to cache it's value, and re-calculate it only when the corresponding CDOView's 'invalidateURICache' flag is set to true.
I have testsd this solution (whithout the invaliadtion part) and it seems to gain about 5 times perfomance increase in our case.
I would be glad to hear comments on this, as I'm not an CDO expert, and there is shure some pitfalls I may have overlooked.
|
|
|
| Re: [CDO] Low performance of CDOResource.getURI() [message #1412498 is a reply to message #1412492] |
Tue, 26 August 2014 09:53 |
Esteban Dugueperoux
Messages: 472
Registered: July 2009 |
Senior Member |
|
|
You are right, I have not read all messages of this thread, but indeed
the CDOResourceImpl.getURI() method is not efficient.
Could you raise also a bugzilla for this issue?
Best Regards.
Le 26/08/2014 11:45, Leonid Ripeynih a écrit :
> Esteban Dugueperoux wrote on Tue, 26 August 2014 05:22
>> Hi Leonid and Ed,
>>
>> Note that MappedResourceLocator could have issue with CDOResource in
>> case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
>> call cdo doesn't trigger EMF notification as it is done in
>> ResourceImpl.setURI().
>>
>> Leonid, could you raise a bugzilla about this issue?
>
>
> Hello, Esteban.
>
> This could sertanly be an issue, but I think that the real issue (and
> it's fix) is withing CDO. If a measure it right, perfomance of
> AbstractCDOView.newResourceInstance() is pretty horrible, both because
> of large overhead CDOResourceNode.getURI() introduce and becouse of it's
> time constraints (n^2 for the cases, when major parts of repository is
> displayed).
> Unfortunately theese cases are very real, for example for us, when
> opening EMF Compare window, 70% of times (that's minutes) is taken by an
> AbstractCDOView.newResourceInstance(). The rest 30% is both creating of
> CDOComparisonScope requesting the MergeData, which both requires round
> trips to server. And iterating over a local cache (!) takes much longer.
>
> Here is an idea, on optimizing it:
>
> As CDO controls each change to every CDOObject, it seems possible to
> see, of it changes CDOResourceNode.name or CDOResourceFolder.nodes
> feature, and, in simplest case, set an 'invalidateURICache' flag on
> CDOView to true.
> getURI() implementation should be rewritten to cache it's value, and
> re-calculate it only when the corresponding CDOView's
> 'invalidateURICache' flag is set to true.
> I have testsd this solution (whithout the invaliadtion part) and it
> seems to gain about 5 times perfomance increase in our case.
> I would be glad to hear comments on this, as I'm not an CDO expert, and
> there is shure some pitfalls I may have overlooked.
--
Esteban Dugueperoux - Obeo
Need professional services for Sirius?
http://www.obeodesigner.com/sirius
|
|
| | |
Goto Forum:
Current Time: Wed Apr 24 14:13:34 GMT 2024
Powered by FUDForum. Page generated in 0.04295 seconds |
] 
Search
Help
Register
Login
Home
