Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Low performance of CDOResource.getURI()
[CDO] Low performance of CDOResource.getURI() [message #1411639] Sun, 24 August 2014 00:14 Go to next message
Leonid Ripeynih is currently offline Leonid RipeynihFriend
Messages: 150
Registered: February 2012
Senior Member
Hello!

I have encoutered some problems with CDOResource.getURI() performance, and subsequent problems with AbstractCDOView.newResourceInstance().

First, each call to getURI is getting about 150ms on my machine (Core i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in memory (revisions are cached). The resource im testing on is level 10 nested resource (common situation for our repository layout). This 150ms is constant, they do not depends on any possible cache, and is entirely made by recursive getPath() calls.

With AbstractCDOView.newResourceInstance() things gets much worse. It seems to call getURI() on each resource in current transaction three times - first in

// Bug 334995: Check if locally there is already a resource with the same URI
CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri, false);


Then in AbstractCDOView.getResource(String, boolean)
ensureURIs(resourceSet);

and
(CDOResource)resourceSet.getResource(uri, loadOnDemand);


For example, for relatively small repository of 30 000 resources, it takes about an hour for transfer framework to map (just map!) resources to their URIs. That seems pretty long to me. Transfer framework seems to exposure the problem in a most severe way - for each resource in repository (n) it checks uri of every other resource (n) which in order acceses all of it's parents (log (n)). It seems like mapping process takes n^2*log (n) accesses to getName() which is quite a large number.

Disabling the ensure URI check, and setting a resource to URI cache in resourceSet seems to help, as mapping time is reduced to abot one minute.

Still, opening a transaction with a dozen of resources under editing leads to a noticable lag to a user (few seconds) which is not desirable.

Is there anything that could be done about it, or am I doing something horribly wrong? Smile

Many thanks in advance!
Re: [CDO] Low performance of CDOResource.getURI() [message #1411830 is a reply to message #1411639] Sun, 24 August 2014 15:17 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 30353
Registered: July 2009
Senior Member
Leonid,

You might try creating a
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
Just creating one adds it as an adapter of the ResourceSet (and
contained resources), and then it's used to preemptively maintain a
perfect map from URI to resource, including being responsive addition,
removeals, direct changes to a Resource's URI, and even changes to the
URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
log(n)). Of course reducing the cost of the Resource.getURI would help
further in combination.

On 24/08/2014 2:14 AM, Leonid Ripeynih wrote:
> Hello!
>
> I have encoutered some problems with CDOResource.getURI() performance,
> and subsequent problems with AbstractCDOView.newResourceInstance().
>
> First, each call to getURI is getting about 150ms on my machine (Core
> i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in
> memory (revisions are cached). The resource im testing on is level 10
> nested resource (common situation for our repository layout). This
> 150ms is constant, they do not depends on any possible cache, and is
> entirely made by recursive getPath() calls.
> With AbstractCDOView.newResourceInstance() things gets much worse. It
> seems to call getURI() on each resource in current transaction three
> times - first in
>
> // Bug 334995: Check if locally there is already a resource with the
> same URI
> CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri,
> false);
>
>
> Then in AbstractCDOView.getResource(String, boolean)
>
> ensureURIs(resourceSet);
>
> and
>
> (CDOResource)resourceSet.getResource(uri, loadOnDemand);
>
>
> For example, for relatively small repository of 30 000 resources, it
> takes about an hour for transfer framework to map (just map!)
> resources to their URIs. That seems pretty long to me. Transfer
> framework seems to exposure the problem in a most severe way - for
> each resource in repository (n) it checks uri of every other resource
> (n) which in order acceses all of it's parents (log (n)). It seems
> like mapping process takes n^2*log (n) accesses to getName() which is
> quite a large number.
> Disabling the ensure URI check, and setting a resource to URI cache in
> resourceSet seems to help, as mapping time is reduced to abot one minute.
> Still, opening a transaction with a dozen of resources under editing
> leads to a noticable lag to a user (few seconds) which is not desirable.
> Is there anything that could be done about it, or am I doing something
> horribly wrong? :)
>
> Many thanks in advance!
Re: [CDO] Low performance of CDOResource.getURI() [message #1411863 is a reply to message #1411830] Sun, 24 August 2014 17:22 Go to previous messageGo to next message
Leonid Ripeynih is currently offline Leonid RipeynihFriend
Messages: 150
Registered: February 2012
Senior Member
Ed Merks wrote on Sun, 24 August 2014 11:17
Leonid,

You might try creating a
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
Just creating one adds it as an adapter of the ResourceSet (and
contained resources), and then it's used to preemptively maintain a
perfect map from URI to resource, including being responsive addition,
removeals, direct changes to a Resource's URI, and even changes to the
URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
log(n)). Of course reducing the cost of the Resource.getURI would help
further in combination.


Ed, thanks for you answer!

I'm afraid that this will not solve the problem completely.
There is more cases in CDO when resource can change it's URI/Path:

1. Setting the resource's 'name' feature to another value.
2. Setting one of the resource's containers (CDOResourceFolders) 'name' feature to another value.
3. Changing the container (CDOResourceFolder) of the resource to another one.

I have also looked at it's implementation, and it seems to use EContentAdapter internally, which can trigger loading of the entire repository in CDO which is a very long process.
Re: [CDO] Low performance of CDOResource.getURI() [message #1411867 is a reply to message #1411863] Sun, 24 August 2014 17:35 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 30353
Registered: July 2009
Senior Member
Leonid,

Comments below.

On 24/08/2014 7:22 PM, Leonid Ripeynih wrote:
> Ed Merks wrote on Sun, 24 August 2014 11:17
>> Leonid,
>>
>> You might try creating a
>> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
>> Just creating one adds it as an adapter of the ResourceSet (and
>> contained resources), and then it's used to preemptively maintain a
>> perfect map from URI to resource, including being responsive
>> addition, removeals, direct changes to a Resource's URI, and even
>> changes to the URIConverter's URI map. This should reduce the O(n^2
>> log(n)) to O(n log(n)). Of course reducing the cost of the
>> Resource.getURI would help further in combination.
>
>
> Ed, thanks for you answer!
>
> I'm afraid that this will not solve the problem completely.
> There is more cases in CDO when resource can change it's URI/Path:
>
> 1. Setting the resource's 'name' feature to another value.
That might just be a matter of specializing
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.ResourceAdapter.notifyChanged(Notification)
to deal with more than just the Resource.RESOURCE__URI notification...
> 2. Setting one of the resource's containers (CDOResourceFolders)
> 'name' feature to another value.
Is the container also a resource? It's certainly an object that could
be listened to I assume...
> 3. Changing the container (CDOResourceFolder) of the resource to
> another one.
So mostly this comes down to notification handling...
>
> I have also looked at it's implementation, and it seems to use
> EContentAdapter internally, which can trigger loading of the entire
> repository in CDO which is a very long process.
The adapter doesn't visit Resource.contents itself so should only end up
attached to the resource set and any resource added to the resource set,
but I don't know about this container thing or the rest of the CDO
details...

I expect Eike will have some more to say, you might you want to explore
further given your insights....
Re: [CDO] Low performance of CDOResource.getURI() [message #1412423 is a reply to message #1411639] Tue, 26 August 2014 05:50 Go to previous messageGo to next message
Leonid Ripeynih is currently offline Leonid RipeynihFriend
Messages: 150
Registered: February 2012
Senior Member
An attempt to use out-of-the-box MappedResourceLocator results in the following exception:

[ERROR] CDO view is read-only: View 3 [MAIN, 2014-08-18 16:57:39.278]
org.eclipse.emf.cdo.util.ReadOnlyException: CDO view is read-only: View 3 [MAIN, 2014-08-18 16:57:39.278]
	at org.eclipse.emf.internal.cdo.view.AbstractCDOView.attachResource(AbstractCDOView.java:1278)
	at org.eclipse.emf.cdo.eresource.impl.CDOResourceImpl.basicSetResourceSet(CDOResourceImpl.java:1534)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourcesEList.inverseAdd(ResourceSetImpl.java:586)
	at org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUnique(NotifyingListImpl.java:286)
	at org.eclipse.emf.common.util.AbstractEList.add(AbstractEList.java:303)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandCreateResource(ResourceSetImpl.java:243)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourceLocator.demandCreateResource(ResourceSetImpl.java:663)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$MappedResourceLocator.getResource(ResourceSetImpl.java:1057)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:352)
	at org.eclipse.emf.internal.cdo.view.AbstractCDOView.getResource(AbstractCDOView.java:654)
	at org.eclipse.emf.internal.cdo.view.AbstractCDOView.newResourceInstance(AbstractCDOView.java:1050)
	at org.eclipse.emf.internal.cdo.view.AbstractCDOView.createObject(AbstractCDOView.java:1020)
	at org.eclipse.emf.internal.cdo.view.AbstractCDOView.getObject(AbstractCDOView.java:914)
Re: [CDO] Low performance of CDOResource.getURI() [message #1412431 is a reply to message #1412423] Tue, 26 August 2014 06:23 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 30353
Registered: July 2009
Senior Member
Leonid,

Eike will need to comment. Even without a mapped locator, one should
still get into these same stack frames:

at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
at
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandCreateResource(ResourceSetImpl.java:243)

After all, from the stack trace you show, it's trying to demand load a
resource into the resource set, and one way or another, that needs to
create a resource and load it, which should not be considered a
non-read-only update...

Maybe you compare what's happening in

at
org.eclipse.emf.internal.cdo.view.AbstractCDOView.attachResource(AbstractCDOView.java:1278)

in both the case there is a mapped locator and in the case there isn't;
in either case, I would expect demandCreateResource followed by
createResource to be called...

On 26/08/2014 7:50 AM, Leonid Ripeynih wrote:
> An attempt to use out-of-the-box MappedResourceLocator results in the
> following exception:
>
>
> [ERROR] CDO view is read-only: View 3 [MAIN, 2014-08-18 16:57:39.278]
> org.eclipse.emf.cdo.util.ReadOnlyException: CDO view is read-only:
> View 3 [MAIN, 2014-08-18 16:57:39.278]
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.attachResource(AbstractCDOView.java:1278)
> at
> org.eclipse.emf.cdo.eresource.impl.CDOResourceImpl.basicSetResourceSet(CDOResourceImpl.java:1534)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourcesEList.inverseAdd(ResourceSetImpl.java:586)
> at
> org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUnique(NotifyingListImpl.java:286)
> at
> org.eclipse.emf.common.util.AbstractEList.add(AbstractEList.java:303)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandCreateResource(ResourceSetImpl.java:243)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$ResourceLocator.demandCreateResource(ResourceSetImpl.java:663)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$MappedResourceLocator.getResource(ResourceSetImpl.java:1057)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:352)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.getResource(AbstractCDOView.java:654)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.newResourceInstance(AbstractCDOView.java:1050)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.createObject(AbstractCDOView.java:1020)
> at
> org.eclipse.emf.internal.cdo.view.AbstractCDOView.getObject(AbstractCDOView.java:914)
>
Re: [CDO] Low performance of CDOResource.getURI() [message #1412484 is a reply to message #1411830] Tue, 26 August 2014 09:22 Go to previous messageGo to next message
Esteban Dugueperoux is currently offline Esteban DugueperouxFriend
Messages: 472
Registered: July 2009
Senior Member
Hi Leonid and Ed,

Note that MappedResourceLocator could have issue with CDOResource in
case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
call cdo doesn't trigger EMF notification as it is done in
ResourceImpl.setURI().

Leonid, could you raise a bugzilla about this issue?

Best Regards.

Le 24/08/2014 17:17, Ed Merks a écrit :
> Leonid,
>
> You might try creating a
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.MappedResourceLocator.
> Just creating one adds it as an adapter of the ResourceSet (and
> contained resources), and then it's used to preemptively maintain a
> perfect map from URI to resource, including being responsive addition,
> removeals, direct changes to a Resource's URI, and even changes to the
> URIConverter's URI map. This should reduce the O(n^2 log(n)) to O(n
> log(n)). Of course reducing the cost of the Resource.getURI would help
> further in combination.
>
> On 24/08/2014 2:14 AM, Leonid Ripeynih wrote:
>> Hello!
>>
>> I have encoutered some problems with CDOResource.getURI() performance,
>> and subsequent problems with AbstractCDOView.newResourceInstance().
>>
>> First, each call to getURI is getting about 150ms on my machine (Core
>> i5, 8G RAM, Heap 300Mb/1G). From CDO point of view - everything is in
>> memory (revisions are cached). The resource im testing on is level 10
>> nested resource (common situation for our repository layout). This
>> 150ms is constant, they do not depends on any possible cache, and is
>> entirely made by recursive getPath() calls.
>> With AbstractCDOView.newResourceInstance() things gets much worse. It
>> seems to call getURI() on each resource in current transaction three
>> times - first in
>>
>> // Bug 334995: Check if locally there is already a resource with the
>> same URI
>> CDOResource resource1 = (CDOResource)getResourceSet().getResource(uri,
>> false);
>>
>>
>> Then in AbstractCDOView.getResource(String, boolean)
>>
>> ensureURIs(resourceSet);
>>
>> and
>>
>> (CDOResource)resourceSet.getResource(uri, loadOnDemand);
>>
>>
>> For example, for relatively small repository of 30 000 resources, it
>> takes about an hour for transfer framework to map (just map!)
>> resources to their URIs. That seems pretty long to me. Transfer
>> framework seems to exposure the problem in a most severe way - for
>> each resource in repository (n) it checks uri of every other resource
>> (n) which in order acceses all of it's parents (log (n)). It seems
>> like mapping process takes n^2*log (n) accesses to getName() which is
>> quite a large number.
>> Disabling the ensure URI check, and setting a resource to URI cache in
>> resourceSet seems to help, as mapping time is reduced to abot one minute.
>> Still, opening a transaction with a dozen of resources under editing
>> leads to a noticable lag to a user (few seconds) which is not desirable.
>> Is there anything that could be done about it, or am I doing something
>> horribly wrong? :)
>>
>> Many thanks in advance!
>


--
Esteban Dugueperoux - Obeo

Need professional services for Sirius?
http://www.obeodesigner.com/sirius
Re: [CDO] Low performance of CDOResource.getURI() [message #1412492 is a reply to message #1412484] Tue, 26 August 2014 09:45 Go to previous messageGo to next message
Leonid Ripeynih is currently offline Leonid RipeynihFriend
Messages: 150
Registered: February 2012
Senior Member
Esteban Dugueperoux wrote on Tue, 26 August 2014 05:22
Hi Leonid and Ed,

Note that MappedResourceLocator could have issue with CDOResource in
case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
call cdo doesn't trigger EMF notification as it is done in
ResourceImpl.setURI().

Leonid, could you raise a bugzilla about this issue?


Hello, Esteban.

This could sertanly be an issue, but I think that the real issue (and it's fix) is withing CDO. If a measure it right, perfomance of AbstractCDOView.newResourceInstance() is pretty horrible, both because of large overhead CDOResourceNode.getURI() introduce and becouse of it's time constraints (n^2 for the cases, when major parts of repository is displayed).

Unfortunately theese cases are very real, for example for us, when opening EMF Compare window, 70% of times (that's minutes) is taken by an AbstractCDOView.newResourceInstance(). The rest 30% is both creating of CDOComparisonScope requesting the MergeData, which both requires round trips to server. And iterating over a local cache (!) takes much longer.

Here is an idea, on optimizing it:

As CDO controls each change to every CDOObject, it seems possible to see, of it changes CDOResourceNode.name or CDOResourceFolder.nodes feature, and, in simplest case, set an 'invalidateURICache' flag on CDOView to true.

getURI() implementation should be rewritten to cache it's value, and re-calculate it only when the corresponding CDOView's 'invalidateURICache' flag is set to true.

I have testsd this solution (whithout the invaliadtion part) and it seems to gain about 5 times perfomance increase in our case.

I would be glad to hear comments on this, as I'm not an CDO expert, and there is shure some pitfalls I may have overlooked.
Re: [CDO] Low performance of CDOResource.getURI() [message #1412498 is a reply to message #1412492] Tue, 26 August 2014 09:53 Go to previous messageGo to next message
Esteban Dugueperoux is currently offline Esteban DugueperouxFriend
Messages: 472
Registered: July 2009
Senior Member
You are right, I have not read all messages of this thread, but indeed
the CDOResourceImpl.getURI() method is not efficient.

Could you raise also a bugzilla for this issue?

Best Regards.

Le 26/08/2014 11:45, Leonid Ripeynih a écrit :
> Esteban Dugueperoux wrote on Tue, 26 August 2014 05:22
>> Hi Leonid and Ed,
>>
>> Note that MappedResourceLocator could have issue with CDOResource in
>> case of changed CDOResource's URI. Because on CDOResourceImpl.setURI()
>> call cdo doesn't trigger EMF notification as it is done in
>> ResourceImpl.setURI().
>>
>> Leonid, could you raise a bugzilla about this issue?
>
>
> Hello, Esteban.
>
> This could sertanly be an issue, but I think that the real issue (and
> it's fix) is withing CDO. If a measure it right, perfomance of
> AbstractCDOView.newResourceInstance() is pretty horrible, both because
> of large overhead CDOResourceNode.getURI() introduce and becouse of it's
> time constraints (n^2 for the cases, when major parts of repository is
> displayed).
> Unfortunately theese cases are very real, for example for us, when
> opening EMF Compare window, 70% of times (that's minutes) is taken by an
> AbstractCDOView.newResourceInstance(). The rest 30% is both creating of
> CDOComparisonScope requesting the MergeData, which both requires round
> trips to server. And iterating over a local cache (!) takes much longer.
>
> Here is an idea, on optimizing it:
>
> As CDO controls each change to every CDOObject, it seems possible to
> see, of it changes CDOResourceNode.name or CDOResourceFolder.nodes
> feature, and, in simplest case, set an 'invalidateURICache' flag on
> CDOView to true.
> getURI() implementation should be rewritten to cache it's value, and
> re-calculate it only when the corresponding CDOView's
> 'invalidateURICache' flag is set to true.
> I have testsd this solution (whithout the invaliadtion part) and it
> seems to gain about 5 times perfomance increase in our case.
> I would be glad to hear comments on this, as I'm not an CDO expert, and
> there is shure some pitfalls I may have overlooked.


--
Esteban Dugueperoux - Obeo

Need professional services for Sirius?
http://www.obeodesigner.com/sirius
Re: [CDO] Low performance of CDOResource.getURI() [message #1412504 is a reply to message #1412498] Tue, 26 August 2014 10:13 Go to previous messageGo to next message
Leonid Ripeynih is currently offline Leonid RipeynihFriend
Messages: 150
Registered: February 2012
Senior Member
Esteban Dugueperoux wrote on Tue, 26 August 2014 05:53


Could you raise also a bugzilla for this issue?


Done, it's here https://bugs.eclipse.org/bugs/show_bug.cgi?id=442558

Also, can we discuss ways to fix it? It's quite important for us, so I have time and willingness to work on a fix (and i will contribute it to CDO of course), given a few ideas on there to look.
Re: [CDO] Low performance of CDOResource.getURI() [message #1412554 is a reply to message #1412504] Tue, 26 August 2014 12:43 Go to previous message
Esteban Dugueperoux is currently offline Esteban DugueperouxFriend
Messages: 472
Registered: July 2009
Senior Member
Perhaps we could have a cache similar to what we have when we call
CDOView.getResource().
I have no time currently to analyse this performance issue.
But you can provide an analysis, a solution with a Gerrit with some
tests and I will review that gracefully :).

Best Regards.

Le 26/08/2014 12:13, Leonid Ripeynih a écrit :
> Esteban Dugueperoux wrote on Tue, 26 August 2014 05:53
>> Could you raise also a bugzilla for this issue?
>
>
> Done, it's here https://bugs.eclipse.org/bugs/show_bug.cgi?id=442558
>
> Also, can we discuss ways to fix it? It's quite important for us, so I
> have time and willingness to work on a fix (and i will contribute it to
> CDO of course), given a few ideas on there to look.


--
Esteban Dugueperoux - Obeo

Need professional services for Sirius?
http://www.obeodesigner.com/sirius
Previous Topic:[Compare] How to customize EMF compare mechanism - Match Phase?
Next Topic:Character Encoding
Goto Forum:
  


Current Time: Tue Aug 20 19:31:00 GMT 2019

Powered by FUDForum. Page generated in 0.03164 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top