 [CDO][Net4j] Revision loading performance, compareRevisions [message #843344] |
Thu, 12 April 2012 20:58  |
Eclipse User |
|
|
|
Hi,
Our application uses CDOView.compareRevisions(CDOBranchPoint) and a pair of audit views to find out what happened on the server while the client was off-line; the resulting ChangeSetData is used in one place to update Lucene indexes in the client.
We ran into performance problems when getting CDOObjects for the revision keys in this change set, and using these as a starting point to reach out into the object graph, as revisions for referenced objects that did not change between the two branch points in question are missing from the client-side cache and require individual loading requests. These references are rarely re-used, so cache warming and cdoPrefetch(int) calls did not help. In addition to that, my general experience was that this stream of LoadRevisionsRequests take substantially more time to complete over a TCP connection when compared to the JVM transport.
My question is twofold:
a) One idea for solving the problem was to register additional revisions into both source and target availability infos so they can be returned with the response for the merge data loading request, cached in the client and then hidden from the change set (as any created revision delta would be empty for them). Does this sound feasible by any means? Perhaps fetch rules could be applied when loading merge data?
b) Is there any tweakable setting (eg. buffer sizes, timeouts) that would reduce Net4j's latency when using TCP connections, assuming that there are no other externally introduced delays that we can trim down?
Regards,
András
|
|
|
| Re: [CDO][Net4j] Revision loading performance, compareRevisions [message #845004 is a reply to message #843344] |
Sat, 14 April 2012 12:24  |
| Eclipse User |
|
|
|
Hi Andras,
Comments below...
Am 13.04.2012 02:58, schrieb András Péteri:
> Hi,
>
> Our application uses CDOView.compareRevisions(CDOBranchPoint) and a pair of audit views to find out what happened on
> the server while the client was off-line; the resulting ChangeSetData is used in one place to update Lucene indexes in
> the client.
> We ran into performance problems
Can you please be more specific about this problem? Which IStore and database are you using? How big is the model at the
two CDOBranchPoints being compared with each other? How long does it take? What did you hope for?
> when getting CDOObjects for the revision keys in this change set, and using these as a starting point to reach out
> into the object graph, as revisions for referenced objects that did not change between the two branch points in
> question are missing from the client-side cache and require individual loading requests.
Sounds normal to me.
> These references are rarely re-used, so cache warming and cdoPrefetch(int) calls did not help.
Seems to make sense.
> In addition to that, my general experience was that this stream of LoadRevisionsRequests take substantially more time
> to complete over a TCP connection when compared to the JVM transport.
Also what I'd expect.
>
> My question is twofold:
>
> a) One idea for solving the problem was to register additional revisions into both source and target availability
> infos so they can be returned with the response for the merge data loading request, cached in the client and then
> hidden from the change set (as any created revision delta would be empty for them). Does this sound feasible by any
> means?
In general, yes. But when/where and how would those additional revisions be determined?
> Perhaps fetch rules could be applied when loading merge data?
Are you referring to org.eclipse.emf.cdo.common.util.CDOFetchRule?
>
> b) Is there any tweakable setting (eg. buffer sizes, timeouts) that would reduce Net4j's latency when using TCP
> connections, assuming that there are no other externally introduced delays that we can trim down?
Buffer sizes could be configured. Depending on your setup for example by replacing the default
org.eclipse.internal.net4j.buffer.BufferProviderFactory in your IPluginContainer.INSTANCE.
See also org.eclipse.net4j/plugin.xml:
<extension
point="org.eclipse.net4j.util.factories">
<factory
class="org.eclipse.internal.net4j.buffer.BufferProviderFactory"
productGroup="org.eclipse.net4j.bufferProviders"
type="default"/>
<factory
class="org.eclipse.net4j.signal.heartbeat.HeartBeatProtocol$Server$Factory"
productGroup="org.eclipse.net4j.serverProtocols"
type="heartbeat"/>
</extension>
The impact of buffer size changes is not very predictable. Please let me know about your experiences.
Timeout values are usually not good candidates to increase throughput under normal conditions, i.e., they should just be
noticeable in case of errors/problems.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO][Net4j] Revision loading performance, compareRevisions [message #846607 is a reply to message #845004] |
Mon, 16 April 2012 04:04 |
| Eclipse User |
|
|
|
Hello Eike,
Thank you for the reply! I'm using DBStore with MySQL configured as the database backend. InnoDB's buffer pool was sized to keep the entire data set in memory (~6GB), the CDO server heap is also set to 6GB.
The changes are expected to be almost entirely (100-30k) new revisions, with a smaller number of deltas (50-100) and a similar number of detachments (50-100).
The actual LoadMergeDataRequest is completed in the expected timeframe, on the order of a few minutes. In an embedded setting, change set processing in application code finishes not long after that; what I'm looking for is a way to accelerate loading of related, but unchanged revisions, as this is what seems to take much more time when clients connect over the network (an additional 15-20 minutes).
I was indeed referring to using org.eclipse.emf.cdo.common.util.CDOFetchRule, but now I see that this would require either having a rule manager on the server, or a magic ball which tells the client what CDOIDs are expected to be changed in advance. 
I'll start experimenting with the buffer provider factory and see what I can find.
Regards,
András
|
|
|
Powered by
FUDForum. Page generated in 0.03912 seconds
Home