Eclipse Community Forums: EMF » [CDO] Add content of XMI file to existing CDO resource

Help

Home

Home » Modeling » EMF » [CDO] Add content of XMI file to existing CDO resource - too slow(CDO too slow?)

Show: Today's Messages :: Show Polls :: Message Navigator

[CDO] Add content of XMI file to existing CDO resource - too slow [message #1392611]

Wed, 02 July 2014 15:30

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

Hello,

I am trying to add content from several XMI files into one CDOResource and store the result in MySQL (or HSQLDB). Each XMI file is imported separately. Here is the code I have (similar to the approach suggested by Eike here):

CDOTransaction transaction = CDOUtil.beginTransaction();
		
CDOResource cdoResource = transaction.getOrCreateResource("/" + resourceId);
		
Resource resourceToImport = new XMIResourceImpl();
resourceToImport.load(is, Collections.EMPTY_MAP);

EcoreUtil.Copier copier = new EcoreUtil.Copier();
Collection<EObject> copiedObjects = copier.copyAll(resourceToImport.getContents());
			
cdoResource.getContents().addAll(copiedObjects);

copier.copyReferences();
			
transaction.commit();

The code above works fine with small XMI files and I am able to add content from several XMI files into one CDO resource. However when I tried to add XMI file of about 8 Mb size with about 30000 object in it, the code above worked very slow. Particularly, this line:

cdoResource.getContents().addAll(copiedObjects);

It took about 40 minutes to add all copied objects to cdoResource. One of the things I noticed is that when the addAll is executed, catalina.out file grew in size from basically 0 to about 60 Gb.

Here are several lines from catalina.out file:

http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID: oid2
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting branchPoint Trace@oid2v0: BranchPoint[Branch[id=0, name=MAIN], *]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering Trace@oid2[PREPARED]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl] Registering new object Trace@oid2[PREPARED]
http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine] Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT] (data=Pair[Transaction 1, [TraceLine[TRANSIENT]]])
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID: oid3
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting branchPoint TraceLine@oid3v0: BranchPoint[Branch[id=0, name=MAIN], *]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering TraceLine@oid3[PREPARED]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl] Registering new object TraceLine@oid3[PREPARED]
http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine] Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT] (data=Pair[Transaction 1, [TraceLine@oid3[PREPARED], TraceLine[TRANSIENT]]])

Note the data field in the FinalStateMachine log records. It grows with each added object.

So I guess my quests are:
1) Am I doing the addition of XMI content to existing CDOResource correctly? Is there a more efficient way to do that?
2) Is the huge log file the issue that makes addAll so slow? Can I configure CDO client not to log that much?
3) Do you see anything weird in the log records? Maybe I need to try to configure CDO to be more efficient? If so, how?

Thank you in advance.

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392646 is a reply to message #1392611]

Wed, 02 July 2014 16:41

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

I found here that Eike suggests to:
"Don't forget to disable console tracing in production. It consumes
between 30% and 50% of the overall load!"

How exactly can I disable logging in my client?

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392649 is a reply to message #1392646]

Wed, 02 July 2014 16:45

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

Seems like I found a place to configure logging. I found following the lines when I initialize sessionConfiguraiton:

 
OMPlatform.INSTANCE.setDebugging(true);
OMPlatform.INSTANCE.addLogHandler(PrintLogHandler.CONSOLE);
OMPlatform.INSTANCE.addTraceHandler(PrintTraceHandler.CONSOLE);

Hopefully that will help.

I still curious though. Even if disabling loggin will help to increase performance by 50%, then still my 8Mb XMI fill will take like 20 minutes to be process. That is still too long and in the future I plan to have huge XMI files.

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392656 is a reply to message #1392611]

Wed, 02 July 2014 16:55

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Evgeny,

Comments below.

On 02/07/2014 5:30 PM, Evgeny Karataev wrote:
> Hello,
>
> I am trying to add content from several XMI files into one CDOResource
> and store the result in MySQL (or HSQLDB). Each XMI file is imported
> separately. Here is the code I have (similar to the approach suggested
> by Eike
> https://www.eclipse.org/forums/index.php/mv/msg/202347/646845/#msg_646845):
>
>
> CDOTransaction transaction = CDOUtil.beginTransaction();
>
> CDOResource cdoResource = transaction.getOrCreateResource("/" +
> resourceId);
>
> Resource resourceToImport = new XMIResourceImpl();
> resourceToImport.load(is, Collections.EMPTY_MAP);
>
> EcoreUtil.Copier copier = new EcoreUtil.Copier();
I'm not CDO expert but I don't see the point in copying objects you just
loaded.
> Collection<EObject> copiedObjects =
> copier.copyAll(resourceToImport.getContents());
>
> cdoResource.getContents().addAll(copiedObjects);
>
> copier.copyReferences();
I'm also not sure it's a good idea to copy the references after
attaching it to the cdoResource. It's probably better to do it before.
And as I said, I don't see the point of a copy.
>
> transaction.commit();
>
>
> The code above works fine with small XMI files and I am able to add
> content from several XMI files into one CDO resource. However when I
> tried to add XMI file of about 8 Mb size with about 30000 object in
> it, the code above worked very slow. Particularly, this line:
>
> cdoResource.getContents().addAll(copiedObjects);
>
> It took about 40 minutes to add all copied objects to cdoResource. One
> of the things I noticed is that when the addAll is executed,
> catalina.out file grew in size from basically 0 to about 60 Gb.
> Here are several lines from catalina.out file:
>
>
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID:
> oid2
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting
> branchPoint Trace@oid2v0: BranchPoint[Branch[id=0, name=MAIN], *]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering
> Trace@oid2[PREPARED]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl]
> Registering new object Trace@oid2[PREPARED]
> http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine]
> Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT]
> (data=Pair[Transaction 1, [TraceLine[TRANSIENT]]])
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID:
> oid3
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting
> branchPoint TraceLine@oid3v0: BranchPoint[Branch[id=0, name=MAIN], *]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering
> TraceLine@oid3[PREPARED]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl]
> Registering new object TraceLine@oid3[PREPARED]
> http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine]
> Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT]
> (data=Pair[Transaction 1, [TraceLine@oid3[PREPARED],
> TraceLine[TRANSIENT]]])
>
>
> Note the data field in the FinalStateMachine log records. It grows
> with each added object.
>
> So I guess my quests are:
> 1) Am I doing the addition of XMI content to existing CDOResource
> correctly? Is there a more efficient way to do that?
> 2) Is the huge log file the issue that makes addAll so slow? Can I
> configure CDO client not to log that much?
> 3) Do you see anything weird in the log records? Maybe I need to try
> to configure CDO to be more efficient? If so, how?
>
> Thank you in advance.

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392672 is a reply to message #1392649]

Wed, 02 July 2014 17:26

Eike Stepper

Messages: 6682
Registered: July 2009

Senior Member

Am 02.07.2014 18:45, schrieb Evgeny Karataev:
> Seems like I found a place to configure logging. I found following the lines when I initialize sessionConfiguraiton:
>
>
> OMPlatform.INSTANCE.setDebugging(true);
> OMPlatform.INSTANCE.addLogHandler(PrintLogHandler.CONSOLE);
> OMPlatform.INSTANCE.addTraceHandler(PrintTraceHandler.CONSOLE);
>
>
> Hopefully that will help.
>
> I still curious though. Even if disabling loggin will help to increase performance by 50%, then still my 8Mb XMI fill
> will take like 20 minutes to be process. That is still too long and in the future I plan to have huge XMI files.
So, how long does it actually take without tracing enabled?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392747 is a reply to message #1392672]

Wed, 02 July 2014 19:58

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

Eike, disabling tracing reduced the time significantly. Right now the code that I showed above finishes in about 40 seconds(it used to take like 30 minutes).

Now this line cdoResource.getContents().addAll(copiedObjects); actually executes fast and most of the time is spend on the transaction.commit();.

Even though it is much faster now, the XMI file is pretty small - only 8 Mb. I am not sure how long it will take to import for example XMI file of about 200 Mb size (I have not tried it yet).

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392752 is a reply to message #1392747]

Wed, 02 July 2014 20:05

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

Ed,

Thank you for your comments. At first I actually didn't use EcoreUtil.Copier. I simply did following:


CDOTransaction transaction = CDOUtil.beginTransaction();
		
CDOResource cdoResource = transaction.getOrCreateResource("/" + resourceId);
		
Resource resourceToImport = new XMIResourceImpl();
resourceToImport.load(is, Collections.EMPTY_MAP);

cdoResource.getContents().addAll(resourceToImport.getContents());
			
transaction.commit();

But it was too slow, so I started to look online how to add content from one resource to existing CDO resource and I found the example as I mentioned above.

It seems like it doesn't make much difference in speed whether I am using copier or not. Disabling tracing increased the performance, but still it takes like 40 seconds to process 8 Mg XMI file.

I try to see how long it takes on larger file (e.g. 200 Mb) and report here.

Report message to a moderator

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1403250 is a reply to message #1392611]

Tue, 22 July 2014 15:59

Evgeny Karataev

Messages: 18
Registered: May 2014

Junior Member

I have tried to add content of a file of about 200 Mb to existing CDO repository. This line took about 15 minutes to execute:

cdoResource.getContents().addAll(copiedObjects);

copiedObjects collection had 1177353 items.

How can it be speed up?

Report message to a moderator

Previous Topic:	[cdo] Containment always has 0 as its cdo_resource
Next Topic:	[CDO] How to merge 2 repos that have diverged

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Wed Apr 24 18:24:04 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter