Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Add content of XMI file to existing CDO resource - too slow(CDO too slow?)
[CDO] Add content of XMI file to existing CDO resource - too slow [message #1392611] Wed, 02 July 2014 15:30 Go to next message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
Hello,

I am trying to add content from several XMI files into one CDOResource and store the result in MySQL (or HSQLDB). Each XMI file is imported separately. Here is the code I have (similar to the approach suggested by Eike here):

CDOTransaction transaction = CDOUtil.beginTransaction();
		
CDOResource cdoResource = transaction.getOrCreateResource("/" + resourceId);
		
Resource resourceToImport = new XMIResourceImpl();
resourceToImport.load(is, Collections.EMPTY_MAP);

EcoreUtil.Copier copier = new EcoreUtil.Copier();
Collection<EObject> copiedObjects = copier.copyAll(resourceToImport.getContents());
			
cdoResource.getContents().addAll(copiedObjects);

copier.copyReferences();
			
transaction.commit();


The code above works fine with small XMI files and I am able to add content from several XMI files into one CDO resource. However when I tried to add XMI file of about 8 Mb size with about 30000 object in it, the code above worked very slow. Particularly, this line:

cdoResource.getContents().addAll(copiedObjects);


It took about 40 minutes to add all copied objects to cdoResource. One of the things I noticed is that when the addAll is executed, catalina.out file grew in size from basically 0 to about 60 Gb.

Here are several lines from catalina.out file:

http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID: oid2
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting branchPoint Trace@oid2v0: BranchPoint[Branch[id=0, name=MAIN], *]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering Trace@oid2[PREPARED]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl] Registering new object Trace@oid2[PREPARED]
http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine] Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT] (data=Pair[Transaction 1, [TraceLine[TRANSIENT]]])
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID: oid3
http-bio-8180-exec-9 [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting branchPoint TraceLine@oid3v0: BranchPoint[Branch[id=0, name=MAIN], *]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering TraceLine@oid3[PREPARED]
http-bio-8180-exec-9 [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl] Registering new object TraceLine@oid3[PREPARED]
http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine] Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT] (data=Pair[Transaction 1, [TraceLine@oid3[PREPARED], TraceLine[TRANSIENT]]])


Note the data field in the FinalStateMachine log records. It grows with each added object.

So I guess my quests are:
1) Am I doing the addition of XMI content to existing CDOResource correctly? Is there a more efficient way to do that?
2) Is the huge log file the issue that makes addAll so slow? Can I configure CDO client not to log that much?
3) Do you see anything weird in the log records? Maybe I need to try to configure CDO to be more efficient? If so, how?

Thank you in advance.
Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392646 is a reply to message #1392611] Wed, 02 July 2014 16:41 Go to previous messageGo to next message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
I found here that Eike suggests to:
"Don't forget to disable console tracing in production. It consumes
between 30% and 50% of the overall load!"

How exactly can I disable logging in my client?
Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392649 is a reply to message #1392646] Wed, 02 July 2014 16:45 Go to previous messageGo to next message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
Seems like I found a place to configure logging. I found following the lines when I initialize sessionConfiguraiton:

 
OMPlatform.INSTANCE.setDebugging(true);
OMPlatform.INSTANCE.addLogHandler(PrintLogHandler.CONSOLE);
OMPlatform.INSTANCE.addTraceHandler(PrintTraceHandler.CONSOLE);


Hopefully that will help.

I still curious though. Even if disabling loggin will help to increase performance by 50%, then still my 8Mb XMI fill will take like 20 minutes to be process. That is still too long and in the future I plan to have huge XMI files.
Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392656 is a reply to message #1392611] Wed, 02 July 2014 16:55 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 30352
Registered: July 2009
Senior Member
Evgeny,

Comments below.

On 02/07/2014 5:30 PM, Evgeny Karataev wrote:
> Hello,
>
> I am trying to add content from several XMI files into one CDOResource
> and store the result in MySQL (or HSQLDB). Each XMI file is imported
> separately. Here is the code I have (similar to the approach suggested
> by Eike
> https://www.eclipse.org/forums/index.php/mv/msg/202347/646845/#msg_646845):
>
>
> CDOTransaction transaction = CDOUtil.beginTransaction();
>
> CDOResource cdoResource = transaction.getOrCreateResource("/" +
> resourceId);
>
> Resource resourceToImport = new XMIResourceImpl();
> resourceToImport.load(is, Collections.EMPTY_MAP);
>
> EcoreUtil.Copier copier = new EcoreUtil.Copier();
I'm not CDO expert but I don't see the point in copying objects you just
loaded.
> Collection<EObject> copiedObjects =
> copier.copyAll(resourceToImport.getContents());
>
> cdoResource.getContents().addAll(copiedObjects);
>
> copier.copyReferences();
I'm also not sure it's a good idea to copy the references after
attaching it to the cdoResource. It's probably better to do it before.
And as I said, I don't see the point of a copy.
>
> transaction.commit();
>
>
> The code above works fine with small XMI files and I am able to add
> content from several XMI files into one CDO resource. However when I
> tried to add XMI file of about 8 Mb size with about 30000 object in
> it, the code above worked very slow. Particularly, this line:
>
> cdoResource.getContents().addAll(copiedObjects);
>
> It took about 40 minutes to add all copied objects to cdoResource. One
> of the things I noticed is that when the addAll is executed,
> catalina.out file grew in size from basically 0 to about 60 Gb.
> Here are several lines from catalina.out file:
>
>
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID:
> oid2
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting
> branchPoint Trace@oid2v0: BranchPoint[Branch[id=0, name=MAIN], *]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering
> Trace@oid2[PREPARED]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl]
> Registering new object Trace@oid2[PREPARED]
> http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine]
> Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT]
> (data=Pair[Transaction 1, [TraceLine[TRANSIENT]]])
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting ID:
> oid3
> http-bio-8180-exec-9
> [org.eclipse.emf.cdo.spi.common.revision.BaseCDORevision] Setting
> branchPoint TraceLine@oid3v0: BranchPoint[Branch[id=0, name=MAIN], *]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.view.AbstractCDOView] Registering
> TraceLine@oid3[PREPARED]
> http-bio-8180-exec-9
> [org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl]
> Registering new object TraceLine@oid3[PREPARED]
> http-bio-8180-exec-9 [org.eclipse.net4j.util.fsm.FiniteStateMachine]
> Processing event PREPARE in state TRANSIENT for TraceLine[TRANSIENT]
> (data=Pair[Transaction 1, [TraceLine@oid3[PREPARED],
> TraceLine[TRANSIENT]]])
>
>
> Note the data field in the FinalStateMachine log records. It grows
> with each added object.
>
> So I guess my quests are:
> 1) Am I doing the addition of XMI content to existing CDOResource
> correctly? Is there a more efficient way to do that?
> 2) Is the huge log file the issue that makes addAll so slow? Can I
> configure CDO client not to log that much?
> 3) Do you see anything weird in the log records? Maybe I need to try
> to configure CDO to be more efficient? If so, how?
>
> Thank you in advance.
Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392672 is a reply to message #1392649] Wed, 02 July 2014 17:26 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6441
Registered: July 2009
Senior Member
Am 02.07.2014 18:45, schrieb Evgeny Karataev:
> Seems like I found a place to configure logging. I found following the lines when I initialize sessionConfiguraiton:
>
>
> OMPlatform.INSTANCE.setDebugging(true);
> OMPlatform.INSTANCE.addLogHandler(PrintLogHandler.CONSOLE);
> OMPlatform.INSTANCE.addTraceHandler(PrintTraceHandler.CONSOLE);
>
>
> Hopefully that will help.
>
> I still curious though. Even if disabling loggin will help to increase performance by 50%, then still my 8Mb XMI fill
> will take like 20 minutes to be process. That is still too long and in the future I plan to have huge XMI files.
So, how long does it actually take without tracing enabled?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392747 is a reply to message #1392672] Wed, 02 July 2014 19:58 Go to previous messageGo to next message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
Eike, disabling tracing reduced the time significantly. Right now the code that I showed above finishes in about 40 seconds(it used to take like 30 minutes).

Now this line cdoResource.getContents().addAll(copiedObjects); actually executes fast and most of the time is spend on the transaction.commit();.

Even though it is much faster now, the XMI file is pretty small - only 8 Mb. I am not sure how long it will take to import for example XMI file of about 200 Mb size (I have not tried it yet).

Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1392752 is a reply to message #1392747] Wed, 02 July 2014 20:05 Go to previous messageGo to next message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
Ed,

Thank you for your comments. At first I actually didn't use EcoreUtil.Copier. I simply did following:

CDOTransaction transaction = CDOUtil.beginTransaction();
		
CDOResource cdoResource = transaction.getOrCreateResource("/" + resourceId);
		
Resource resourceToImport = new XMIResourceImpl();
resourceToImport.load(is, Collections.EMPTY_MAP);

cdoResource.getContents().addAll(resourceToImport.getContents());
			
transaction.commit();



But it was too slow, so I started to look online how to add content from one resource to existing CDO resource and I found the example as I mentioned above.

It seems like it doesn't make much difference in speed whether I am using copier or not. Disabling tracing increased the performance, but still it takes like 40 seconds to process 8 Mg XMI file.

I try to see how long it takes on larger file (e.g. 200 Mb) and report here.
Re: [CDO] Add content of XMI file to existing CDO resource - too slow [message #1403250 is a reply to message #1392611] Tue, 22 July 2014 15:59 Go to previous message
Evgeny Karataev is currently offline Evgeny KarataevFriend
Messages: 18
Registered: May 2014
Junior Member
I have tried to add content of a file of about 200 Mb to existing CDO repository. This line took about 15 minutes to execute:
cdoResource.getContents().addAll(copiedObjects);


copiedObjects collection had 1177353 items.

How can it be speed up?
Previous Topic:[cdo] Containment always has 0 as its cdo_resource
Next Topic:[CDO] How to merge 2 repos that have diverged
Goto Forum:
  


Current Time: Mon Aug 19 05:27:30 GMT 2019

Powered by FUDForum. Page generated in 0.02114 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top