Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] scalability test problem
[CDO] scalability test problem [message #500491] Thu, 26 November 2009 11:13 Go to next message
Lothar Werzinger is currently offline Lothar WerzingerFriend
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
--nextPart21350895.kcvHbq28VH
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8Bit

Hi,

I am trying to do a scalability test where I try to add 500000 objects to a
CDO repository. The model I use is proprietary, but similar to the Library
example in the fact that the library has a feature containing all the books
in the library and my model has a similar feature where I try to add those
500000 objects.

The code that I use to add the objects opens a new session and transaction
for every 10000 objects and closes both transaction and session after the
commit. I did that to minimize the resource needs in both server and
client.

I also use a CollectionLoadingPolicy to avoid loading all the objects into
memory:

> CDOCollectionLoadingPolicy policy =
> CDOUtil.createCollectionLoadingPolicy(0, 100);
> session.options().setCollectionLoadingPolicy(policy);

However after ca. 200000 objects I get exceptions. The server and
application log is attached. I also attached the cdo server config file and
a snippet of the code where I create the objects.

I would appreciate any help to create a configuration that works with 500000
or more (millions) of objects.

P.S.
After I can successfully create such a large repository I am planing to
access it with queries to test the scalability accessing/modifying such a
large repository.

Lothar

--nextPart21350895.kcvHbq28VH
Content-Type: text/plain; name="addobjects.txt"
Content-Transfer-Encoding: 8Bit
Content-Disposition: attachment; filename="addobjects.txt"

try
{
// Prepare container
IManagedContainer container = ContainerUtil.createContainer();
Net4jUtil.prepareContainer(container); // Register Net4j factories
TCPUtil.prepareContainer(container); // Register TCP factories
CDONet4jUtil.prepareContainer(container); // Register CDO factories
container.activate();

// Create connector
IConnector connector = TCPUtil.getConnector(container, "localhost:2036"); //$NON-NLS-1$

CDOSessionConfiguration configuration = CDONet4jUtil.createSessionConfiguration();
configuration.setConnector(connector);
configuration.setRepositoryName("testapp");

long numiterations = 50;
for(long j = 1; j <= numiterations; ++j)
{
System.out.println("" + j + " of " + numiterations + " creating ...");

// Open session
CDOSession session = configuration.openSession();

//chunk load large ELists
CDOCollectionLoadingPolicy policy = CDOUtil.createCollectionLoadingPolicy(0, 100);
session.options().setCollectionLoadingPolicy(policy);

CDOTransaction transaction = session.openTransaction();

// add 10000 objects

System.out.println("" + j + " of " + numiterations + " committing ...");
transaction.commit();
transaction.close();
session.close();
System.out.println("" + j + " of " + numiterations + " committing done");
}
}
catch(Throwable t)
{
t.printStackTrace();
}


--nextPart21350895.kcvHbq28VH
Content-Type: text/plain; name="app.log"
Content-Transfer-Encoding: 8Bit
Content-Disposition: attachment; filename="app.log"

calcName(209997): vqyl
calcName(209998): wqyl
calcName(209999): xqyl
j=20of 50 committing ...
[ERROR] EOFException
java.io.EOFException
at java.io.DataInputStream.readBoolean(DataInputStream.java:227 )
at org.eclipse.net4j.util.io.ExtendedDataInput$Delegating.readB oolean(ExtendedDataInput.java:55)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirmingCheckError(CommitTransactionRequest.java: 229)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:162)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:120)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:1)
at org.eclipse.net4j.signal.RequestWithMonitoring.confirming(Re questWithMonitoring.java:169)
at org.eclipse.net4j.signal.RequestWithConfirmation.doExtendedI nput(RequestWithConfirmation.java:123)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.RequestWithConfirmation.doExecute(R equestWithConfirmation.java:103)
at org.eclipse.net4j.signal.RequestWithMonitoring.doExecute(Req uestWithMonitoring.java:233)
at org.eclipse.net4j.signal.SignalActor.execute(SignalActor.jav a:66)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.SignalProtocol.startSignal(SignalPr otocol.java:423)
at org.eclipse.net4j.signal.RequestWithConfirmation.doSend(Requ estWithConfirmation.java:87)
at org.eclipse.net4j.signal.RequestWithConfirmation.send(Reques tWithConfirmation.java:73)
at org.eclipse.net4j.signal.RequestWithMonitoring.send(RequestW ithMonitoring.java:108)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.send(CDOClientProtocol.java:302)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.commitTransaction(CDOClientProtocol.java:220)
at org.eclipse.emf.internal.cdo.transaction.CDOSingleTransactio nStrategyImpl.commit(CDOSingleTransactionStrategyImpl.java:5 8)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:598)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:612)
at testapp.perspective.handlers.CreateEntitiesHandler.execute(C reateEntitiesHandler.java:138)
at org.eclipse.ui.internal.handlers.HandlerProxy.execute(Handle rProxy.java:294)
at org.eclipse.core.commands.Command.executeWithChecks(Command. java:476)
at org.eclipse.core.commands.ParameterizedCommand.executeWithCh ecks(ParameterizedCommand.java:508)
at org.eclipse.ui.internal.handlers.HandlerService.executeComma nd(HandlerService.java:169)
at org.eclipse.ui.internal.handlers.SlaveHandlerService.execute Command(SlaveHandlerService.java:241)
at org.eclipse.ui.menus.CommandContributionItem.handleWidgetSel ection(CommandContributionItem.java:770)
at org.eclipse.ui.menus.CommandContributionItem.access$10(Comma ndContributionItem.java:756)
at org.eclipse.ui.menus.CommandContributionItem$5.handleEvent(C ommandContributionItem.java:746)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java :84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1176)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.ja va:3493)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java :3112)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.jav a:2405)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2369)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:22 21)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:500)
at org.eclipse.core.databinding.observable.Realm.runWithDefault (Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Work bench.java:493)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.j ava:149)
at testapp.Application.start(Application.java:20)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:194)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:368)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
at org.eclipse.equinox.launcher.Main.main(Main.java:1287)
!SESSION 2009-11-26 01:42:17.864 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.6.0_01
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -application TestApp.common.application
Command-line arguments: -application TestApp.common.application -data /workspaces/rcp-3.5/../runtime-TestApp.common.application -dev file:/workspaces/rcp-3.5/.metadata/.plugins/org.eclipse.pde. core/TestApp.rcp/dev.properties -os linux -ws gtk -arch x86_64 -consolelog -console

!ENTRY org.eclipse.net4j 4 0 2009-11-26 02:30:19.963
!MESSAGE EOFException
!STACK 0
java.io.EOFException
at java.io.DataInputStream.readBoolean(DataInputStream.java:227 )
at org.eclipse.net4j.util.io.ExtendedDataInput$Delegating.readB oolean(ExtendedDataInput.java:55)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirmingCheckError(CommitTransactionRequest.java: 229)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:162)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:120)
at org.eclipse.emf.internal.cdo.net4j.protocol.CommitTransactio nRequest.confirming(CommitTransactionRequest.java:1)
at org.eclipse.net4j.signal.RequestWithMonitoring.confirming(Re questWithMonitoring.java:169)
at org.eclipse.net4j.signal.RequestWithConfirmation.doExtendedI nput(RequestWithConfirmation.java:123)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.RequestWithConfirmation.doExecute(R equestWithConfirmation.java:103)
at org.eclipse.net4j.signal.RequestWithMonitoring.doExecute(Req uestWithMonitoring.java:233)
at org.eclipse.net4j.signal.SignalActor.execute(SignalActor.jav a:66)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.SignalProtocol.startSignal(SignalPr otocol.java:423)
at org.eclipse.net4j.signal.RequestWithConfirmation.doSend(Requ estWithConfirmation.java:87)
at org.eclipse.net4j.signal.RequestWithConfirmation.send(Reques tWithConfirmation.java:73)
at org.eclipse.net4j.signal.RequestWithMonitoring.send(RequestW ithMonitoring.java:108)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.send(CDOClientProtocol.java:302)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.commitTransaction(CDOClientProtocol.java:220)
at org.eclipse.emf.internal.cdo.transaction.CDOSingleTransactio nStrategyImpl.commit(CDOSingleTransactionStrategyImpl.java:5 8)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:598)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:612)
at testapp.perspective.handlers.CreateEntitiesHandler.execute(C reateEntitiesHandler.java:138)
at org.eclipse.ui.internal.handlers.HandlerProxy.execute(Handle rProxy.java:294)
at org.eclipse.core.commands.Command.executeWithChecks(Command. java:476)
at org.eclipse.core.commands.ParameterizedCommand.executeWithCh ecks(ParameterizedCommand.java:508)
at org.eclipse.ui.internal.handlers.HandlerService.executeComma nd(HandlerService.java:169)
at org.eclipse.ui.internal.handlers.SlaveHandlerService.execute Command(SlaveHandlerService.java:241)
at org.eclipse.ui.menus.CommandContributionItem.handleWidgetSel ection(CommandContributionItem.java:770)
at org.eclipse.ui.menus.CommandContributionItem.access$10(Comma ndContributionItem.java:756)
at org.eclipse.ui.menus.CommandContributionItem$5.handleEvent(C ommandContributionItem.java:746)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java :84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1176)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.ja va:3493)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java :3112)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.jav a:2405)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2369)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:22 21)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:500)
at org.eclipse.core.databinding.observable.Realm.runWithDefault (Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Work bench.java:493)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.j ava:149)
at testapp.Application.start(Application.java:20)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:194)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:368)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
at org.eclipse.equinox.launcher.Main.main(Main.java:1287)
org.eclipse.net4j.util.transaction.TransactionException: org.eclipse.emf.cdo.common.util.TransportException: java.util.concurrent.TimeoutException: Timeout
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:606)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:612)
at testapp.perspective.handlers.CreateEntitiesHandler.execute(C reateEntitiesHandler.java:138)
at org.eclipse.ui.internal.handlers.HandlerProxy.execute(Handle rProxy.java:294)
at org.eclipse.core.commands.Command.executeWithChecks(Command. java:476)
at org.eclipse.core.commands.ParameterizedCommand.executeWithCh ecks(ParameterizedCommand.java:508)
at org.eclipse.ui.internal.handlers.HandlerService.executeComma nd(HandlerService.java:169)
at org.eclipse.ui.internal.handlers.SlaveHandlerService.execute Command(SlaveHandlerService.java:241)
at org.eclipse.ui.menus.CommandContributionItem.handleWidgetSel ection(CommandContributionItem.java:770)
at org.eclipse.ui.menus.CommandContributionItem.access$10(Comma ndContributionItem.java:756)
at org.eclipse.ui.menus.CommandContributionItem$5.handleEvent(C ommandContributionItem.java:746)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java :84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1176)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.ja va:3493)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java :3112)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.jav a:2405)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2369)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:22 21)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:500)
at org.eclipse.core.databinding.observable.Realm.runWithDefault (Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Work bench.java:493)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.j ava:149)
at testapp.Application.start(Application.java:20)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:194)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:368)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
at org.eclipse.equinox.launcher.Main.main(Main.java:1287)
Caused by: org.eclipse.emf.cdo.common.util.TransportException: java.util.concurrent.TimeoutException: Timeout
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.send(CDOClientProtocol.java:310)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.commitTransaction(CDOClientProtocol.java:220)
at org.eclipse.emf.internal.cdo.transaction.CDOSingleTransactio nStrategyImpl.commit(CDOSingleTransactionStrategyImpl.java:5 8)
at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl. commit(CDOTransactionImpl.java:598)
... 35 more
Caused by: java.util.concurrent.TimeoutException: Timeout
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:243)
at org.eclipse.net4j.signal.SignalProtocol.startSignal(SignalPr otocol.java:423)
at org.eclipse.net4j.signal.RequestWithConfirmation.doSend(Requ estWithConfirmation.java:87)
at org.eclipse.net4j.signal.RequestWithConfirmation.send(Reques tWithConfirmation.java:73)
at org.eclipse.net4j.signal.RequestWithMonitoring.send(RequestW ithMonitoring.java:108)
at org.eclipse.emf.internal.cdo.net4j.protocol.CDOClientProtoco l.send(CDOClientProtocol.java:302)
... 38 more
[ERROR] [Ljava.lang.StackTraceElement;
java.lang.ClassNotFoundException: [Ljava.lang.StackTraceElement;
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInter nal(BundleLoader.java:494)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Bund leLoader.java:410)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Bund leLoader.java:398)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loa dClass(DefaultClassLoader.java:105)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.eclipse.net4j.util.io.ExtendedIOUtil$ClassLoaderClassRes olver.resolveClass(ExtendedIOUtil.java:231)
at org.eclipse.net4j.util.io.ExtendedIOUtil$3.resolveClass(Exte ndedIOUtil.java:163)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream .java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.ja va:1496)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1 624)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java :1323)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea m.java:1945)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.j ava:1869)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre am.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java :1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java: 351)
at org.eclipse.net4j.util.io.ExtendedIOUtil.readObject(Extended IOUtil.java:177)
at org.eclipse.net4j.util.io.ExtendedIOUtil.readObject(Extended IOUtil.java:131)
at org.eclipse.net4j.util.io.ExtendedDataInputStream.readObject (ExtendedDataInputStream.java:46)
at org.eclipse.net4j.signal.RemoteExceptionIndication.indicatin g(RemoteExceptionIndication.java:46)
at org.eclipse.net4j.signal.Indication.doExtendedInput(Indicati on.java:55)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.Indication.execute(Indication.java: 49)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.Signal.run(Signal.java:147)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

!ENTRY org.eclipse.net4j.util 4 0 2009-11-26 02:35:22.578
!MESSAGE [Ljava.lang.StackTraceElement;
!STACK 0
java.lang.ClassNotFoundException: [Ljava.lang.StackTraceElement;
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInter nal(BundleLoader.java:494)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Bund leLoader.java:410)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Bund leLoader.java:398)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loa dClass(DefaultClassLoader.java:105)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.eclipse.net4j.util.io.ExtendedIOUtil$ClassLoaderClassRes olver.resolveClass(ExtendedIOUtil.java:231)
at org.eclipse.net4j.util.io.ExtendedIOUtil$3.resolveClass(Exte ndedIOUtil.java:163)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream .java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.ja va:1496)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1 624)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java :1323)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea m.java:1945)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.j ava:1869)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre am.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java :1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java: 351)
at org.eclipse.net4j.util.io.ExtendedIOUtil.readObject(Extended IOUtil.java:177)
at org.eclipse.net4j.util.io.ExtendedIOUtil.readObject(Extended IOUtil.java:131)
at org.eclipse.net4j.util.io.ExtendedDataInputStream.readObject (ExtendedDataInputStream.java:46)
at org.eclipse.net4j.signal.RemoteExceptionIndication.indicatin g(RemoteExceptionIndication.java:46)
at org.eclipse.net4j.signal.Indication.doExtendedInput(Indicati on.java:55)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.Indication.execute(Indication.java: 49)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.Signal.run(Signal.java:147)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)


--nextPart21350895.kcvHbq28VH
Content-Type: text/plain; name="cdo-server.xml"
Content-Transfer-Encoding: 8Bit
Content-Disposition: attachment; filename="cdo-server.xml"

<?xml version="1.0" encoding="UTF-8"?>
<cdoServer>

<acceptor type="tcp" listenAddr="0.0.0.0" port="2036">
<!--
<negotiator type="challenge" description="/temp/users.db"/>
-->
</acceptor>

<repository name="testapp">
<property name="overrideUUID" value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name="supportingAudits" value="true"/>
<property name="supportingRevisionDeltas" value="true"/>
<property name="verifyingRevisions" value="false"/>
<!--
<property name="currentLRUCapacity" value="10000"/>
<property name="revisedLRUCapacity" value="100"/>
-->
<property name="currentLRUCapacity" value="0"/>
<property name="revisedLRUCapacity" value="0"/>

<store type="db">
<!-- type: horizontal | vertical | <any user-contributed type>-->
<mappingStrategy type="horizontal">
<!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS | ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES-->
<property name="toOneReferences" value="LIKE_ATTRIBUTES"/>

<!-- MODEL | STRATEGY-->
<property name="mappingPrecedence" value="MODEL"/>
</mappingStrategy>

<dbAdapter name="mysql"/>
<dataSource class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource"
url="jdbc:mysql://localhost/testapp"
user="cdo"
password="cdo"/>

<!--<dbAdapter name="derby-embedded"/>
<dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/temp/cdodb1"
createDatabase="create"/>-->

<!--<dbAdapter name="hsqldb"/>
<dataSource class="org.eclipse.net4j.db.hsqldb.HSQLDBDataSource"
database="jdbc:hsqldb:mem:cdodb1"
user="sa"/>-->

<!--<dbAdapter name="mysql"/>
<dataSource class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource"
url="jdbc:mysql://localhost/cdodb1"
user="sa"/>-->

<!--<dbAdapter name="postgresql"/>
<dataSource class="org.postgresql.ds.PGSimpleDataSource"
url="jdbc:postgresql://localhost:5432/cdo"
databaseName="cdo"
user="cdo"
password="cdo"/>-->
</store>
</repository>

</cdoServer>


--nextPart21350895.kcvHbq28VH
Content-Type: text/plain; name="server.log"
Content-Transfer-Encoding: 8Bit
Content-Disposition: attachment; filename="server.log"


osgi> [INFO] CDO Server starting
!SESSION 2009-11-26 01:42:10.896 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.6.0_01
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -product org.eclipse.emf.cdo.server.product
Command-line arguments: -product org.eclipse.emf.cdo.server.product -data /workspaces/rap-rcp-common-3.5/cdoserver -dev file:/workspaces/rcp-3.5/.metadata/.plugins/org.eclipse.pde. core/cdoserver/dev.properties -os linux -ws gtk -arch x86_64 -console -consolelog

!ENTRY org.eclipse.emf.cdo.server 1 0 2009-11-26 01:42:11.814
!MESSAGE CDO Server starting
[INFO] First start: Nov 26, 2009 1:42:12 AM

!ENTRY org.eclipse.emf.cdo.server.db 1 0 2009-11-26 01:42:12.295
!MESSAGE First start: Nov 26, 2009 1:42:12 AM
[INFO] No user manager configured. Users will not be authenticated

!ENTRY org.eclipse.emf.cdo.server 1 0 2009-11-26 01:42:12.297
!MESSAGE No user manager configured. Users will not be authenticated
[INFO] CDO Server started

!ENTRY org.eclipse.emf.cdo.server 1 0 2009-11-26 01:42:12.724
!MESSAGE CDO Server started
[ERROR] FATAL: Memory infrastructure corrupted after successful commit operation of the store
[ERROR] Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedWriter.<init>(BufferedWriter.java:87)
at java.io.BufferedWriter.<init>(BufferedWriter.java:70)
at org.eclipse.core.runtime.adaptor.EclipseLog.logForStream(Ecl ipseLog.java:463)
at org.eclipse.core.runtime.adaptor.EclipseLog.openFile(Eclipse Log.java:245)
at org.eclipse.core.runtime.adaptor.EclipseLog.log(EclipseLog.j ava:301)
at org.eclipse.core.internal.runtime.PlatformLogWriter.logging( PlatformLogWriter.java:35)
at org.eclipse.core.internal.runtime.RuntimeLog.log(RuntimeLog. java:95)
at org.eclipse.core.internal.runtime.InternalPlatform.log(Inter nalPlatform.java:711)
at org.eclipse.core.internal.runtime.Log.log(Log.java:56)
at org.eclipse.net4j.util.om.log.EclipseLoggingBridge.logged(Ec lipseLoggingBridge.java:41)
at org.eclipse.net4j.internal.util.bundle.AbstractPlatform.log( AbstractPlatform.java:222)
at org.eclipse.net4j.util.om.log.Logger.log(Logger.java:35)
at org.eclipse.net4j.util.om.log.Logger.log(Logger.java:60)
at org.eclipse.net4j.util.om.log.Logger.error(Logger.java:65)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext Impl.updateInfraStructure(TransactionCommitContextImpl.java: 569)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext Impl.commit(TransactionCommitContextImpl.java:287)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication$2.runLoop(CommitTransactionIndication.java:83)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication$2.runLoop(CommitTransactionIndication.java:1)
at org.eclipse.net4j.util.om.monitor.ProgressDistributor.run(Pr ogressDistributor.java:96)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicatingCommit(CommitTransactionIndication.ja va:325)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:198 )
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:140 )
at org.eclipse.net4j.signal.IndicationWithMonitoring.indicating (IndicationWithMonitoring.java:84)
at org.eclipse.net4j.signal.IndicationWithResponse.doExtendedIn put(IndicationWithResponse.java:90)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:63)
at org.eclipse.net4j.signal.IndicationWithMonitoring.execute(In dicationWithMonitoring.java:63)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.Signal.run(Signal.java:147)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "Thread-10" java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedWriter.<init>(BufferedWriter.java:87)
at java.io.BufferedWriter.<init>(BufferedWriter.java:70)
at org.eclipse.core.runtime.adaptor.EclipseLog.logForStream(Ecl ipseLog.java:463)
at org.eclipse.core.runtime.adaptor.EclipseLog.openFile(Eclipse Log.java:245)
at org.eclipse.core.runtime.adaptor.EclipseLog.log(EclipseLog.j ava:301)
at org.eclipse.core.internal.runtime.PlatformLogWriter.logging( PlatformLogWriter.java:35)
at org.eclipse.core.internal.runtime.RuntimeLog.log(RuntimeLog. java:95)
at org.eclipse.core.internal.runtime.InternalPlatform.log(Inter nalPlatform.java:711)
at org.eclipse.core.internal.runtime.Log.log(Log.java:56)
at org.eclipse.net4j.util.om.log.EclipseLoggingBridge.logged(Ec lipseLoggingBridge.java:41)
at org.eclipse.net4j.internal.util.bundle.AbstractPlatform.log( AbstractPlatform.java:222)
at org.eclipse.net4j.util.om.log.Logger.log(Logger.java:35)
at org.eclipse.net4j.util.om.log.Logger.log(Logger.java:91)
at org.eclipse.net4j.util.om.log.Logger.error(Logger.java:96)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext Impl.handleException(TransactionCommitContextImpl.java:305)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext Impl.commit(TransactionCommitContextImpl.java:295)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication$2.runLoop(CommitTransactionIndication.java:83)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication$2.runLoop(CommitTransactionIndication.java:1)
at org.eclipse.net4j.util.om.monitor.ProgressDistributor.run(Pr ogressDistributor.java:96)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicatingCommit(CommitTransactionIndication.ja va:325)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:198 )
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:140 )
at org.eclipse.net4j.signal.IndicationWithMonitoring.indicating (IndicationWithMonitoring.java:84)
at org.eclipse.net4j.signal.IndicationWithResponse.doExtendedIn put(IndicationWithResponse.java:90)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:312)
at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:63)
at org.eclipse.net4j.signal.IndicationWithMonitoring.execute(In dicationWithMonitoring.java:63)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:239)
at org.eclipse.net4j.signal.Signal.run(Signal.java:147)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)


--nextPart21350895.kcvHbq28VH--

Report message to a moderator

Re: [CDO] scalability test problem [message #500524 is a reply to message #500491] Thu, 26 November 2009 14:18 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar WerzingerFriend
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Lothar Werzinger wrote:

> However after ca. 200000 objects I get exceptions.

I killed and restarted the server and I again could add approx. 200000
objects before it became unresponsive. So it seems that it is not dependent
on the overall number of objects, but that something accumulates on the
server despite my efforts to use sessions and transactions for every 10000
objects and closing them.

I did queries against the now 400000 objects and did not perceive it slower
than a query on a small repository :-) Great! I will create some more
objects and then try editing some of them.

I am still worried about the fact that the server fails at the insertion of
the objects. In a real production environment the server could be running
for months working with countless sessions and transactions and if it leaks
resources somewhere that would be fatal.

Lothar

Report message to a moderator

Re: [CDO] scalability test problem [message #500568 is a reply to message #500491] Thu, 26 November 2009 19:12 Go to previous messageGo to next message
Victor Roldan Betancort is currently offline Victor Roldan BetancortFriend
Messages: 524
Registered: July 2009
Senior Member
Lothar,

some comments below:

> I also use a CollectionLoadingPolicy to avoid loading all the objects into
> memory:
>
>> CDOCollectionLoadingPolicy policy =
>> CDOUtil.createCollectionLoadingPolicy(0, 100);
>> session.options().setCollectionLoadingPolicy(policy);
>
> However after ca. 200000 objects I get exceptions. The server and
> application log is attached. I also attached the cdo server config file and
> a snippet of the code where I create the objects.


This problem is related to a timeout in the signaling protocol due to a
very long commit operation. If you take a deeper look in the stack
trace, you see:

org.eclipse.net4j.util.transaction.TransactionException:
org.eclipse.emf.cdo.common.util.TransportException:
java.util.concurrent.TimeoutException: Timeout

Yes, I know the EOFException is very misleading. Both I and Stefan went
through the same problem in the past as well :P There is a bug to
improve the exception message for it:

287812: [Net4j] Poor exception/message when a timeout occurs
https://bugs.eclipse.org/bugs/show_bug.cgi?id=287812

To fix your problem, this should do the trick:

((org.eclipse.emf.cdo.net4j.CDOSession)session).options().ge tProtocol().setTimeout(-1L);

There is also the commit timeout value:

((org.eclipse.emf.cdo.net4j.CDOSession)session).options().se tCommitTimeout(Integer.MAX_VALUE);

> I would appreciate any help to create a configuration that works with 500000
> or more (millions) of objects.

I would recommend you to read this:

http://wiki.eclipse.org/Tweaking_CDO_Performance

If you take a look there, it suggest an even bigger collection loading
policy. Also I would suggest to increase server cache memory, it
improves a lot the performance.

HTH,
Víctor.

Report message to a moderator

Re: [CDO] scalability test problem [message #500582 is a reply to message #500568] Thu, 26 November 2009 20:40 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar WerzingerFriend
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Víctor Roldán Betancort wrote:
some comments below:

> Lothar,
>
> some comments below:
>
> This problem is related to a timeout in the signaling protocol due to a
> very long commit operation. If you take a deeper look in the stack
> trace, you see:
>
> org.eclipse.net4j.util.transaction.TransactionException:
> org.eclipse.emf.cdo.common.util.TransportException:
> java.util.concurrent.TimeoutException: Timeout
>
> Yes, I know the EOFException is very misleading. Both I and Stefan went
> through the same problem in the past as well :P There is a bug to
> improve the exception message for it:
>
> 287812: [Net4j] Poor exception/message when a timeout occurs
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=287812
>
> To fix your problem, this should do the trick:
>
> ((org.eclipse.emf.cdo.net4j.CDOSession)session).options()
> .getProtocol().setTimeout(-1L);
>
> There is also the commit timeout value:
>
> ((org.eclipse.emf.cdo.net4j.CDOSession)session).options()
> .setCommitTimeout(Integer.MAX_VALUE);

Thanks, I will try to use these settings ASAP.

>> I would appreciate any help to create a configuration that works with
>> 500000 or more (millions) of objects.
>
> I would recommend you to read this:
>
> http://wiki.eclipse.org/Tweaking_CDO_Performance
>
> If you take a look there, it suggest an even bigger collection loading
> policy. Also I would suggest to increase server cache memory, it
> improves a lot the performance.

Actually I made all these parameters so small to make sure I don't get an
out of memory situation on the client/server. Performance is not what I was
trying to evaluate, but the scalability itself. Thanks for the pointers
anyway :-)

> HTH,
> Víctor.

Thanks!
Lothar

Report message to a moderator

Re: [CDO] scalability test problem [message #500589 is a reply to message #500582] Thu, 26 November 2009 21:46 Go to previous message
Victor Roldan Betancort is currently offline Victor Roldan BetancortFriend
Messages: 524
Registered: July 2009
Senior Member
Lothar,

>> If you take a look there, it suggest an even bigger collection loading
>> policy. Also I would suggest to increase server cache memory, it
>> improves a lot the performance.
>
> Actually I made all these parameters so small to make sure I don't get an
> out of memory situation on the client/server. Performance is not what I was
> trying to evaluate, but the scalability itself. Thanks for the pointers
> anyway :-)

Jeez... you are right, you were talking about scalability... Boy, I
shouldn't be witting to the newsgroup after 15 hours of work... :P

Don't hesitate asking at any time ;)

Cheers,
Víctor.

Report message to a moderator

Previous Topic:[CDO] Query on an audit view
Next Topic:[Teneo] Annotations required to support 2 associations between two classes
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Fri Apr 19 18:41:35 GMT 2024

Powered by FUDForum. Page generated in 0.03222 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top

Sign up to our Newsletter

A fresh new issue delivered monthly