Home » Modeling » EMF » [CDO] Question about LRU cache and tx.commit()
| [CDO] Question about LRU cache and tx.commit() [message #421330] |
Mon, 04 August 2008 17:08  |
 Cedric Brun
Messages: 431
Registered: July 2009 |
Senior Member |
|
|
Hi Eike and Simmon (or anybody familiar with CDO internals),
I'm still digging in CDO, and I wrote yet another dummy testcase just filling a model with thousands of instances using a fake backend which does not save anything at all (the addRevision do nothing so I have no references on the java objects). Of course I'm not even trying to browse the model on the repository ;)
My loop was something like this :
int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}
I was surprised to see that my test ended up with an OutOfMemory error
Digging with the *great* memory analyzer I've seen that all the revisions were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering, shouldn't this state be cleared after a commit ?
I changed my my test to create a new transaction and close it on each loop, but it did not helped as I guess the resolver is linked to the session instance.
So I'm wondering, when should the CDORevisionResolverImpl be cleared ? And more especially the "revisions" hashmap ?
Just in case you're wondering I'm using the ganymede CDO.
Thanks,
Cédric
http://cedric.brun.io news and articles on eclipse and eclipse modeling.
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421331 is a reply to message #421330] |
Mon, 04 August 2008 18:07  |
|
Hi Cédric,
Comments below...
Cédric Brun schrieb:
> Hi Eike and Simmon (or anybody familiar with CDO internals),
>
> I'm still digging in CDO, and I wrote yet another dummy testcase just filling a model with thousands of instances using a fake backend which does not save anything at all (the addRevision do nothing so I have no references on the java objects). Of course I'm not even trying to browse the model on the repository ;)
>
> My loop was something like this :
>
> int nb_loops = 10000;
> for (int i = 0; i < nb_loops; i++) {
> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
> newF.setName("" + i);
> root.getMembers().add(newF);
> tx.commit();
> }
>
> I was surprised to see that my test ended up with an OutOfMemory error
>
> Digging with the *great* memory analyzer I've seen that all the revisions were kept by CDO through the CDORevisionResolverImpl
> Setting the LRU cache to 0 did not helped and then I am wondering, shouldn't this state be cleared after a commit ?
>
Setting the LRU cache to 0 is a bad idea since it disables eviction
completely. I've added respective JavaDoc. If you feel that -1 is a
better value please open a Bugzilla.
Please try to set the capacity to a value greater than zero but small
enough that you don't run into OutOfMemory.
The revisedLRUCapacity can be much smaller than the currentLRUCapacity
if you don't use audit views.
> I changed my my test to create a new transaction and close it on each loop, but it did not helped as I guess the resolver is linked to the session instance.
>
> So I'm wondering, when should the CDORevisionResolverImpl be cleared ? And more especially the "revisions" hashmap ?
>
Why should it be cleared without need? It's a cache to prevent from
continuous server requests or database queries for missing revisions.
You might ask why I didn't choose a memory sensitive cache
implementation. This cache is the most prominent cache in the server
side repository. With a memory sensitive implementation it would not be
possible to influence which elements to select on eviction. In other
words, no eviction policy would be possible, thereby degrading the cache
hit rate.
Cheers
/Eike
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| |
| Re: [CDO] Question about LRU cache and tx.commit() [message #421333 is a reply to message #421332] |
Mon, 04 August 2008 19:45 |
|
This is a multi-part message in MIME format.
--------------050503070702040100020705
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit
Simon McDuff schrieb:
> Which value is your LRU set ? (Client and server)
>
> I know in my case I was able to load 51 millions objects...
> OutOfmemory are always tricky.
>
> Can you send us a complete test (or even better a testcase) ? We will know
> exactly your settings you are using.
>
> By the way, do not forget that you need to configure 2 cache.... client
> cache and the server cache.
> Are your client and server the same java process ?
Just a side note: Currently even if both client and server are running
in the same JVM process they would not share the same revision manager
instance.
Cheers
/Eike
> How much memory you JAVA
> can use ?(maximum)
>
>
>
> "C
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421334 is a reply to message #421333] |
Mon, 04 August 2008 19:53 |
Simon Mc Duff
Messages: 596
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
------=_NextPart_000_0103_01C8F64A.4C460850
Content-Type: text/plain;
charset="iso-8859-15"
Content-Transfer-Encoding: quoted-printable
Exactly!!
I was asking the question about server and the client to determine at =
which speed the memory should fill up.
If they are in the same java process the memory will fill up twice as =
faster exactly because we have 2 revisionmanager in the same process.
If they are not, we need to determine which (client or the server) have =
the exception.
The only things I can see so far is the maximum memory use for the JAVA =
process is less than the number of objects the cache(both of them) can =
hold.
Simon
"Eike Stepper" <stepper@sympedia.de> a =E9crit dans le message de news: =
g77m8g$ngq$1@build.eclipse.org...
Simon McDuff schrieb:=20
Which value is your LRU set ? (Client and server)
I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.
Can you send us a complete test (or even better a testcase) ? We will =
know=20
exactly your settings you are using.
By the way, do not forget that you need to configure 2 cache.... client=20
cache and the server cache.
Are your client and server the same java process ? Just a side note: =
Currently even if both client and server are running in the same JVM =
process they would not share the same revision manager instance.
Cheers
/Eike
How much memory you JAVA=20
can use ?(maximum)
"C=E9dric Brun" <cedric.brun@obeo.fr> a =E9crit dans le message de news: =
g77d26$rs8$1@build.eclipse.org...
Hi Eike and Simmon (or anybody familiar with CDO internals),
I'm still digging in CDO, and I wrote yet another dummy testcase just=20
filling a model with thousands of instances using a fake backend which=20
does not save anything at all (the addRevision do nothing so I have no=20
references on the java objects). Of course I'm not even trying to browse =
the model on the repository ;)
My loop was something like this :
int nb_loops =3D 10000;
for (int i =3D 0; i < nb_loops; i++) {
MyEClass newF =3D MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}
I was surprised to see that my test ended up with an OutOfMemory error
Digging with the *great* memory analyzer I've seen that all the =
revisions=20
were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,=20
shouldn't this state be cleared after a commit ?
I changed my my test to create a new transaction and close it on each=20
loop, but it did not helped as I guess the resolver is linked to the=20
session instance.
So I'm wondering, when should the CDORevisionResolverImpl be cleared ?=20
And more especially the "revisions" hashmap ?
Just in case you're wondering I'm using the ganymede CDO.
Thanks,
C=E9dric=20
=20
------=_NextPart_000_0103_01C8F64A.4C460850
Content-Type: text/html;
charset="iso-8859-15"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type =
content=3Dtext/html;charset=3DISO-8859-15>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Exactly!!</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>I was asking the question about server =
and the=20
client to determine at which speed the memory should =
fill=20
up.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>If they are in the same java process =
the memory=20
will fill up twice as faster exactly because we have 2 =
revisionmanager=20
in the same process.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>If they are not, we need to determine =
which (client=20
or the server) have the exception.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>The only things I can see so far =
is=20
the maximum memory use for the JAVA process is less than the number =
of=20
objects the cache(both of them) can hold.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>Simon</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV>"Eike Stepper" <<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>> a =
=E9crit dans le=20
message de news: <A=20
href=3D"mailto:g77m8g$ngq$1@build.eclipse.org">g77m8g$ngq$1@build.eclipse=
..org</A>...</DIV>
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">Simon=20
McDuff schrieb:=20
<BLOCKQUOTE cite=3Dmid:g77l0s$bi4$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Which value is your LRU set ? (Client and =
server)
I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.
Can you send us a complete test (or even better a testcase) ? We will =
know=20
exactly your settings you are using.
By the way, do not forget that you need to configure 2 cache.... client=20
cache and the server cache.
Are your client and server the same java process ? =
</PRE></BLOCKQUOTE>Just a=20
side note: Currently even if both client and server are running in the =
same=20
JVM process they would not share the same revision manager=20
instance.<BR><BR>Cheers<BR>/Eike<BR><BR>
<BLOCKQUOTE cite=3Dmid:g77l0s$bi4$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">How much memory you JAVA=20
can use ?(maximum)
"C=E9dric Brun" <A class=3Dmoz-txt-link-rfc2396E =
href=3D"mailto:cedric.brun@obeo.fr"><cedric.brun@obeo.fr></A> a =
=E9crit dans le message de news:=20
<A class=3Dmoz-txt-link-abbreviated =
href=3D"mailto:g77d26$rs8$1@build.eclipse.org">g77d26$rs8$1@build.eclipse=
..org</A>...
</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Hi Eike and Simmon (or =
anybody familiar with CDO internals),
I'm still digging in CDO, and I wrote yet another dummy testcase just=20
filling a model with thousands of instances using a fake backend which=20
does not save anything at all (the addRevision do nothing so I have no=20
references on the java objects). Of course I'm not even trying to browse =
the model on the repository ;)
My loop was something like this :
int nb_loops =3D 10000;
for (int i =3D 0; i < nb_loops; i++) {
MyEClass newF =3D MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}
I was surprised to see that my test ended up with an OutOfMemory error
Digging with the *great* memory analyzer I've seen that all the =
revisions=20
were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,=20
shouldn't this state be cleared after a commit ?
I changed my my test to create a new transaction and close it on each=20
loop, but it did not helped as I guess the resolver is linked to the=20
session instance.
So I'm wondering, when should the CDORevisionResolverImpl be cleared ?=20
And more especially the "revisions" hashmap ?
Just in case you're wondering I'm using the ganymede CDO.
Thanks,
C=E9dric=20
</PRE></BLOCKQUOTE><PRE wrap=3D""><!---->
</PRE></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>
------=_NextPart_000_0103_01C8F64A.4C460850--
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421353 is a reply to message #421332] |
Tue, 05 August 2008 13:05 |
Cedric Brun
Messages: 431
Registered: July 2009 |
Senior Member |
|
|
Hi Simon,
> Which value is your LRU set ? (Client and server)
>
> I know in my case I was able to load 51 millions objects...
> OutOfmemory are always tricky.
>
> Can you send us a complete test (or even better a testcase) ? We will know
> exactly your settings you are using.
>
> By the way, do not forget that you need to configure 2 cache.... client
> cache and the server cache.
That's a usefull hint, I was indeed only configuring the client cache, configuring the server one seems way better (the test runs longer), but I still ends up with an OutOfMemory.
My caches are set to 100/50 for current/revised.
> Are your client and server the same java process ? How much memory you
> JAVA can use ?(maximum)
Client and Server are sharing the same process and I volontary limited heap space to 256m. My goal is to see wether CDO have leaks or not.
When I get to the OOM, the dump tells me I have 16 000 instances of LRURevisionHolder kept in the client revision manager. Digging a step further it appears that the session activation create a new RevisionManager with a 0 as a current/revised capacity (and then create non-evicting LRU).
This creation happens during the session creation which happens during a :
CDOSession session = (CDOSession) IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.
So I added a postProcessor which sets the current/revised LRU size and it's now working flawlessly :) though I don't know if I've done it the way I should.
Thanks for your help, it was really valuable, as usual :)
Cédric
>
>
> "C�dric Brun" <cedric.brun@obeo.fr> a �crit dans le message de news:
> g77d26$rs8$1@build.eclipse.org...
>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>
>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>> filling a model with thousands of instances using a fake backend which
>> does not save anything at all (the addRevision do nothing so I have no
>> references on the java objects). Of course I'm not even trying to browse
>> the model on the repository ;)
>>
>> My loop was something like this :
>>
>> int nb_loops = 10000;
>> for (int i = 0; i < nb_loops; i++) {
>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>> newF.setName("" + i);
>> root.getMembers().add(newF);
>> tx.commit();
>> }
>>
>> I was surprised to see that my test ended up with an OutOfMemory error
>>
>> Digging with the *great* memory analyzer I've seen that all the revisions
>> were kept by CDO through the CDORevisionResolverImpl
>> Setting the LRU cache to 0 did not helped and then I am wondering,
>> shouldn't this state be cleared after a commit ?
>>
>> I changed my my test to create a new transaction and close it on each
>> loop, but it did not helped as I guess the resolver is linked to the
>> session instance.
>>
>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>> And more especially the "revisions" hashmap ?
>>
>> Just in case you're wondering I'm using the ganymede CDO.
>>
>> Thanks,
>>
>> C�dric
http://cedric.brun.io news and articles on eclipse and eclipse modeling.
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421360 is a reply to message #421353] |
Tue, 05 August 2008 14:23 |
|
This is a multi-part message in MIME format.
--------------080402090506070304060201
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Cédric,
Yes, currently an IElementProcessor is the only chance to configure the
internals of the CDOSessionImpl (here: revisionManager) in an
IManagedContainer. Could you please file a Bugzilla so that I can add
respective properties to the CDOSessionConfiguration?
Cheers
/Eike
Cédric Brun schrieb:
> Hi Simon,
>
>
>> Which value is your LRU set ? (Client and server)
>>
>> I know in my case I was able to load 51 millions objects...
>> OutOfmemory are always tricky.
>>
>> Can you send us a complete test (or even better a testcase) ? We will know
>> exactly your settings you are using.
>>
>> By the way, do not forget that you need to configure 2 cache.... client
>> cache and the server cache.
>>
>
> That's a usefull hint, I was indeed only configuring the client cache, configuring the server one seems way better (the test runs longer), but I still ends up with an OutOfMemory.
>
> My caches are set to 100/50 for current/revised.
>
>
>> Are your client and server the same java process ? How much memory you
>> JAVA can use ?(maximum)
>>
>
> Client and Server are sharing the same process and I volontary limited heap space to 256m. My goal is to see wether CDO have leaks or not.
>
> When I get to the OOM, the dump tells me I have 16 000 instances of LRURevisionHolder kept in the client revision manager. Digging a step further it appears that the session activation create a new RevisionManager with a 0 as a current/revised capacity (and then create non-evicting LRU).
>
> This creation happens during the session creation which happens during a :
> CDOSession session = (CDOSession) IPluginContainer.INSTANCE.getElement(productGroup, type, description);
> instruction.
>
> So I added a postProcessor which sets the current/revised LRU size and it's now working flawlessly :) though I don't know if I've done it the way I should.
>
> Thanks for your help, it was really valuable, as usual :)
>
> Cédric
>
>> "C�dric Brun" <cedric.brun@obeo.fr> a �crit dans le message de news:
>> g77d26$rs8$1@build.eclipse.org...
>>
>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>
>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>> filling a model with thousands of instances using a fake backend which
>>> does not save anything at all (the addRevision do nothing so I have no
>>> references on the java objects). Of course I'm not even trying to browse
>>> the model on the repository ;)
>>>
>>> My loop was something like this :
>>>
>>> int nb_loops = 10000;
>>> for (int i = 0; i < nb_loops; i++) {
>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>> newF.setName("" + i);
>>> root.getMembers().add(newF);
>>> tx.commit();
>>> }
>>>
>>> I was surprised to see that my test ended up with an OutOfMemory error
>>>
>>> Digging with the *great* memory analyzer I've seen that all the revisions
>>> were kept by CDO through the CDORevisionResolverImpl
>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>> shouldn't this state be cleared after a commit ?
>>>
>>> I changed my my test to create a new transaction and close it on each
>>> loop, but it did not helped as I guess the resolver is linked to the
>>> session instance.
>>>
>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>>> And more especially the "revisions" hashmap ?
>>>
>>> Just in case you're wondering I'm using the ganymede CDO.
>>>
>>> Thanks,
>>>
>>> C�dric
>>>
>
>
--------------080402090506070304060201
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Cédric,<br>
<br>
Yes, currently an IElementProcessor is the only chance to configure the
internals of the CDOSessionImpl (here: revisionManager) in an
IManagedContainer. Could you please file a Bugzilla so that I can add
respective properties to the CDOSessionConfiguration?<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g79j6a$8ce$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Simon,
</pre>
<blockquote type="cite">
<pre wrap="">Which value is your LRU set ? (Client and server)
I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.
Can you send us a complete test (or even better a testcase) ? We will know
exactly your settings you are using.
By the way, do not forget that you need to configure 2 cache.... client
cache and the server cache.
</pre>
</blockquote>
<pre wrap=""><!---->
That's a usefull hint, I was indeed only configuring the client cache, configuring the server one seems way better (the test runs longer), but I still ends up with an OutOfMemory.
My caches are set to 100/50 for current/revised.
</pre>
<blockquote type="cite">
<pre wrap="">Are your client and server the same java process ? How much memory you
JAVA can use ?(maximum)
</pre>
</blockquote>
<pre wrap=""><!---->
Client and Server are sharing the same process and I volontary limited heap space to 256m. My goal is to see wether CDO have leaks or not.
When I get to the OOM, the dump tells me I have 16 000 instances of LRURevisionHolder kept in the client revision manager. Digging a step further it appears that the session activation create a new RevisionManager with a 0 as a current/revised capacity (and then create non-evicting LRU).
This creation happens during the session creation which happens during a :
CDOSession session = (CDOSession) IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.
So I added a postProcessor which sets the current/revised LRU size and it's now working flawlessly :) though I don't know if I've done it the way I should.
Thanks for your help, it was really valuable, as usual :)
Cédric
</pre>
<blockquote type="cite">
<pre wrap="">
"C�dric Brun" <a class="moz-txt-link-rfc2396E" href="mailto:cedric.brun@obeo.fr"><cedric.brun@obeo.fr></a> a �crit dans le message de news:
<a class="moz-txt-link-abbreviated" href="mailto:g77d26$rs8$1@build.eclipse.org">g77d26$rs8$1@build.eclipse.org</a>...
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike and Simmon (or anybody familiar with CDO internals),
I'm still digging in CDO, and I wrote yet another dummy testcase just
filling a model with thousands of instances using a fake backend which
does not save anything at all (the addRevision do nothing so I have no
references on the java objects). Of course I'm not even trying to browse
the model on the repository ;)
My loop was something like this :
int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}
I was surprised to see that my test ended up with an OutOfMemory error
Digging with the *great* memory analyzer I've seen that all the revisions
were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,
shouldn't this state be cleared after a commit ?
I changed my my test to create a new transaction and close it on each
loop, but it did not helped as I guess the resolver is linked to the
session instance.
So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
And more especially the "revisions" hashmap ?
Just in case you're wondering I'm using the ganymede CDO.
Thanks,
C�dric
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>
--------------080402090506070304060201--
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421381 is a reply to message #421360] |
Wed, 06 August 2008 09:46 |
Cedric Brun
Messages: 431
Registered: July 2009 |
Senior Member |
|
|
Hi Eike,
Bug is tracked : https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279
Since I change the settings with the revised/current LRU, I'm now experiencing the following issue :
java.lang.IllegalStateException: Can not retrieve origin revision for org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisi onDeltaImpl @5e34ce
at org.eclipse.emf.cdo.internal.server.Transaction.computeDirty Objects(Transaction.java:290)
at org.eclipse.emf.cdo.internal.server.Transaction.commit(Trans action.java:179)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:109 )
at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool Executor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
I'm using a MEMStore for test purposes.
Do you have a hint about what's going wrong ?
Eike Stepper wrote:
> Hi Cédric,
>
> Yes, currently an IElementProcessor is the only chance to configure the
> internals of the CDOSessionImpl (here: revisionManager) in an
> IManagedContainer. Could you please file a Bugzilla so that I can add
> respective properties to the CDOSessionConfiguration?
>
> Cheers
> /Eike
>
>
>
> Cédric Brun schrieb:
>> Hi Simon,
>>
>>
>>> Which value is your LRU set ? (Client and server)
>>>
>>> I know in my case I was able to load 51 millions objects...
>>> OutOfmemory are always tricky.
>>>
>>> Can you send us a complete test (or even better a testcase) ? We will
>>> know exactly your settings you are using.
>>>
>>> By the way, do not forget that you need to configure 2 cache.... client
>>> cache and the server cache.
>>>
>>
>> That's a usefull hint, I was indeed only configuring the client cache,
>> configuring the server one seems way better (the test runs longer), but I
>> still ends up with an OutOfMemory.
>>
>> My caches are set to 100/50 for current/revised.
>>
>>
>>> Are your client and server the same java process ? How much memory you
>>> JAVA can use ?(maximum)
>>>
>>
>> Client and Server are sharing the same process and I volontary limited
>> heap space to 256m. My goal is to see wether CDO have leaks or not.
>>
>> When I get to the OOM, the dump tells me I have 16 000 instances of
>> LRURevisionHolder kept in the client revision manager. Digging a step
>> further it appears that the session activation create a new
>> RevisionManager with a 0 as a current/revised capacity (and then create
>> non-evicting LRU).
>>
>> This creation happens during the session creation which happens during a
>> : CDOSession session = (CDOSession)
>> IPluginContainer.INSTANCE.getElement(productGroup, type, description);
>> instruction.
>>
>> So I added a postProcessor which sets the current/revised LRU size and
>> it's now working flawlessly :) though I don't know if I've done it the
>> way I should.
>>
>> Thanks for your help, it was really valuable, as usual :)
>>
>> Cédric
>>
>>> "C�dric Brun" <cedric.brun@obeo.fr> a �crit dans le message de news:
>>> g77d26$rs8$1@build.eclipse.org...
>>>
>>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>>
>>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>>> filling a model with thousands of instances using a fake backend which
>>>> does not save anything at all (the addRevision do nothing so I have no
>>>> references on the java objects). Of course I'm not even trying to
>>>> browse the model on the repository ;)
>>>>
>>>> My loop was something like this :
>>>>
>>>> int nb_loops = 10000;
>>>> for (int i = 0; i < nb_loops; i++) {
>>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>>> newF.setName("" + i);
>>>> root.getMembers().add(newF);
>>>> tx.commit();
>>>> }
>>>>
>>>> I was surprised to see that my test ended up with an OutOfMemory error
>>>>
>>>> Digging with the *great* memory analyzer I've seen that all the
>>>> revisions were kept by CDO through the CDORevisionResolverImpl
>>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>>> shouldn't this state be cleared after a commit ?
>>>>
>>>> I changed my my test to create a new transaction and close it on each
>>>> loop, but it did not helped as I guess the resolver is linked to the
>>>> session instance.
>>>>
>>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>>>> And more especially the "revisions" hashmap ?
>>>>
>>>> Just in case you're wondering I'm using the ganymede CDO.
>>>>
>>>> Thanks,
>>>>
>>>> C�dric
>>>>
>>
>>
http://cedric.brun.io news and articles on eclipse and eclipse modeling.
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421383 is a reply to message #421381] |
Wed, 06 August 2008 09:56 |
|
This is a multi-part message in MIME format.
--------------040007020101020500040806
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Cédric Brun schrieb:
> Hi Eike,
>
> Bug is tracked : https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279
>
> Since I change the settings with the revised/current LRU, I'm now experiencing the following issue :
>
> java.lang.IllegalStateException: Can not retrieve origin revision for org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisi onDeltaImpl @5e34ce
> at org.eclipse.emf.cdo.internal.server.Transaction.computeDirty Objects(Transaction.java:290)
> at org.eclipse.emf.cdo.internal.server.Transaction.commit(Trans action.java:179)
> at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:109 )
> at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
> at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool Executor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
> I'm using a MEMStore for test purposes.
>
I guess it is either a bug in the MEMStore implementation or it is a
misconfiguration of PROP_SUPPORTING_REVISION_DELTAS (which then should
have a better exception).
The best would be to file a Bugzilla and attach your server config and
the whole console trace of the server.
Cheers
/Eike
> Do you have a hint about what's going wrong ?
>
> Eike Stepper wrote:
>
>
>> Hi Cédric,
>>
>> Yes, currently an IElementProcessor is the only chance to configure the
>> internals of the CDOSessionImpl (here: revisionManager) in an
>> IManagedContainer. Could you please file a Bugzilla so that I can add
>> respective properties to the CDOSessionConfiguration?
>>
>> Cheers
>> /Eike
>>
>>
>>
>> Cédric Brun schrieb:
>>
>>> Hi Simon,
>>>
>>>
>>>
>>>> Which value is your LRU set ? (Client and server)
>>>>
>>>> I know in my case I was able to load 51 millions objects...
>>>> OutOfmemory are always tricky.
>>>>
>>>> Can you send us a complete test (or even better a testcase) ? We will
>>>> know exactly your settings you are using.
>>>>
>>>> By the way, do not forget that you need to configure 2 cache.... client
>>>> cache and the server cache.
>>>>
>>>>
>>> That's a usefull hint, I was indeed only configuring the client cache,
>>> configuring the server one seems way better (the test runs longer), but I
>>> still ends up with an OutOfMemory.
>>>
>>> My caches are set to 100/50 for current/revised.
>>>
>>>
>>>
>>>> Are your client and server the same java process ? How much memory you
>>>> JAVA can use ?(maximum)
>>>>
>>>>
>>> Client and Server are sharing the same process and I volontary limited
>>> heap space to 256m. My goal is to see wether CDO have leaks or not.
>>>
>>> When I get to the OOM, the dump tells me I have 16 000 instances of
>>> LRURevisionHolder kept in the client revision manager. Digging a step
>>> further it appears that the session activation create a new
>>> RevisionManager with a 0 as a current/revised capacity (and then create
>>> non-evicting LRU).
>>>
>>> This creation happens during the session creation which happens during a
>>> : CDOSession session = (CDOSession)
>>> IPluginContainer.INSTANCE.getElement(productGroup, type, description);
>>> instruction.
>>>
>>> So I added a postProcessor which sets the current/revised LRU size and
>>> it's now working flawlessly :) though I don't know if I've done it the
>>> way I should.
>>>
>>> Thanks for your help, it was really valuable, as usual :)
>>>
>>> Cédric
>>>
>>>
>>>> "C�dric Brun" <cedric.brun@obeo.fr> a �crit dans le message de news:
>>>> g77d26$rs8$1@build.eclipse.org...
>>>>
>>>>
>>>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>>>
>>>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>>>> filling a model with thousands of instances using a fake backend which
>>>>> does not save anything at all (the addRevision do nothing so I have no
>>>>> references on the java objects). Of course I'm not even trying to
>>>>> browse the model on the repository ;)
>>>>>
>>>>> My loop was something like this :
>>>>>
>>>>> int nb_loops = 10000;
>>>>> for (int i = 0; i < nb_loops; i++) {
>>>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>>>> newF.setName("" + i);
>>>>> root.getMembers().add(newF);
>>>>> tx.commit();
>>>>> }
>>>>>
>>>>> I was surprised to see that my test ended up with an OutOfMemory error
>>>>>
>>>>> Digging with the *great* memory analyzer I've seen that all the
>>>>> revisions were kept by CDO through the CDORevisionResolverImpl
>>>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>>>> shouldn't this state be cleared after a commit ?
>>>>>
>>>>> I changed my my test to create a new transaction and close it on each
>>>>> loop, but it did not helped as I guess the resolver is linked to the
>>>>> session instance.
>>>>>
>>>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>>>>> And more especially the "revisions" hashmap ?
>>>>>
>>>>> Just in case you're wondering I'm using the ganymede CDO.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> C�dric
>>>>>
>>>>>
>>>
>
>
--------------040007020101020500040806
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Cédric Brun schrieb:
<blockquote cite="mid:g7brst$5i7$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Eike,
Bug is tracked : <a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279">https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279</a>
Since I change the settings with the revised/current LRU, I'm now experiencing the following issue :
java.lang.IllegalStateException: Can not retrieve origin revision for org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisi onDeltaImpl @5e34ce
at org.eclipse.emf.cdo.internal.server.Transaction.computeDirty Objects(Transaction.java:290)
at org.eclipse.emf.cdo.internal.server.Transaction.commit(Trans action.java:179)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:109 )
at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool Executor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
I'm using a MEMStore for test purposes.
</pre>
</blockquote>
I guess it is either a bug in the MEMStore implementation or it is a
misconfiguration of PROP_SUPPORTING_REVISION_DELTAS (which then should
have a better exception).<br>
The best would be to file a Bugzilla and attach your server config and
the whole console trace of the server.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<blockquote cite="mid:g7brst$5i7$1@build.eclipse.org" type="cite">
<pre wrap="">
Do you have a hint about what's going wrong ?
Eike Stepper wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Cédric,
Yes, currently an IElementProcessor is the only chance to configure the
internals of the CDOSessionImpl (here: revisionManager) in an
IManagedContainer. Could you please file a Bugzilla so that I can add
respective properties to the CDOSessionConfiguration?
Cheers
/Eike
Cédric Brun schrieb:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Simon,
</pre>
<blockquote type="cite">
<pre wrap="">Which value is your LRU set ? (Client and server)
I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.
Can you send us a complete test (or even better a testcase) ? We will
know exactly your settings you are using.
By the way, do not forget that you need to configure 2 cache.... client
cache and the server cache.
</pre>
</blockquote>
<pre wrap="">That's a usefull hint, I was indeed only configuring the client cache,
configuring the server one seems way better (the test runs longer), but I
still ends up with an OutOfMemory.
My caches are set to 100/50 for current/revised.
</pre>
<blockquote type="cite">
<pre wrap="">Are your client and server the same java process ? How much memory you
JAVA can use ?(maximum)
</pre>
</blockquote>
<pre wrap="">Client and Server are sharing the same process and I volontary limited
heap space to 256m. My goal is to see wether CDO have leaks or not.
When I get to the OOM, the dump tells me I have 16 000 instances of
LRURevisionHolder kept in the client revision manager. Digging a step
further it appears that the session activation create a new
RevisionManager with a 0 as a current/revised capacity (and then create
non-evicting LRU).
This creation happens during the session creation which happens during a
: CDOSession session = (CDOSession)
IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.
So I added a postProcessor which sets the current/revised LRU size and
it's now working flawlessly :) though I don't know if I've done it the
way I should.
Thanks for your help, it was really valuable, as usual :)
Cédric
</pre>
<blockquote type="cite">
<pre wrap="">"C�dric Brun" <a class="moz-txt-link-rfc2396E" href="mailto:cedric.brun@obeo.fr"><cedric.brun@obeo.fr></a> a �crit dans le message de news:
<a class="moz-txt-link-abbreviated" href="mailto:g77d26$rs8$1@build.eclipse.org">g77d26$rs8$1@build.eclipse.org</a>...
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike and Simmon (or anybody familiar with CDO internals),
I'm still digging in CDO, and I wrote yet another dummy testcase just
filling a model with thousands of instances using a fake backend which
does not save anything at all (the addRevision do nothing so I have no
references on the java objects). Of course I'm not even trying to
browse the model on the repository ;)
My loop was something like this :
int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}
I was surprised to see that my test ended up with an OutOfMemory error
Digging with the *great* memory analyzer I've seen that all the
revisions were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,
shouldn't this state be cleared after a commit ?
I changed my my test to create a new transaction and close it on each
loop, but it did not helped as I guess the resolver is linked to the
session instance.
So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
And more especially the "revisions" hashmap ?
Just in case you're wondering I'm using the ganymede CDO.
Thanks,
C�dric
</pre>
</blockquote>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>
--------------040007020101020500040806--
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Question about LRU cache and tx.commit() [message #421385 is a reply to message #421383] |
Wed, 06 August 2008 10:13 |
Cedric Brun
Messages: 431
Registered: July 2009 |
Senior Member |
|
|
Woah , you are so quick ! Did Ed gave you some kind of doping substance ?
Thanks for being so quick :)
I was exactly doing that, I tracked a new bug and added a sample project with associated junit testcase.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=243282
Cédric
Eike Stepper wrote:
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> Bug is tracked : https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279
>>
>> Since I change the settings with the revised/current LRU, I'm now
>> experiencing the following issue :
>>
>> java.lang.IllegalStateException: Can not retrieve origin revision for
>> org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisi onDeltaImpl @5e34ce
>> at
>> org.eclipse.emf.cdo.internal.server.Transaction.computeDirty Objects(Transaction.java:290)
>> at
>> org.eclipse.emf.cdo.internal.server.Transaction.commit(Trans action.java:179)
>> at
>> org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:109 )
>> at
>> org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
>> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143) at
>> org.eclipse.net4j.signal.Signal.run(Signal.java:124) at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool Executor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> I'm using a MEMStore for test purposes.
>>
> I guess it is either a bug in the MEMStore implementation or it is a
> misconfiguration of PROP_SUPPORTING_REVISION_DELTAS (which then should
> have a better exception).
> The best would be to file a Bugzilla and attach your server config and
> the whole console trace of the server.
>
> Cheers
> /Eike
>
>> Do you have a hint about what's going wrong ?
>>
>> Eike Stepper wrote:
>>
>>
>>> Hi Cédric,
>>>
>>> Yes, currently an IElementProcessor is the only chance to configure the
>>> internals of the CDOSessionImpl (here: revisionManager) in an
>>> IManagedContainer. Could you please file a Bugzilla so that I can add
>>> respective properties to the CDOSessionConfiguration?
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>>
>>> Cédric Brun schrieb:
>>>
>>>> Hi Simon,
>>>>
>>>>
>>>>
>>>>> Which value is your LRU set ? (Client and server)
>>>>>
>>>>> I know in my case I was able to load 51 millions objects...
>>>>> OutOfmemory are always tricky.
>>>>>
>>>>> Can you send us a complete test (or even better a testcase) ? We will
>>>>> know exactly your settings you are using.
>>>>>
>>>>> By the way, do not forget that you need to configure 2 cache....
>>>>> client cache and the server cache.
>>>>>
>>>>>
>>>> That's a usefull hint, I was indeed only configuring the client cache,
>>>> configuring the server one seems way better (the test runs longer), but
>>>> I still ends up with an OutOfMemory.
>>>>
>>>> My caches are set to 100/50 for current/revised.
>>>>
>>>>
>>>>
>>>>> Are your client and server the same java process ? How much memory you
>>>>> JAVA can use ?(maximum)
>>>>>
>>>>>
>>>> Client and Server are sharing the same process and I volontary limited
>>>> heap space to 256m. My goal is to see wether CDO have leaks or not.
>>>>
>>>> When I get to the OOM, the dump tells me I have 16 000 instances of
>>>> LRURevisionHolder kept in the client revision manager. Digging a step
>>>> further it appears that the session activation create a new
>>>> RevisionManager with a 0 as a current/revised capacity (and then create
>>>> non-evicting LRU).
>>>>
>>>> This creation happens during the session creation which happens during
>>>> a
>>>> : CDOSession session = (CDOSession)
>>>> IPluginContainer.INSTANCE.getElement(productGroup, type, description);
>>>> instruction.
>>>>
>>>> So I added a postProcessor which sets the current/revised LRU size and
>>>> it's now working flawlessly :) though I don't know if I've done it the
>>>> way I should.
>>>>
>>>> Thanks for your help, it was really valuable, as usual :)
>>>>
>>>> Cédric
>>>>
>>>>
>>>>> "C�dric Brun" <cedric.brun@obeo.fr> a �crit dans le message de news:
>>>>> g77d26$rs8$1@build.eclipse.org...
>>>>>
>>>>>
>>>>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>>>>
>>>>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>>>>> filling a model with thousands of instances using a fake backend
>>>>>> which does not save anything at all (the addRevision do nothing so I
>>>>>> have no references on the java objects). Of course I'm not even
>>>>>> trying to browse the model on the repository ;)
>>>>>>
>>>>>> My loop was something like this :
>>>>>>
>>>>>> int nb_loops = 10000;
>>>>>> for (int i = 0; i < nb_loops; i++) {
>>>>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>>>>> newF.setName("" + i);
>>>>>> root.getMembers().add(newF);
>>>>>> tx.commit();
>>>>>> }
>>>>>>
>>>>>> I was surprised to see that my test ended up with an OutOfMemory
>>>>>> error
>>>>>>
>>>>>> Digging with the *great* memory analyzer I've seen that all the
>>>>>> revisions were kept by CDO through the CDORevisionResolverImpl
>>>>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>>>>> shouldn't this state be cleared after a commit ?
>>>>>>
>>>>>> I changed my my test to create a new transaction and close it on each
>>>>>> loop, but it did not helped as I guess the resolver is linked to the
>>>>>> session instance.
>>>>>>
>>>>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared
>>>>>> ? And more especially the "revisions" hashmap ?
>>>>>>
>>>>>> Just in case you're wondering I'm using the ganymede CDO.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> C�dric
>>>>>>
>>>>>>
>>>>
>>
>>
http://cedric.brun.io news and articles on eclipse and eclipse modeling.
|
|
|
Goto Forum:
Current Time: Thu Sep 26 07:17:22 GMT 2024
Powered by FUDForum. Page generated in 0.06141 seconds |
] 
Search
Help
Register
Login
Home
