Eclipse Community Forums: EclipseLink » OutOfMemoryError from JTASynchronizationListener

Home » Eclipse Projects » EclipseLink » OutOfMemoryError from JTASynchronizationListener(Huge amount fo memory referenced from JTASynchronizationListener could not get garbage collected)

Show: Today's Messages :: Show Polls :: Message Navigator

OutOfMemoryError from JTASynchronizationListener [message #1003426]

Mon, 21 January 2013 08:10

Uwe Oetken

Messages: 4
Registered: January 2013

Junior Member

Hello everybody,

50 minutes after application start out server is running out of memory.
HeapDump shows that there is a JTASynchronizationListener holding over 93 percent of memory.
Several use cases have been successfully executed until this, none of them seems to cause the problem. The used heap memory rises 35 minutes after application start from normal level (~ 200 MByte) up to 1.5 GByte and could not get garbage collected any more.
Environment is WebSphere 6.1 Fixpack 45 with EJB 3.0 Feature pack, Eclipselink 2.3.0 with JPA 1.0
EclipseLink shared cache is disabled via persistence.xml, class specific cache in descriptors have default values SOFT_CACHE.
This problem occures non deterministic on productive instances and could not be seen in development instances after running some thousands of use cases.

HeapDump shows:
JTASynchronizationListener with size of 1.4 GByte references RepeatableWriteUnitOfWork. Inside this there is an UnitOfWorkChangeSet, containing an IdentityHashMap with more than 151.000 entity objects of different classes (packages .../core/server/..., see below). Some of these entities will never be modified by the application. It seems as if these objects could not have been read in an single transaction.
There are 9 other instances of JTASynchronizationListener with heap size between 40 bytes and 8.8 MBytes, they seem to be okay.
Is looks like this UnitOfWorkChangeSet collects objects from a lot of transactions and is not cleaned up after transactions ended.

Here is the reference tree from IBM HeapAnalyzer:
1.421.397.360 (93,37%) [296] 16 com/ibm/ejs/j2c/MCWrapper 0x9caa10f8
1.418.944.768 (93,2%) [224] 7 com/ibm/ws/Transaction/JTA/TransactionImpl 0xa6c80f40
1.418.943.264 (93,2%) [40] 2 com/ibm/ws/Transaction/JTA/RegisteredSyncs 0xa500bae0
1.418.943.224 (93,2%) [32] 2 array of java/util/ArrayList 0xa494cfb0
1.418.942.056 (93,2%) [24] 1 java/util/ArrayList 0xa494ed20
1.418.942.032 (93,2%) [56] 3 array of java/lang/Object 0xa494fd70
1.418.593.048 (93,18%) [40] 5 org/eclipse/persistence/transaction/JTASynchronizationListener 0xa49513f8
1.412.911.600 (92,81%) [392] 16 org/eclipse/persistence/internal/sessions/RepeatableWriteUnitOfWork 0xa6c8a598
1.376.790.656 (90,44%) [56] 7 org/eclipse/persistence/internal/sessions/UnitOfWorkChangeSet 0xa6f56fa8
1.376.767.904 (90,43%) [40] 1 java/util/IdentityHashMap 0xa6f57190
1.376.767.864 (90,43%) [1.040] 164 array of java/lang/Object 0xc4359110
21.540.832 (1,41%) [112] 6 org/eclipse/persistence/internal/sessions/ObjectChangeSet 0x9abdda68
21.540.520 (1,41%) [56] 6 org/eclipse/persistence/internal/sessions/UnitOfWorkChangeSet 0x9b592c10
20.491.184 (1,35%) [40] 1 java/util/IdentityHashMap 0x9b5f47a8
20.491.144 (1,35%) [1.048.592] 151.462 array of java/lang/Object 0x9ed38758
4.760 (0%) [112] 6 .../core/server/fzg/bo/MarVO 0xa73cce78
1.352 (0%) [112] .../core/server/fzg/bo/MarVO 0xa738cd58
912 (0%) [112] 6 .../core/server/fzg/bo/MarVO 0xa72cad38
544 (0%) [96] 6 .../core/server/util/bo/KlaVO 0xa71cc700
528 (0%) [96] 6 .../core/server/util/bo/KlaVO 0xa8e0f9b0
528 (0%) [104] .../core/server/st/bo/DiaVO 0xa73cd188
528 (0%) [96] 6 .../core/server/util/bo/KlaVO 0xa73ccff0
...

Why these entities are still referenced? How can I avoid this?
Thanks for help,

Uwe

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1004091 is a reply to message #1003426]

Tue, 22 January 2013 15:03

Chris Delahunt

Messages: 1389
Registered: July 2009

Senior Member

How have you defined the persistence unit and managing the EntityManager lifecycle? EntityManagers (wrappers overtop the RepeatableWriteUnitOfWork) only release entities when cleared or closed/gc'd. So a long lived EntityManager might be the cause of your issue.
My guess would be to look at any stateful sessionbeans and check if one is holding onto an EntityManager that has a PersistenceContextType of Extended. If so, it may need to be cleared occasionally to cut down on the memory foot print.

Best Regards,
Chris

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1004099 is a reply to message #1003426]

Tue, 22 January 2013 15:19

James Sutherland

Messages: 1939
Registered: July 2009
Location: Ottawa, Canada

Senior Member

Odd, what is different between the development and production servers?
What is WebSphere is holding onto the old transaction? Or is this the current transaction? Does it read a lot of objects?

WebSphere does do some sort of caching of EntityManagers in its EJB support, this may be related to the issue, although it should be clearing them to avoid such issues.

Are you encountering any failed transactions or errors, it could be an issue that things are not getting properly cleaned up for failed transactions.

James : Wiki : Book : Blog : Twitter

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1004546 is a reply to message #1004091]

Wed, 23 January 2013 12:36

Uwe Oetken

Messages: 4
Registered: January 2013

Junior Member

Thanks for the answer.

There are absolutly no statefull session beans, only stateless session beans.

Entity manager is annotated in common super class of all stateless session beans, so entity manager is completely under control of websphere ejb container:
public abstract class BaseStatelessSessionBean {
@PersistenceContext(name = "S42Core", unitName = "S42Core")
private EntityManager entityManager;
...
}

Persistence unit is defined as composite persistence unit in persistence.xml as follows:
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" ...>
<persistence-unit name="S42Core" transaction-type="JTA">
<provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
<jta-data-source>jdbc/s42core</jta-data-source>
<jar-file>S42SearchEJB.jar</jar-file>
<jar-file>S42ServerEJB.jar</jar-file>
<jar-file>S42StatusEJB.jar</jar-file>
<jar-file>S42UtilEJB.jar</jar-file>
...

<properties>
<property name="eclipselink.cache.shared.default" value="false" />
<property name="eclipselink.composite-unit" value="true" />
<property name="eclipselink.jdbc.batch-writing" value="Oracle-JDBC" />
<property name="eclipselink.jdbc.batch-writing.size" value="500" />
<property name="eclipselink.jdbc.bind-parameters" value="true" />
<property name="eclipselink.jdbc.cache-statements" value="true" />
<property name="eclipselink.jdbc.cache-statements.size" value="1000" />
<property name="eclipselink.target-database" value="org.eclipse.persistence.platform.database.oracle.Oracle11Platform" />
<property name="eclipselink.target-server" value="WebSphere_6_1" />
<property name="eclipselink.weaving" value="static" />
<property name="eclipselink.weaving.eager" value="true" />
</properties>
</persistence-unit>
</persistence>

with composite persistence unit members as follows:
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" ...>
<persistence-unit name="S42ServerEJB" transaction-type="JTA">
<provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
<jta-data-source>jdbc/s42core</jta-data-source>

<class>...core.server.fzg.bo.MarVO</class>
<class>...core.server.st.bo.DiaVO</class>
...

<exclude-unlisted-classes>true</exclude-unlisted-classes>

<properties>
<property name="eclipselink.cache.shared.default" value="false" />
<property name="eclipselink.composite-unit.member" value="true" />
<property name="eclipselink.jdbc.batch-writing" value="Oracle-JDBC" />
<property name="eclipselink.jdbc.batch-writing.size" value="500" />
<property name="eclipselink.jdbc.bind-parameters" value="true" />
<property name="eclipselink.jdbc.cache-statements" value="true" />
<property name="eclipselink.jdbc.cache-statements.size" value="1000" />
<property name="eclipselink.target-database" value="org.eclipse.persistence.platform.database.oracle.Oracle11Platform" />
<property name="eclipselink.target-server" value="WebSphere_6_1" />
<property name="eclipselink.weaving" value="static" />
<property name="eclipselink.weaving.eager" value="true" />
</properties>
</persistence-unit>
</persistence>

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1004560 is a reply to message #1004099]

Wed, 23 January 2013 13:02

Uwe Oetken

Messages: 4
Registered: January 2013

Junior Member

Sure, there are some transactions failed or rolled back on every server because of regular user errors.

And there are some transaction timeouts. Only with WebSphere 6.1 Fixpack 45 for these timeouts there is thrown an exception, see IBM PM63801. Maybe this doesn't still cleanup everything as required?

One difference between development and production server is that production uses XA datasources (jdbc provider oracle.jdbc.xa.client.OracleXADataSource from ojdbc5.jar) while in development non-XA datasources are used
(jdbc provider oracle.jdbc.pool.OracleConnectionPoolDataSource from ojdbc5.jar). Could there be any relation to this? Maybe locks on database are hold over server restart or xa_recover is doing some magic after server start? This will be next thing to change in production server.

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1005131 is a reply to message #1004560]

Thu, 24 January 2013 14:31

James Sutherland

Messages: 1939
Registered: July 2009
Location: Ottawa, Canada

Senior Member

Could you try removing the composite persistence unit to see if it has any affect. Both of your persistence units are using the same database/data-source, so there is no reason to be using a composite persistence unit.

Also note that,

<property name="eclipselink.jdbc.cache-statements" value="true" />

has no real affect when using a DataSource, you need to enable statement caching in the data-source config, not in EclipseLink (this is only relevant when using EclipseLink connection pooling).

James : Wiki : Book : Blog : Twitter

Report message to a moderator

Re: OutOfMemoryError from JTASynchronizationListener [message #1007567 is a reply to message #1005131]

Wed, 06 February 2013 14:06

Uwe Oetken

Messages: 4
Registered: January 2013

Junior Member

Hello James,
thanks very much for your suggestions.

I think we found the errors:

1. There are some methods in the session beans implemented by a super class:
public abstract class BaseStatelessSessionBean {
public MyObject[] findSomething() {
...
}
}

@Stateless
@TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
public class MyStatelessSessionBean extends BaseStatelessSessionBean {
public MyObject[] findOther() {
...
}
}
This super class does not get the default J2EE transaction attributes from the derived session bean as expected. So calls to findOther() run with J2EE transaction attribute NOT_SUPPORTED from session bean, but calls to findSomething() run with J2EE transaction attribute REQUIRED. So it has an transaction, and Eclipselink flushes the result of every former read query from the ejb call to the RepeatableWriteUnitOfWork (see performPreQueryFlush() in executeReadQuery() at class org.eclipse.persistence.internal.jpa. EJBQueryImpl).
With explicit J2EE transaction attributes at session bean methods from super class the application runs with heap usage between 0.4 and 0.8 GByte vs. 1.1 to 1.5 GByte before.

2. These super class methods are called in production, but not in development, because they implement generic compression of the results to save network load on WAN communication. That's why this problem could not be identified in development.

3. Additional: Compound persistence units seem to consume more memory and processor runtime than single persistence units as we use now. Indeed, they are here not really required, they only existed for better code organization (enlisting of > 300 entities at their specific ejb project). Without compound persistence units the application's standard test cases run 42 minutes vs. 52 minutes before.

4. And XA transactions are expensive, and we can go without until now.

Report message to a moderator

Previous Topic:	Index on dynamic entities?
Next Topic:	JPA CriteriaQuery MD5

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 07:18:56 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter