| Re: [eclipselink-users] blocked threads, oracle11g and tomcat |
Christopher,
This is our main 'connect' method:
<<<<<<<
// Load the Toplink mapping file.
Project project = new Toplink();
// Retrieve login
DatabaseLogin login = project.getLogin();
// Set connector
login.setConnector(new JNDIConnector(new InitialContext(),
PILConfig.getInstance().getPCSDatasourceName()));
// Clear all properties that are initially set on the project
login.getProperties().clear();
// Set the use of the OC4J ConnectionPooling
login.setUsesExternalConnectionPooling(true);
// set sequencePreAllocationSize
SessionProviderUtil.loadSequencePreAllocation(project);
// load all available QueryRepositories defined in
QueryRepositories.properties
SessionProviderUtil.loadQueries(project);
// Add descriptor events, these methods have been added
SessionProviderUtil.loadAutoMappings(project);
// Create initial Server session with all the correct settings
sessionServer = project.createServerSession();
// Add the logging class to the session.
sessionServer.setSessionLog(new ToplinkSessionLog());
sessionServer.setLogLevel(SessionLog.ALL);
// set the performance profiler, if enabled
SessionProviderUtil.setPerformanceProfiler(sessionServer);
DatabaseLogin databaseLogin = sessionServer.getLogin();
// Turning use batching writing on has as result that for
committing
// multiple insert, update and delete statements will be sent over
one
// JDBC connection
if (PILConfig.getInstance().isToplinkBatchWritingEnabled())
{
databaseLogin.useBatchWriting();
databaseLogin.setMaxBatchWritingSize(PILConfig.getInstance().getToplinkB
atchWritingMaxBatchSize());
}
>>>>>>>>>>
PILConfig.getInstance().getPCSDatasourceName() is the jndi name of a
connection pool configured in tomcat.
We do not enable statement caching, so I guess it is enabled by default
!?
I have added the following lines to disable statement caching:
databaseLogin.dontCacheAllStatements();
databaseLogin.setShouldCacheAllStatements(false);
databaseLogin.setStatementCacheSize(0);
although the first two seem redundant.
What I still don't understand is why the first thread hangs on a socket
read (Thread t@185 in the threaddump).
Any ideas ?
Greetings,
Huub Sepers
-----Original Message-----
From: eclipselink-users-bounces@xxxxxxxxxxx
[mailto:eclipselink-users-bounces@xxxxxxxxxxx] On Behalf Of Christopher
Delahunt
Sent: Wednesday, June 05, 2013 2:30 PM
To: eclipselink-users@xxxxxxxxxxx
Subject: Re: [eclipselink-users] blocked threads, oracle11g and tomcat
EclipseLink supports datasources, but how have you configured them?
Statement caching is what seems to be causing the issue, as the other
thread is attempting to close statements it has cached - statement
caching is not intended to be used with an external datasource though as
per:
http://wiki.eclipse.org/Optimizing_the_EclipseLink_Application_(ELUG)#Ho
w_to_Use_Parameterized_SQL_.28Parameter_Binding.29_and_Prepared_Statemen
t_Caching_for_Optimization
It looks like this feature was mistakenly enabled.
Best Regards,
Chris
On 04/06/2013 4:50 PM, Huub Sepers wrote:
> Tom,
>
> "Since you are using an external connection pool, we don't have much
> control over the connections."
> But surely Eclipselink supports external connection pools?
> In what manner is the tomcat pool failing Eclipselink connection pool
> requirements?
>
> No, I haven't tried the internal connection pool yet
>
> Greetings,
> Huub Sepers
>
> -----Original Message-----
> From: eclipselink-users-bounces@xxxxxxxxxxx
> [mailto:eclipselink-users-bounces@xxxxxxxxxxx] On Behalf Of Tom Ware
> Sent: Tuesday, June 04, 2013 9:49 PM
> To: eclipselink-users@xxxxxxxxxxx
> Subject: Re: [eclipselink-users] blocked threads, oracle11g and tomcat
>
> Since you are using an external connection pool, we don't have much
> control over
> the connections. Have you tried switching to use our internal pool?
>
> On 04/06/2013 3:44 PM, Huub Sepers wrote:
>> Tom,
>>
>> Upgrading is not an option for us. This would involve a long
> integration
>> and test traject.
>>
>> The kind of issue your link points to is not the issue we are facing.
>> See attached threaddump.
>> Thread t@185 holds a lock on a physical connection. A lot of other
>> threads are blocked on acquiring this lock. Those threads are all
> trying
>> to close a statement which holds a reference to a connection which is
>> now being used by another thread namely t@185.
>> Basic question is: why is Eclipselink trying to access a connection
>> which has already been closed (and retrieved from the pool by another
>> thread).
>>
>>
>> Greetings,
>> Huub
>>
>>
>> -----Original Message-----
>> From: eclipselink-users-bounces@xxxxxxxxxxx
>> [mailto:eclipselink-users-bounces@xxxxxxxxxxx] On Behalf Of Tom Ware
>> Sent: Tuesday, June 04, 2013 2:29 PM
>> To: eclipselink-users@xxxxxxxxxxx
>> Subject: Re: [eclipselink-users] blocked threads, oracle11g and
tomcat
>>
>> 'any chance of trying a newer version of EclipseLink. We have fixed
>> quite a
>> number of concurrency issues since 1.1.4. A quick search yielded
this
>> as a
>> candidate:
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=333902
>>
>> On 03/06/2013 4:17 PM, Huub Sepers wrote:
>>> Hi,
>>>
>>> Setup of our application:
>>>
>>> eclipselink 1.1.4
>>>
>>> tomcat-7.0.29 (using tomcat jdbc pool or dbcp pool)
>>>
>>> Oracle 11G. We use latest oracle thin 11G jdbc driver.
>>>
>>> All our connections are obtained through JNDI from an external
>> connection pool
>>> (tomcat-jdbc or dbcp)
>>>
>>> We are experiencing blocked threads while running our application.
>>>
>>> The scenario is roughly as follows:
>>>
>>> Thread-A has send a batch of insert statements to the database and
is
>> in a wait
>>> state. It holds a lock
>>>
>>> on a T4CConnection object.
>>>
>>> Thread-B also tries to do a batch of inserts. However it tries to
>> obtain a lock
>>> on the same T4CConnection object
>>>
>>> as Thread-A.
>>>
>>> Thread-B's stack includes a call to DatabaseAccessor.reconnect()
> which
>> calls
>>> DatabaseAccessor.clearStatementCache()
>>>
>>> which tries to close a statement in the DatabaseAccessor statement
>> cache.
>>>
>>> So it seems that Thread-B has a stament in the cache which has a
>> connection that
>>> had already been closed and
>>>
>>> returned to the pool and that connection is now used by Thread-A
> (!?).
>>>
>>> Note: when running our application on oc4j in combination with it's
>> connection
>>> pool and Oracle 10G we never experienced these locks.
>>>
>>> Question:
>>>
>>> - How is the DatabaseAccessor statement cache supposed to work with
a
>> connection
>>> pool. Trying to close
>>>
>>> statements in the cache will lead to accessing connections that have
>> been closed?
>>>
>>> "http-bio-8280-exec-8" - Thread t@106
>>>
>>> java.lang.Thread.State: RUNNABLE
>>>
>>> at java.net.SocketInputStream.socketRead0(Native
> Method)
>>>
>>> at
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>
>>> at oracle.net.ns.Packet.receive(Packet.java:300)
>>>
>>> at
> oracle.net.ns.DataPacket.receive(DataPacket.java:106)
>>>
>>> at
>> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
>>>
>>> at
>> oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
>>>
>>> at
>> oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
>>>
>>> at
>> oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketI
>> nputStreamWrapper.java:124)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStream
>> Wrapper.java:80)
>>>
>>> at
>> oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
>>>
>>> at
>> oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
>>>
>>> at
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
>>>
>>> at
> oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.jav
>> a:207)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatem
>> ent.java:1044)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(Ora
>> clePreparedStatement.java:10143)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedSt
>> atement.java:10249)
>>>
>>> - locked <19740c7> (a
oracle.jdbc.driver.T4CConnection)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWr
>> apper.java:230)
>>>
>>> at
>>>
>>
>
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingS
>> tatement.java:297)
>>>
>>> at
>>>
>>
>
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingS
>> tatement.java:297)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.platform.database.oracle.Oracle10Platform.execut
>> eBatch(Oracle10Platform.java:53)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.execute
>> JDK12BatchStatement(DatabaseAccessor.java:801)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWri
>>
>
tingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMecha
>> nism.java:138)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWri
>>
>
tingMechanism.appendCall(ParameterizedSQLBatchWritingMechanism.java:82)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicEx
>> ecuteCall(DatabaseAccessor.java:546)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.execute
>> Call(DatabaseAccessor.java:512)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.executeCall(Ab
>> stractSession.java:872)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.ex
>> ecuteCall(DatasourceCallQueryMechanism.java:205)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.ex
>> ecuteCall(DatasourceCallQueryMechanism.java:191)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.ex
>> ecuteNoSelectCall(DatasourceCallQueryMechanism.java:234)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.ex
>> ecuteNoSelect(DatasourceCallQueryMechanism.java:214)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.queries.StatementQueryMechanism.execute
>> NoSelect(StatementQueryMechanism.java:115)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(Dat
>> aModifyQuery.java:85)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java
>> :664)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(Databa
>> seQuery.java:583)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecute
>> Query(UnitOfWorkImpl.java:2750)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(A
>> bstractSession.java:1181)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(A
>> bstractSession.java:1165)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.mappings.ManyToManyMapping.insertIntoRelationTab
>> le(ManyToManyMapping.java:852)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.mappings.ManyToManyMapping.performDataModificati
>> onEvent(ManyToManyMapping.java:1010)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjects
>> WithChangeSet(CommitManager.java:130)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObject
>> sWithChangeSet(AbstractSession.java:3176)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabas
>> e(UnitOfWorkImpl.java:1299)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabas
>> eWithChangeSet(UnitOfWorkImpl.java:1399)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitRootUnitO
>> fWork(UnitOfWorkImpl.java:1230)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commit(UnitOfWo
>> rkImpl.java:1008)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.domain.handler.PilDataObjectHandlerImpl.commitU
>> nitOfWork(PilDataObjectHandlerImpl.java:329)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.domain.handler.PilDataObjectHandlerImpl.commitC
>> hanges(PilDataObjectHandlerImpl.java:174)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.util.ToplinkUtils.commitChanges(ToplinkUtils.ja
>> va:414)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.facade.BaseWebFacade.commitWebTransaction(BaseW
>> ebFacade.java:66)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.facade.BaseWebFacade.handleWebTransaction(BaseW
>> ebFacade.java:92)
>>>
>>> at
>>>
>>
>
nl.portbase.road.facade.RoadPlanningWebFacade.save(RoadPlanningWebFacade
>> .java:153)
>>>
>>> at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>>
>>> at
>>>
>>
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>> a:39)
>>>
>>> at
>>>
>>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>
>>> at
>>>
>>
>
nl.portinfolink.portal.actions.ServiceProcessWebFacadeCaller.invokeWebFa
>> cadeMethod(ServiceProcessWebFacadeCaller.java:314)
>>>
>>> at
>>>
>>
>
nl.portinfolink.portal.actions.ServiceProcessWebFacadeCaller.execute(Ser
>> viceProcessWebFacadeCaller.java:268)
>>>
>>> "http-bio-8280-exec-53" - Thread t@166
>>>
>>> java.lang.Thread.State: BLOCKED
>>>
>>> at
>> oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1559)
>>>
>>> - waiting to lock <19740c7> (a
>> oracle.jdbc.driver.T4CConnection)
>>> owned by "http-bio-8280-exec-8" t@106
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.OracleStatementWrapper.close(OracleStatementWrapper.j
>> ava:94)
>>>
>>> at
>>>
>>
>
oracle.jdbc.driver.OraclePreparedStatementWrapper.close(OraclePreparedSt
>> atementWrapper.java:80)
>>>
>>> at
>>>
>>
>
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close(DelegatingStatemen
>> t.java:168)
>>>
>>> at
>>>
>>
>
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close(DelegatingStatemen
>> t.java:168)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.clearSt
>> atementCache(DatabaseAccessor.java:308)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconne
>> ct(DatabaseAccessor.java:1399)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incre
>> mentCallCount(DatasourceAccessor.java:300)
>>>
>>> - locked <5f3edd> (a
>>> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.begin
>> Transaction(DatasourceAccessor.java:233)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTran
>> saction(AbstractSession.java:398)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTran
>> saction(AbstractSession.java:387)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.sessions.server.ClientSession.basicBeginTransact
>> ion(ClientSession.java:127)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.AbstractSession.beginTransacti
>> on(AbstractSession.java:552)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.beginTransactio
>> n(UnitOfWorkImpl.java:509)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabas
>> e(UnitOfWorkImpl.java:1263)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabas
>> eWithChangeSet(UnitOfWorkImpl.java:1399)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitRootUnitO
>> fWork(UnitOfWorkImpl.java:1230)
>>>
>>> at
>>>
>>
>
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commit(UnitOfWo
>> rkImpl.java:1008)
>>>
>>> at
>>>
>>
>
nl.portinfolink.platform.domain.handler.PilDataObjectHandlerImpl.commitU
>> nitOfWork(PilDataObjectHandlerImpl.java:329)
>>>
>>> ...
>>>
>>> Greetings,
>>>
>>> Huub
>>>
>>>
>>>
>>> _______________________________________________
>>> eclipselink-users mailing list
>>> eclipselink-users@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>>>
>> _______________________________________________
>> eclipselink-users mailing list
>> eclipselink-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>>
>>
>>
>> _______________________________________________
>> eclipselink-users mailing list
>> eclipselink-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>>
> _______________________________________________
> eclipselink-users mailing list
> eclipselink-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> _______________________________________________
> eclipselink-users mailing list
> eclipselink-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>
_______________________________________________
eclipselink-users mailing list
eclipselink-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-users
Attachment:
threaddump-1370016129244.tdump
Description: threaddump-1370016129244.tdump