Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » Eclipselink 1.1.3 query hang up
Eclipselink 1.1.3 query hang up [message #503648] Tue, 15 December 2009 08:34 Go to next message
Thomas Haskes is currently offline Thomas HaskesFriend
Messages: 147
Registered: July 2009
Senior Member
Hi all,

I'm experiencing some strange behaviour of Eclipselink.

I'm using Eclipselinks internal Connection Pool. After some time of
inactivity the first query that is executed hangs for some minutes. I
assume that this is related to some netork configuration in the
environment the application runs, as I cannot reproduce the behaviour in
any test environmant. Now I need you guys to help me find out what is
going wrong. I don't know where to start looking.

Here is what happens exactly:

Application is deployed and started - everything runs fine.

After some time (around 60 minutes) of inactivity (no user runs any
queries) the first query is executed. This query hangs for about 8
minutes before I see following output in the logs.

--Communication failure detected when attempting to perform read query
outside of a transaction. Attempting to retry query.

SELECT 1 FROM DUAL

This takes another 10 minutes in which nothing happens. After that I see
that a new Connection is opened and everything runs fine again until
there is another period of inactivity that is long enough.

I've tried to reproduce the behaviour in a test environment by shutting
down the network connection and I can see the same log output there, the
difference is that it takes EL almost no time to recognize the
connection is gone and to open a new one, there is no hang up at all.

Can anyone give me a hint on what I could do to avoid those hang ups, or
even where I could start looking. Any Information on what EL does
exactly when it recognizes a "communitcation failure" would be helpful,
too.

I think the log output "communication failure detected" is not helpful
at all, as it gives no information about what went wrong. Is there a way
to gather some more information on that error - I've already set the
log level to FINEST, but this is all I get.

I put the full log output here so that you guys can see what happens.


2009-12-14 13:06:21,180 INFO [STDOUT] [EL Finest]: 2009-12-14
13:06:21.18--UnitOfWork(6031650)--Thread(Thread[UIThread
[02FC066D1AFD678C30C
A38DC940CAA16],6,jboss])--Execute query DataReadQuery(sql="select
PCK_IM_WEB_USER.Pruefe_Anmeldung(?) from dual")
2009-12-14 13:06:21,180 INFO [STDOUT] [EL Fine]: 2009-12-14
13:06:21.18--ServerSession(29423625)--Connection(33259689)-- Thread(Thread[UIThr
ead [02FC066D1AFD678C30CA38DC940CAA16],6,jboss])--select
PCK_IM_WEB_USER.Pruefe_Anmeldung(?) from dual
bind => [[xxxxxx]

---- time is going by.... query hangs--------

###### the following is some server log noise ###########

2009-12-14 13:07:10,660 DEBUG
[org.jboss.remoting.transport.socket.ServerThread]
WorkerThread#0[172.18.64.95:59256] closing socketWrapper: S
erverSocketWrapper[Socket[addr=/172.18.64.95,port=59256,loca lport=4446].a6a8c0]
2009-12-14 13:07:58,579 DEBUG
[org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover
java:/DefaultJMSProvider
2009-12-14 13:07:58,580 DEBUG
[org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover
java:/DefaultJMSProvider
2009-12-14 13:10:08,585 DEBUG
[org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover


[...]


------------ here EL recognizes the communication problem ------
### after 7 minutes !!! #####

2009-12-14 13:14:37,807 INFO [STDOUT] [EL Config]: 2009-12-14
13:14:37.807--ServerSession(29423625)--Connection(33259689)- -Thread(Thread[UI
Thread [02FC066D1AFD678C30CA38DC940CAA16],6,jboss])--disconnect
2009-12-14 13:14:37,807 INFO [STDOUT] [EL Info]: 2009-12-14
13:14:37.807--UnitOfWork(6031650)--Thread(Thread[UIThread
[02FC066D1AFD678C30CA
38DC940CAA16],6,jboss])--Communication failure detected when attempting
to perform read query outside of a transaction. Attempting to retry
query.
2009-12-14 13:14:37,807 INFO [STDOUT] [EL Fine]: 2009-12-14
13:14:37.807--ServerSession(29423625)--Thread(Thread[UIThrea d
[02FC066D1AFD678C
30CA38DC940CAA16],6,jboss])--SELECT 1 FROM DUAL


----- again hanging... -----


..... server noise......

2009-12-14 13:15:10,056 DEBUG
[org.jboss.remoting.transport.socket.ServerThread]
WorkerThread#0[172.18.64.95:34622] closing socketWrapper: S
erverSocketWrapper[Socket[addr=/172.18.64.95,port=34622,loca lport=4446].190eb19]
2009-12-14 13:16:10,633 DEBUG
[org.jboss.remoting.transport.socket.ServerThread]
WorkerThread#0[172.18.64.95:38863] closing socketWrapper: S
erverSocketWrapper[Socket[addr=/172.18.64.95,port=38863,loca lport=4446].15966d0]
2009-12-14 13:16:38,603 DEBUG
[org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover
java:/DefaultJMSProvider
2009-12-14 13:16:38,603 DEBUG
[org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover
java:/DefaultJMSProvider





------ another 7 minutes later --------

2009-12-14 13:22:56,839 INFO [STDOUT] [EL Config]: 2009-12-14
13:22:56.839--ServerSession(29423625)--Connection(15986869)- -Thread(Thread[UI
Thread
[02FC066D1AFD678C30CA38DC940CAA16],6,jboss])--connecting(Dat abaseLogin(
platform=>Oracle10Platform
user name=> "xxxxxx"
datasource URL=> "xxxxxx"
))
2009-12-14 13:22:56,968 INFO [STDOUT] [EL Config]: 2009-12-14
13:22:56.968--ServerSession(29423625)--Connection(24726183)- -Thread(Thread[UI
Thread [02FC066D1AFD678C30CA38DC940CAA16],6,jboss])--Connected:
jdbc:oracle:thin:@imq2-db.ages.at:1521:imq2
User: xxxxxx
Database: Oracle Version: Oracle Database 10g Enterprise
Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing
options
Driver: Oracle JDBC driver Version: 11.1.0.7.0-Production
2009-12-14 13:22:56,969 INFO [STDOUT] [EL Fine]: 2009-12-14
13:22:56.968--ServerSession(29423625)--Connection(24726183)- -Thread(Thread[UITh
read [02FC066D1AFD678C30CA38DC940CAA16],6,jboss])--select
PCK_IM_WEB_USER.Pruefe_Anmeldung(?) from dual
bind => [[xxxxxx]
2009-12-14 13:22:56,975 INFO [STDOUT] [EL Finest]: 2009-12-14
13:22:56.975--UnitOfWork(6031650)--Thread(Thread[UIThread
[02FC066D1AFD678C30
CA38DC940CAA16],6,jboss])--Execute query



------- now everything works again --------

Any idea will be helpful, as I don't know what to do about it.

TIA

Tom
Re: Eclipselink 1.1.3 query hang up [message #503780 is a reply to message #503648] Tue, 15 December 2009 17:34 Go to previous messageGo to next message
James Sutherland is currently offline James SutherlandFriend
Messages: 1939
Registered: July 2009
Location: Ottawa, Canada
Senior Member

Looks like something is timing out your database connection. This is probably your firewall, but could be the database as well.

I would investigate your firewall settings.

EclipseLink automatically recovers from communication failures, but there is little we can do to diagnose the failure. We basically execute a query and have to wait for the JDBC driver, or DataSource's response, if it hangs for 10minutes, then we are waiting. You could set the JDBC timeout hint on your query, but whether the JDBC driver honors this is up to the driver. You should see the SQLException in your logs, provided your log level is warning or lower.

If you cannot fix your firewall, they you could change your connection pool settings to use a min number of connections of 0, so there will never be idle connections. But then you will be reconnecting a lot.

You could also use a third party connection pool or DataSource that supports a timeout setting on the pooled connections, or spawn a thread in your application the clears the connection pool every 30minutes.


James : Wiki : Book : Blog : Twitter
Re: Eclipselink 1.1.3 query hang up [message #503866 is a reply to message #503780] Wed, 16 December 2009 02:38 Go to previous messageGo to next message
Thomas Haskes is currently offline Thomas HaskesFriend
Messages: 147
Registered: July 2009
Senior Member
Hi James, thanks for all the hints, I'll try them out.

> You should see the SQLException in your logs,
> provided your log level is warning or lower.

That is the thing, there isn't logged any Exception although my log
level is FINEST. There is the "communication failure detected" message
only. Meanwhile I managed to find out that the Exception is a
SQLException that says "I/O Connection reset". But this Exception is
catched and not logged anywhere. I think this is a bug, as in a log
level of FINEST any Exception should be logged.


> If you cannot fix your firewall, they you could change your connection
> pool settings to use a min number of connections of 0, so there will
> never be idle connections. But then you will be reconnecting a lot.

I'll try that, reconnecting shouldn't be a problem (thats at least the
way I look at it but I need to find that out anyway).



> You could also use a third party connection pool or DataSource that
> supports a timeout setting on the pooled connections, or spawn a thread
> in your application the clears the connection pool every 30minutes.
>

This sounds reasonable, too, but I wonder what I had to do to clear the
connection pool. Could you provide a small snippet that shows how that
is done?

Thanks a lot

Tom
Re: Eclipselink 1.1.3 query hang up [message #504000 is a reply to message #503648] Wed, 16 December 2009 18:44 Go to previous messageGo to next message
James Sutherland is currently offline James SutherlandFriend
Messages: 1939
Registered: July 2009
Location: Ottawa, Canada
Senior Member

Looking at the code, it appears the exception is not logged. Please log a bug for this, it should be logged.

You could reconnect, or just execute a query on each connection.

serverSession.getDefaultConnectionPool().setCheckConnections ();

This will cause each of the connections to be pinged on the next connect. You may also need to do this to the readConnectionPool if using a different pool for reading.


James : Wiki : Book : Blog : Twitter
Re: Eclipselink 1.1.3 query hang up [message #504110 is a reply to message #504000] Thu, 17 December 2009 10:03 Go to previous messageGo to next message
Thomas Haskes is currently offline Thomas HaskesFriend
Messages: 147
Registered: July 2009
Senior Member
Hi James,

thanks for your help, setting the connection Pool to use min Connections
= 0 worked.


> Looking at the code, it appears the exception is not logged. Please log
> a bug for this, it should be logged.

Will be done shortly.

> You could reconnect, or just execute a query on each connection.
>
> serverSession.getDefaultConnectionPool().setCheckConnections ();
>
> This will cause each of the connections to be pinged on the next
> connect. You may also need to do this to the readConnectionPool if
> using a different pool for reading.
>

Right now I'm not sure if I can be happy with all the reconnects on the
long run. I have my client trying to figure out the firewall settings,
but I think I'm going to need a backup solution to make this work if
they don't. The problem is that I only worked with the EL Jpa
extensions, so I'm not quite familiar with all the sessions and
connectionpools stuff.

I think the best "Backup" solution for me would be to spawn a thread and
have executed a query on each connection as sort of a "keepalive". My
Problem is that I don't know how to execute a query on a certain
connection from the pool. I thought of something like this:

JpaEntityManager jpaEntityManager = JpaHelper.getEntityManager(em);
ServerSession serverSession = jpaEntityManager.getServerSession();
ConnectionPool connectionPool = serverSession.getConnectionPool("default");
Accessor accessor = connectionPool.acquireConnection();
Connection connection = accessor.getConnection();
Statement statement = connection.createStatement();
statement.execute("select something from somewhere");
statement.close();
connectionPool.releaseConnection(accessor);

But there are three things i'm unsure about.

1. How can I achive to aquire a CERTAIN Connection from the pool to be
sure that keep alive all the connection that it holds at minimum?

2. What happens if my keepalive-thread executes a statement at the exact
same time some user is using it. Could this lead to a hang up as well?
Or do I need to check if a connection is idle before I acquire it, and
how would that be done?

3. I had a look on the serversession, but I could not find anything
about a read-connection Pool, where do I get to that one?

As you see, I'm stuck here. Could you provide me with some more ideas
please?

Thanks again

Tom
Re: Eclipselink 1.1.3 query hang up [message #504175 is a reply to message #503648] Thu, 17 December 2009 15:06 Go to previous messageGo to next message
James Sutherland is currently offline James SutherlandFriend
Messages: 1939
Registered: July 2009
Location: Ottawa, Canada
Senior Member

1) just call, setCheckConnections() on the pool to have all connections pinged. Then the next time a connection is accessed, all the connections will be pinged, so you just need to access a connection.

JpaEntityManager jpaEntityManager = JpaHelper.getEntityManager(em);
ServerSession serverSession = jpaEntityManager.getServerSession();
ConnectionPool connectionPool = serverSession.getConnectionPool("default");
connectionPool.setCheckConnections();
Accessor accessor = connectionPool.acquireConnection();
connectionPool.releaseConnection(accessor);

2) The connection pool is thread safe, so not issue with threading.

3) serverSession.getReadConnectionPool(), by default this is now the same thing as the default connection pool, but can be a different pool if you configured it.



James : Wiki : Book : Blog : Twitter
Re: Eclipselink 1.1.3 query hang up [message #505809 is a reply to message #504175] Tue, 05 January 2010 08:10 Go to previous message
Thomas Haskes is currently offline Thomas HaskesFriend
Messages: 147
Registered: July 2009
Senior Member
Finally got the time to test it, and it works! BTW it turned out that
they had a 60 min timeout on the tcp stack.

Thanks a lot.

Tom



Am 17.12.2009 16:06, schrieb James:
> 1) just call, setCheckConnections() on the pool to have all connections
> pinged. Then the next time a connection is accessed, all the connections
> will be pinged, so you just need to access a connection.
>
> JpaEntityManager jpaEntityManager = JpaHelper.getEntityManager(em);
> ServerSession serverSession = jpaEntityManager.getServerSession();
> ConnectionPool connectionPool = serverSession.getConnectionPool("default");
> connectionPool.setCheckConnections();
> Accessor accessor = connectionPool.acquireConnection();
> connectionPool.releaseConnection(accessor);
>
> 2) The connection pool is thread safe, so not issue with threading.
>
> 3) serverSession.getReadConnectionPool(), by default this is now the
> same thing as the default connection pool, but can be a different pool
> if you configured it.
>
>
Previous Topic:Problem with sdo in OSGi bundle (classloader?)
Next Topic:Help! Weird EclipseLink exception
Goto Forum:
  


Current Time: Mon Nov 24 16:32:51 GMT 2014

Powered by FUDForum. Page generated in 0.03871 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software