Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » Too many open files using embedded derby
Too many open files using embedded derby [message #385432] Tue, 03 February 2009 13:04 Go to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
Hi,

I went into "Too many open files" exceptions on "heavily" opening and
closing derby database (10.4.2.0) using eclipselink (1.0.1) in my tests
with using the derby in embedded mode (with most tests creating the
database from scratch themselves). On closing I do call the entitymanager
and entitymanagerfactory close methods (I even see related derby logging
messages: "undeploy_begin ...", "logout_successful: ...").

Did I miss something and do I have to do anything more to release
resources from JPA point of view?


I found posts about index creation on large databases. But in my tests
there is currently very few data, so I do not assume this to be the root
cause here.

I once increased the max open files limit in my linux box. But with
increasing number of tests (and therefore database creations/connections)
the error now occurs again.

Best regards and thanks for any suggestion,
JOhannes
Re: Too many open files using embedded derby [message #385436 is a reply to message #385432] Wed, 04 February 2009 08:39 Go to previous messageGo to next message
James Sutherland is currently offline James Sutherland
Messages: 1939
Registered: July 2009
Location: Ottawa, Canada
Senior Member

That is odd. How complex are your tests, are they multi-threaded? You
may have reached the limits of what an embedded database can do.

You can try decreasing the connection pool size in EclipseLink, or in your
DataSource if using one.

"eclipselink.jdbc.read-connections.max"="1"
"eclipselink.jdbc.write-connections.max"="1"

---
James
http://www.nabble.com/EclipseLink---Users-f26658.html


James : Wiki : Book : Blog : Twitter
Re: Too many open files using embedded derby [message #385437 is a reply to message #385436] Wed, 04 February 2009 12:08 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
James wrote:
> That is odd. How complex are your tests, are they multi-threaded? You
> may have reached the limits of what an embedded database can do.

The tests are rather simple: just enter some (<10 rows) to some tables
(~5) and test they are managed the expected way. I did not count the
number of tests but a rough estimation is at about 30 test methods opening
and closing (asured by using a common base test implementation verifying
it at the end) the database.
Yes, the database is accessed from multiple threads but there is only one
single entity manager instance active a time. I do no specific
synchronization on accessing the entity manager. Is there some required?


> You can try decreasing the connection pool size in EclipseLink, or in your
> DataSource if using one.

> "eclipselink.jdbc.read-connections.max"="1"
> "eclipselink.jdbc.write-connections.max"="1"

No change on behaviour.

We monitored with lsof the opened file by the test running user. It seems
that for *each* test method the file handles on .dat, .lck and .log are
kept open. So it seems a principle problem to me currently.

Until there is a better proposal I will now try to reproduce the failure
with a single as simple as possible repeated test to break it ...


Thanks for the response! Best regards,
JOhannes
Re: Too many open files using embedded derby [message #385438 is a reply to message #385437] Wed, 04 February 2009 12:55 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
Ok, I sorted this out a little bit more now: I collect all instances
creating/releasing entitymanager(factory) in a WeakHashMap (for verifying
at end of tests them all to be closed - initially without any cleanup
there, shame on me). Not collecting them there at all prevents from the
failure.

I guess the instances do not get garbage collected with being ref'ed in
the *WEAK* map (as there is no need from mem point of view). And releasing
the database resources is done on the objects' finalization only, not on
close calls.

Another observation: the error disappears only with removing the instance
from the map *before* entitymanager(factory) close calls. With doing it
afterwards the tests fail again.

Does this sound reasonable?


To me (with my current standing of jpa-beginners knowledge) this does not
look like expected behavior but a bug as the documentation on
EntityManager#close() states:

* Close the factory, releasing any resources that it holds.
* After a factory instance is closed, all methods invoked on
* it will throw an IllegalStateException, except for isOpen,
* which will return false. Once an EntityManagerFactory has
* been closed, all its entity managers are considered to be
* in the closed state.

And the resources seem not being released as "promised" here. Or am I
wrong with this?

The good for me: I can prevent from this very easy in my tests.
Though the bad: I see problems with this behavior might occur with an
application opening/closing database connections heavily and for some
reasons instances referencing entitymanager(factory) instances not being
garbage collected nearly immediately ...


Best regards,
Johannes
Re: Too many open files using embedded derby [message #385439 is a reply to message #385438] Wed, 04 February 2009 13:02 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
Sh..., I was too fast with this success story: I just overlooked the test
still failing (too late for me?) just looking slightly different: forget
nearly everything about my previous posting. I only keep with my opinion
of the api violation ...

Best regards,
JOhannes
Re: Too many open files using embedded derby [message #385440 is a reply to message #385439] Wed, 04 February 2009 13:54 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
I stripped down my test to a handable portion (running on a kubuntu 8.04
desktop with open files limits set to 2048):

public class JPARegistryBaseTest extends TestCase {

private static final File TEST_DIR;
static {
try {
TEST_DIR =
File.createTempFile(JPARegistryBaseTest.class.getName(), "");
TEST_DIR.delete();
TEST_DIR.mkdirs();
} catch (IOException e) {
throw new RuntimeException(e);
}
}

private static int fInstance;

private volatile EntityManagerFactory fEntityManagerFactory;
private volatile EntityManager fEntityManager;

private void open() throws Exception {
final boolean impliciteCreation = !TEST_DIR.exists();
final Map<String, String> dbProperties = new HashMap<String,
String>();
final String name = "input-registry";
final File registryDir = new File(TEST_DIR, "run-" + fInstance);
registryDir.mkdirs();
final String dbDir = new File(registryDir, name).getPath();
dbProperties.put(PersistenceUnitProperties.JDBC_URL, "jdbc:derby:"
+ dbDir + ";create=true");
dbProperties.put(PersistenceUnitProperties.DDL_GENERATION,
PersistenceUnitProperties.CREATE_ONLY);
if (impliciteCreation) {
dbProperties.put(PersistenceUnitProperties.DDL_GENERATION_MO DE,

PersistenceUnitProperties.DDL_BOTH_GENERATION);
} else {
dbProperties.put(PersistenceUnitProperties.DDL_GENERATION_MO DE,

PersistenceUnitProperties.DDL_SQL_SCRIPT_GENERATION);
}
dbProperties.put(PersistenceUnitProperties.APP_LOCATION, dbDir);
// any effect?
dbProperties.put(PersistenceUnitProperties.DEFAULT_APP_LOCAT ION,
dbDir); // affects ddl scripts location
dbProperties.put(PersistenceUnitProperties.LOGGING_FILE, dbDir +
"-derbydb.log");
dbProperties.put(PersistenceUnitProperties.CREATE_JDBC_DDL_F ILE,
".." + File.separator + name + "-create.jdbc");
dbProperties.put(PersistenceUnitProperties.DROP_JDBC_DDL_FIL E,
".." + File.separator + name + "-drop.jdbc");
fEntityManagerFactory =
Persistence.createEntityManagerFactory(name, dbProperties);
fEntityManager = fEntityManagerFactory.createEntityManager();
fEntityManager.setFlushMode(FlushModeType.COMMIT);
}

public void testDirectEntityManagerUsage() throws Exception {
fInstance++;
open();
}

@Override
protected void tearDown() throws Exception {
close();
}

private void close() {
if (fEntityManager != null) {
fEntityManager.close();
fEntityManager = null;
}
if (fEntityManagerFactory != null) {
fEntityManagerFactory.close();
fEntityManagerFactory = null;
}
}

public static Test suite() {
return new RepeatedTest(new TestSuite(JPARegistryBaseTest.class),
100);
}
}



persistence.xml looks like:

<persistence-unit name="input-registry"> <!-- name relates to
JPAPacketInputRegistry#DB_NAME -->

<provider>org.eclipse.persistence.jpa.PersistenceProvider</provider >

<class> ... a very simple entity class here ... </class>

<properties>
<property name="eclipselink.jdbc.driver"
value="org.apache.derby.jdbc.EmbeddedDriver"/>
<property name="eclipselink.target-database" value="Derby"/>
<property name="eclipselink.jdbc.user" value="" />
<property name="eclipselink.jdbc.password" value="" />
<property name="eclipselink.logging.level" value="INFO"/>
<property name="eclipselink.orm.throw.exceptions"
value="true"/>
<property name="eclipselink.jdbc.batch-writing" value="JDBC"/>
</properties>
</persistence-unit>


Enough for me today, best regards,
JOhannes
Re: Too many open files using embedded derby [message #385442 is a reply to message #385440] Thu, 05 February 2009 08:28 Go to previous messageGo to next message
James Sutherland is currently offline James Sutherland
Messages: 1939
Registered: July 2009
Location: Ottawa, Canada
Senior Member

Calling close() on the EntityManagerFactory will logout from the JDBC
connections. Try turning logging on finest to see what is going on, you
should see a logout call for each login. Please include the log.

We have run our suit of several 1,000 of tests on Derby, as well as
running the JPA TCK, and have not encountered this issue before, so their
must be something specific to what you are doing, or your environment.

What does the RepeatedTest do, is it multi-threaded? In general a single
EntityManager should not used by multiple threads, each
user/thread/transaction should have its own EntityManager, the
EntityManagerFactory should be shared by the entire server, and only
startup/shutdown with the server.

---
James


James : Wiki : Book : Blog : Twitter
Re: Too many open files using embedded derby [message #385443 is a reply to message #385442] Thu, 05 February 2009 09:03 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
James wrote:
> Calling close() on the EntityManagerFactory will logout from the JDBC
> connections. Try turning logging on finest to see what is going on, you
> should see a logout call for each login. Please include the log.

I do see the logout log message (even with the INFO message level), I
think I already mentioned in a previous post. I'm now in a hurry, I will
attach complete (*very* long) log output with FINEST later.


> We have run our suit of several 1,000 of tests on Derby, as well as
> running the JPA TCK, and have not encountered this issue before, so their
> must be something specific to what you are doing, or your environment.

For sure.
Perhaps because of doing this embedded?
Or because with each of the test run, a *new* database instance is
created. The problem does not become obvious with re-using the same
database in all tests. I tell it "not obvious" because in fact the problem
is then there, too, as the file handles are not released eighther until vm
shutdown - but this does not lead to a failure as no limit is violated
then.


> What does the RepeatedTest do, is it multi-threaded? In general a single
> EntityManager should not used by multiple threads, each
> user/thread/transaction should have its own EntityManager, the
> EntityManagerFactory should be shared by the entire server, and only
> startup/shutdown with the server.

Oh, it is a junit.extensions.RepeatedTest. So in the example I provided
the create/close of the database (without enetering any data, no
transaction involved) is run synchronously one after the other.
As I hopefully said already: with lsof we observe all database files
created due to implicite database creation are kept open - until the limit
is reached.

With using my sources provided (I left away the import statements, I
attach them below), adding the derby, eclipselink and junit (I use 3.8.1
here currently) dependent jars and filling with a very simple entity
implementation (I yesterday simply lacked time to extract one for myself)
you should be able to reproduce my problem immediately.


My first way of coming around this will be to re-use same database but
clear it completely on every test begin.

Next I would look if this problem occurs, too, with not using JPA layer in
between the embedded derby database (to identify responsibility of this
behavior).

Any other suggestions are of cause very welcome ;-).


Best regards,
JOhannes


import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.FlushModeType;
import javax.persistence.Persistence;

import junit.extensions.RepeatedTest;
import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;

import org.eclipse.persistence.config.PersistenceUnitProperties;

public class JPARegistryBaseTest extends TestCase {
...
Re: Too many open files using embedded derby [message #385444 is a reply to message #385443] Thu, 05 February 2009 12:17 Go to previous messageGo to next message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
I just implemented a test doing similar thing not using EclipseLink in the
middle but directly interfacing derby (as announced) - and it breaks with
same error. So IMHO I'm wrong here with my "complain" about this and will
have to go over to the derby guys.

Thanks for the feedback though and best regards,
JOhannes
Re: Too many open files using embedded derby [message #385446 is a reply to message #385444] Thu, 05 February 2009 14:36 Go to previous message
Johannes Stamminger is currently offline Johannes Stamminger
Messages: 17
Registered: July 2009
Junior Member
Just a short explanation for the behavior (for details see
http://www.nabble.com/forum/ViewPost.jtp?post=21857863): derby keeps
running after connection close. With the magic sequence

try {
DriverManager.getConnection("jdbc:derby:;shutdown=true");
throw new IllegalStateException("shutdown should have
thrown SQLException");
} catch (SQLException e) {
System.out.println("run " + fInstance + " db closed");
}

it is closed then really (documented in
http://db.apache.org/derby/docs/10.4/devguide/tdevdvlp20349. html).


So with regarding the EntityManagerFactory#close() API doc:

* Close the factory, releasing any resources that it holds.
* After a factory instance is closed, all methods invoked on
* it will throw an IllegalStateException, except for isOpen,
* which will return false. Once an EntityManagerFactory has
* been closed, all its entity managers are considered to be
* in the closed state.

the question for me is left: is this intended behaviour?


Best regards,
JOhannes
Previous Topic:EclipseLink at EclipseCon
Next Topic:EclipseLink cant find PersistenceUnit
Goto Forum:
  


Current Time: Sat Aug 23 09:40:11 EDT 2014

Powered by FUDForum. Page generated in 0.08247 seconds