[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [eclipselink-dev] questions while running JPA JUnit tests
|
Hi Dies,
I am glad you are starting to get some success.
some comments inline:
Dies Koper wrote:
Hi Tom,
I finally got this test suite to stop hanging and locking.
I ended up putting the transaction begin/commit statements here:
- In SchemaManager#createObject/dropObject(in the if
(shouldWriteToDatabase()) block).
Did you take a look at any of the other calls to shouldWriteToDatabase()? Will
they require transactions as well?
- In SequenceDefinition#createOnDatabase I put "checkIfExist(session)"
in a try-finally block, with a rollback in the finally clause, followed
by a beginTransaction. This was necessary because in the case of
sequence objects (not sequence tables), checkIfExist did a SELECT
nextval and createOnDatabase did a CREATE SEQUENCE, so I needed to
separate the DML from the DDL.
With that I still had problems with the global temporary table. A number
of tests in this suite rely on updateAll and deleteAll.
With global temporary tables disabled
(shouldAlwaysUseTempStorageForModifyAll() returns false), I get no
hangs/locks but I get an SQLException (see last week's e-mail "same
table in from clauses of query and subquery").
With global temporary tables implemented, creation of the table goes
fine, but the following INSERT fails because the table is "locked". I'm
not sure if I can (or should) put this table's create statement in a
transaction, I suppose this whole query could already be running in a
transaction.
I am surprised that Delete All and Update All queries are not running in
transactions already. Maybe we should try to isolate a sequence of tests that
cause the failure... (i.e. Does the first Delete all or Update All fail? Or, is
the temporary table already created by another Delete all or Update all? Do we
see the initial create demarked by transactions? Is there an error in the
initial create that causes issues?)
Another issue with Symfoware's temporary tables is that the table space
name for the table must be specified at creation time. It does not
default to the table space that creation of normal tables default to. So
we'd need a way for the user to define this table space.
There is a table passed into the writeCreateTempTableSql() method. Can we use
the table space from that table?
In what cases are updateAll/deleteAll required exactly? Are they
related to JPA 2.0 functions? Are they required for JPA 1.0
functionality too?
UpdateAll and DeleteAll are both JPA 1.0 features and covered in the JPA 1.0 TCK.
As I see no clean solution I wonder what the impact is of making this a
limitation for this platform for now.
Have you looked into Andrei's suggestion of Local temporary tables.
From an EclipseLink point of view, that is a possibility. It is just a matter
of figuring out what that means to you from a TCK point of view?
With the global table disabled, I pass 176 tests (72%) of
EntityManagerJUnitTestSuite. The 45 errors are all due to the problem
above. Good news I suppose.
I'll try some more suites tomorrow.
If I no longer run into locking issues, what will the final solution
look like? Can I add a method createObjectsInTransactions() to the DB
platforms, defaulting to false of course, true for Symfoware, that
begins/commits/rolls back transactions in the locations I described above?
That is likely the best way to address this. We could tweak the method name a
bit, and of course, check the other calls for shouldWriteToDatabase() to see if
they needed to be transactional as well.
-Tom
Thanks,
Dies
Dies Koper wrote:
Hi Tom,
I am trying to enclose DDL calls with transactions. Inside
schemaManager.createSequences() there are mixed DDL and DML (select
and inserts on the sequence table/object), so just I'm moving the
transaction calls deeper into the call stack.
The locking error I get now is from the SELECT statement on table
CMP3_ENTITYB_SEQ in the following call in SchemaManager#createObject.
databaseObjectDefinition.createOnDatabase(getSession());
I did put a getSession().beginTransaction(); before this call, so I'm
not sure yet what the problem is. I'll investigate a bit more and let
you know.
Thanks,
Dies
The idea is that we would add calls that begin and commit
transactions around table creation calls.
Lets see if it works before we design the final solution:
- Find the
org.eclipse.persistence.tools.schemaframework.TableCreator.replaceTables(session,
schemaManager) method
- Add transactional boundaries
public void
replaceTables(org.eclipse.persistence.sessions.DatabaseSession
session, SchemaManager schemaManager) {
session.beginTransaction();
replaceTablesAndConstraints(schemaManager, session);
schemaManager.createSequences();
session.commitTransaction();
}
- Run
org.eclipse.persistence.testing.tests.jpa.advanced.EntityManagerJUnitTestSuite
- Ensure your logging is at FINER or FINEST and look for log messages
like the following around the code that does DDL generation:
[EL Finer]:
ServerSession(16925102)--Connection(14004497)--Thread(Thread[main,5,main])--begin
transaction
<a whole lot of DDL here>
[EL Finer]:
ServerSession(16925102)--Connection(14004497)--Thread(Thread[main,5,main])--commit
transaction
- make sure there are no DDL statements outside of those boundries
- see if it makes any difference to your tests
Assuming we cannot solve the issue with transactions, the problem
really comes down to the session that is passed into the
constructors for SchemaManager. That is the session used to
execute DDL. Changing the API is likely the cleanest way to do
that, but I am a bit concerned about the number of files we will
have to change to achieve that. I'll have to give some thought to
a solution that combines a reasonable number of changes to test
files with logical API.
I am still not sure what is causing the hangs and lock errors, so if
you could first suggest some temporal changes to a subset of tests
that I can apply and try to see if that solves all issues, that
would be great.
We may be able to add a temporary solution by creating some kind
of static session that we use for DDL generation and using that
session instead of the ones passed into the SchemaManager
constructor. Then we can try logging it in and out as DDL calls
are made. I am a bit concerned about that effect that will have on
how long the tests take to run, but I'll try to do some
experimentation.
How do sessions relate to the JDBC connections?
I am hoping that we can find a DB transaction related solution since
a solution like that will fit much better into the EclipseLink schema
creation architecture.
In JPA, an EntityManager holds a construct called a ClientSession. A
ClientSession communicates with a construct called a ServerSession to
make us of connections (an EntityManagerFactory holds the
ServerSession). A ServerSession holds a number of pools of
connections. (for reading, for writing, for sequences)
I also took an initial look at the patch you submitted with the
core changes for Symforware integration. The code looks quite good
- you are certainly going in the right direction. I'll try to find
some time to get this code integrated and tested soon.
Thanks!
Have you also noticed my comments to your request for ideas on the
database certification?
http://wiki.eclipse.org/EclipseLink/Development/Testing/DatabaseCertification
Thanks for posting your comments. You remarks about vendors
supplying database access is certainly one way we could go. The
other option we will consider is allowing vendors to certify
Platforms and documenting who provides the certification in our
documentation.
I've been experimenting to run the JPA tests using TestNG instead of
JUnit. Many work, but there are plenty of issues too to look at. If
you already know you don't want to go this direction, I'd prefer to
know soon before I spend too much time on it.
I have to admit that I am not particularly familiar with TestNG, but
any change in the way our testing framework works would likely have
to be carefully considered by the development team/community. If it
was clear that the amount of work required to either migrate or to
make some use of TestNG would provide more benefit than cost to the
community, it would be something we would consider. Any use of
TestNG would also depend on it's licensing terms being
Eclipse-compatible.
Having said that, I would be hesitant to make any promises about
whether and when we would consider it. Changes that require alot of
development work would have to be carefully scheduled considering
both the needs of the community and of the employers of the
developers that work on EclipseLink.
If you see some clear benefits it might be worth starting a thread on
either the users or dev mailing list pointing those benefits out.
-Tom
Thanks,
Dies
Tom Ware wrote:
Hi Dies,
I am going to try to take a more indepth look at this on Monday.
A couple of comments:
- For the issue with createTables() occurring at one stage in the
tests and not closing the connection, we will likely have to look
into how we could add some functionality to our schema framework
to ensure things happen this way. We have never run into a
database where it was required that the DDL connection be closed
before DML is run. I'll try to provide some more suggestions
about this on Monday.
- For the issue with temporary tables. It seems counter-intuitive
that a database only supports global temporary tables and then
does not allow cooexisting connections to access them. The whole
idea of a temporary table is that it is created on the fly, and
the idea of a global temporary table is that it is shared. Are
there any local temporary table options on Symfoware.
As I mention above, I'll try to put some more work into this on
monday and also have a look at the patch you have attached with
the core changes.
Have a great weekend,
Tom
Dies Koper wrote:
Hi Tom.
Thanks for your support.
It sounds like it is fairly easy to recreate the issue in
EntityManagerJunitTestSuite.
I have set up TestNG to run the tests so I can more easily run
single
tests without needing to change and recompile code.
I'm investigating the following test:
org.eclipse.persistence.testing.tests.jpa.advanced.EntityManagerJUnitTestSuite#testUpdateUsingTempStorage()
I've set up TestNG to call testSetup before it to prepare the
tables.
Using the driver's trace function, I can see that the tables
defined in testSetup are created in one connection, the another
connection is obtained from the driver to create the global
temporary table.
I then tries to access the temporary table through (I think) that
second connection, which fails because it says it's locked by
another user.
So I see two potential issues:
1. why is EclipseLink opening a second connection to create the
temporary table without closing the first one?
2. why can't we access the table we just created?
I'll try to reproduce the second problem in a standalone JDBC
test application and ask the Symfoware team.
Which line in testSequenceObjectsDefinition hangs? How about
testSequenceObjectsDefinition? When we know that, we can take a
look at what has happened previously in the tests suite to see
what could be causing the issue.
Next week I'll go through my logs to see where else the tests
hang before and try to run just that test (and the one before it).
There are some persistence unit properties for connection
configuration (including connection pools) in this section of
the doc that you could use to experiment.
I was hoping you could tell me a place where I can hard-code them
as I don't know what persistence.xml file is used (if any) when
running the JPA test suite.
You mention the following:
> On Symfoware, if you open a connection and create a table,
you can't use
> that table from other connections until the connection with
which the
> table was created is closed.
Is this just a "CREATE TABLE" issue or is it an issue with
other statements like UPDATE and DELETE? What about SELECT?
Does Symfoware
My understanding was that it applies to any DDL followed by DML.
tend to work when used with Connection Pools on application
servers?
I think so, but we usually prepare the tables in advance so there
is no DDL executed while in the J2EE container.
Regards,
Dies
Dies Koper wrote:
Hi Tom,
2. For the connection issues, some comments/questions
- I wonder if some early test failures cause corruption of the
connections. Do you see the connection problems as part of
your first error, or is it something else?
If I run the FullRegressionTestSuite, it almost always hangs
after it tried to drop sequence testSequenceObjectsDefinition.
It failed in the deletion of this sequence because it was
"locked by another user".
- Does your Symfoware database allow a limited number of
connections and if so, what is the limit?
I think there is a limit on the client side (not sure what it's
default is) and on the server side. On the server side I've
increased it twice from 256 to 356 to 456 but it still hangs at
the same spot.
When it is hanging, I cannot connect to the database through
the web-based admin console either, so the limit on the client
side seems irrelevant.
Is there a way I can see how many connections are used at a
certain time within EclipseLink? Some MBean, or a particular
variable I can look at in the debugger?
And is there a way to run the tests with EclipseLink's
connection pooling off?
I've tried setting the following in ConnectionPool but saw no
difference:
public static final int MAX_CONNECTIONS = 1;
public static final int MIN_CONNECTIONS = 0;
public static final int INITIAL_CONNECTIONS = 0;
public static final int WAIT_TIMEOUT = 10000; // 3 minutes.
Can they be specified in the test set somewhere in a single place?
- It sometimes helps to take the individual suites that are
listed in FullRegressionSuite and run them individually to
reduce the number of tests and see if the connectin problems
still occurs. Are you running FullRegressionTestSuite as a
whole, or are you running individual suites that are
components of FullRegressionTestSuite?
I have also tried running individual test suites.
EntityManagerJUnitTestSuite hangs in the same place as when I
run FullRegressionTestSuite. The error messages before it
complain about the temporary table being locked by another user
(testSequenceObjectsDefinition).
All logged SQLExceptions before this are expected errors (DROP
statements that fail because I start with a clean DB,
sequence's next_val failing because the sequence hasn't been
created yet).
I have heard that Symfoware focussed on data integrity so it
locks a lot. We have seen the following issue before:
On Symfoware, if you open a connection and create a table, you
can't use that table from other connections until the
connection with which the table was created is closed.
At least that was my understanding of it. So if I could ensure
EclipseLink uses only one connection, it should stop
complaining about other users using tables..
Thanks,
Dies
Tom Ware wrote:
Hi Dies,
I am hoping to get chanes for the
supportsPrimaryKeyConstraint() and the majority of the SQL
reserved word issues checked in fairly soon. There may be
some SQL reserved word issues we have to deal with separately
(mostly with SEQUENCE). I'll update the bug when I check-in
with the current state.
Does Symfoware support outer join in the Where clause? We
have support for writing SQL statements in an alternate way
for database platforms that have that support. Have a look
at the operatorOuterJoin() and shouldUseJDBCOuterJoinSyntax()
and shouldPrintOuterJoinInWhereClause() methods in various
database platforms. (e.g. OraclePlatform)
-Tom
Dies Koper wrote:
Hi Tom,
Why does it check supportsPrimaryKeyConstraint() and not
supportsUniqueKeyConstraint()?
It is possible, this is a legacy issue. (i.e. from before
we had platform support for unique key constraints) Please
file a bug.
Done!
https://bugs.eclipse.org/bugs/show_bug.cgi?id=289403
2. Check-in: bug 289019, bug 289020, bug 289021 and bug
289023
You merged the patches into one and marked the bugs as
fixed. Have
you committed the patch? I couldn't find it in trunk.
I am not sure what happened to my check-in. The changes
should be there now.
Thanks!
3. INNER JOIN
Symfoware does not support (INNER) JOIN syntax.
"JOIN" is used in SQLSelectStatement twice, at least
one of which is
used when using a self-referencing many to many
relationship and
performing a (JPQL) LEFT OUTER JOIN query. I can prevent
the syntax
error by replacing the inner join by a left outer join for
Symfoware,
but I'm worried that might select too many rows. I suppose
a better
solution is to compare join keys in the where clause. What
do you think?
Do you happen to know an easy way to do that (using
existing code)? If
not, never mind, I'll do some more research myself.
Could you please give an example of a statement that gives
you an issue, what the resulting SQL is, and what part of
it is not supported by Symfoware?
Query query = em.createQuery("SELECT o from Person o LEFT
OUTER JOIN
o.persons AS p");
gives:
SELECT t0.ID, t0.NAME, t1.ID, t1.NAME FROM {oj PERSON t0
LEFT OUTER JOIN
(PERSON_PERSON t2 JOIN PERSON t1 ON (t1.ID = t2.persons_ID)) ON
(t2.Person_ID = t0.ID)}
Symfoware complains about "JOIN", saying that the join type
is not
specified. Legal values are LEFT/RIGHT (OUTER).
So I suppose I should change it to generate something like:
SELECT t0.ID, t0.NAME, t1.ID, t1.NAME FROM {oj PERSON t0
LEFT OUTER JOIN
(SELECT PERSON_PERSON t2, PERSON t1 WHERE t1.ID =
t2.persons_ID) ON
(t2.Person_ID = t0.ID)}
4. Patches.
Start by attaching the single patch to the Symfoware
platform bug and describing the issues. If any of the
changes are large enough to merit their own bug, we can add
the bug later.
Done!
https://bugs.eclipse.org/bugs/show_bug.cgi?id=288715
From your previous email:
1. --
LANGUAGE, VALUE, DOMAIN, SEQUENCE
- I am still hoping to find the time to deal with these
issues.
I have enclosed most if not all occurrences in double-quotes
in the patch I attached to the bug for SymfowarePlatform.
That worked for me with Symfoware, and I have also confirmed
with Derby that none of my changes caused new failures on
Derby.
56% of 1270 tests (I temporarily disabled some) are
currently passing.
I will document some of the limitations in this platform I
found on the Wiki tomorrow.
The biggest issue I still have is that I get errors saying
that the resource (table, etc.) is locked by another user,
or sometimes it even hangs on getConnection. I checked with
the JDBC driver's trace info and found the number of calls
to getConnection to be much higher than the number of calls
to close(). I assume that this could be related. I don't
know whether connections were not closed because cleanup
processing did not complete for failing JUnit tests, or
something else is going on, but if you have any suggestions
I'm all ears.
What I'll try first is to disable ElipseLink connection
pooling and see how that affects it.
I also hope you can find some time to review the patches I
attached to the bug. You might see something obviously wrong
or missing that can help boost the passing rate when fixed.
Thanks!
Dies
_______________________________________________
eclipselink-dev mailing list
eclipselink-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-dev