Re: [eclipselink-dev] questions while running JPA JUnit tests

[Tom Ware <tom.ware@xxxxxxxxxx>]

Hi Dies,

  Some comments inline:

Dies Koper wrote:
> Hi Tom,
>
> >   I have been trying to find a solution for the issue with 
> > createTables.  The reason is is so tricky is that table creation is 
> > quite fundamental to the way we have designed our tests - test suites 
> > clean up prior to running by dropping and recreating tables.
>
>  From my trials with JDBC I have gotten a different understanding of the 
> issue with Symfoware. Well, 'understanding' is not the right word, I 
> certainly don't understand it yet, but what I'm seeing might be 
> different from what I thought (and explained here) was happening.
>
> I now have the impression that creating/dropping tables should be fine 
> as long as 1) the following access to the tables is done in the same 
> connection, or, 2) the connection is closed before the tables are 
> accessed, or, 3. what you are suggesting here below:
>
> >    Is there any chance that simply demarking table DDL execution with 
> > transactions will help?  (i.e. if make make sure all DDL is executed 
> > within a database transaction and that transaction is closed when it 
> > is complete, will that help?)
>
>  From what I observed, yes, that should help.
> If you could tell me how to do that I'd like to give it a try.

The idea is that we would add calls that begin and commit transactions around 
table creation calls.

Lets see if it works before we design the final solution:

- Find the 
org.eclipse.persistence.tools.schemaframework.TableCreator.replaceTables(session, 
schemaManager) method
- Add transactional boundaries

    public void replaceTables(org.eclipse.persistence.sessions.DatabaseSession 
session, SchemaManager schemaManager) {
        session.beginTransaction();
        replaceTablesAndConstraints(schemaManager, session);

        schemaManager.createSequences();
        session.commitTransaction();

    }

- Run org.eclipse.persistence.testing.tests.jpa.advanced.EntityManagerJUnitTestSuite
- Ensure your logging is at FINER or FINEST and look for log messages like the 
following around the code that does DDL generation:

[EL Finer]: 
ServerSession(16925102)--Connection(14004497)--Thread(Thread[main,5,main])--begin 
transaction

<a whole lot of DDL here>

[EL Finer]: 
ServerSession(16925102)--Connection(14004497)--Thread(Thread[main,5,main])--commit 
transaction

- make sure there are no DDL statements outside of those boundries
- see if it makes any difference to your tests

> >   Assuming we cannot solve the issue with transactions, the problem 
> > really comes down to the session that is passed into the constructors 
> > for SchemaManager.  That is the session used to execute DDL. Changing 
> > the API is likely the cleanest way to do that, but I am a bit 
> > concerned about the number of files we will have to change to achieve 
> > that.  I'll have to give some thought to a solution that combines a 
> > reasonable number of changes to test files with logical API.
>
> I am still not sure what is causing the hangs and lock errors, so if you 
> could first suggest some temporal changes to a subset of tests that I 
> can apply and try to see if that solves all issues, that would be great.
>
> >   We may be able to add a temporary solution by creating some kind of 
> > static session that we use for DDL generation and using that session 
> > instead of the ones passed into the SchemaManager constructor.  Then 
> > we can try logging it in and out as DDL calls are made.  I am a bit 
> > concerned about that effect that will have on how long the tests take 
> > to run, but I'll try to do some experimentation.
>
> How do sessions relate to the JDBC connections?

I am hoping that we can find a DB transaction related solution since a solution 
like that will fit much better into the EclipseLink schema creation architecture.

In JPA, an EntityManager holds a construct called a ClientSession.  A 
ClientSession communicates with a construct called a ServerSession to make us of 
connections (an EntityManagerFactory holds the ServerSession).  A ServerSession 
holds a number of pools of connections. (for reading, for writing, for sequences)

> >   I also took an initial look at the patch you submitted with the core 
> > changes for Symforware integration.  The code looks quite good - you 
> > are certainly going in the right direction.  I'll try to find some 
> > time to get this code integrated and tested soon.
>
> Thanks!
>
> Have you also noticed my comments to your request for ideas on the 
> database certification?
> http://wiki.eclipse.org/EclipseLink/Development/Testing/DatabaseCertification 

Thanks for posting your comments.  You remarks about vendors supplying database 
access is certainly one way we could go.  The other option we will consider is 
allowing vendors to certify Platforms and documenting who provides the 
certification in our documentation.

> I've been experimenting to run the JPA tests using TestNG instead of 
> JUnit. Many work, but there are plenty of issues too to look at. If you 
> already know you don't want to go this direction, I'd prefer to know 
> soon before I spend too much time on it.

I have to admit that I am not particularly familiar with TestNG, but any change 
in the way our testing framework works would likely have to be carefully 
considered by the development team/community.  If it was clear that the amount 
of work required to either migrate or to make some use of TestNG would provide 
more benefit than cost to the community, it would be something we would 
consider.  Any use of TestNG would also depend on it's licensing terms being 
Eclipse-compatible.

Having said that, I would be hesitant to make any promises about whether and 
when we would consider it.  Changes that require alot of development work would 
have to be carefully scheduled considering both the needs of the community and 
of the employers of the developers that work on EclipseLink.

If you see some clear benefits it might be worth starting a thread on either the 
users or dev mailing list pointing those benefits out.

-Tom

> Thanks,
> Dies
>
>
> > Tom Ware wrote:
> > > Hi Dies,
> > >
> > >   I am going to try to take a more indepth look at this on Monday.
> > >
> > >   A couple of comments:
> > >
> > > - For the issue with createTables() occurring at one stage in the 
> > > tests and not closing the connection, we will likely have to look 
> > > into how we could add some functionality to our schema framework to 
> > > ensure things happen this way.  We have never run into a database 
> > > where it was required that the DDL connection be closed before DML is 
> > > run.  I'll try to provide some more suggestions about this on Monday.
> > >
> > > - For the issue with temporary tables.  It seems counter-intuitive 
> > > that a database only supports global temporary tables and then does 
> > > not allow cooexisting connections to access them.  The whole idea of 
> > > a temporary table is that it is created on the fly, and the idea of a 
> > > global temporary table is that it is shared.  Are there any local 
> > > temporary table options on Symfoware.
> > >
> > > As I mention above, I'll try to put some more work into this on 
> > > monday and also have a look at the patch you have attached with the 
> > > core changes.
> > >
> > > Have a great weekend,
> > > Tom
> > >
> > > Dies Koper wrote:
> > > > Hi Tom.
> > > >
> > > > Thanks for your support.
> > > >
> > > > >   It sounds like it is fairly easy to recreate the issue in 
> > > > > EntityManagerJunitTestSuite.
> > > >
> > > > I have set up TestNG to run the tests so I can more easily run single
> > > > tests without needing to change and recompile code.
> > > >
> > > > I'm investigating the following test:
> > > > org.eclipse.persistence.testing.tests.jpa.advanced.EntityManagerJUnitTestSuite#testUpdateUsingTempStorage() 
> > > >
> > > >
> > > > I've set up TestNG to call testSetup before it to prepare the tables.
> > > >
> > > > Using the driver's trace function, I can see that the tables defined 
> > > > in testSetup are created in one connection, the another connection 
> > > > is obtained from the driver to create the global temporary table.
> > > > I then tries to access the temporary table through (I think) that 
> > > > second connection, which fails because it says it's locked by 
> > > > another user.
> > > >
> > > > So I see two potential issues:
> > > > 1. why is EclipseLink opening a second connection to create the 
> > > > temporary table without closing the first one?
> > > > 2. why can't we access the table we just created?
> > > >
> > > > I'll try to reproduce the second problem in a standalone JDBC test 
> > > > application and ask the Symfoware team.
> > > >
> > > > >   Which line in testSequenceObjectsDefinition hangs?  How about 
> > > > > testSequenceObjectsDefinition?  When we know that, we can take a 
> > > > > look at what has happened previously in the tests suite to see what 
> > > > > could be causing the issue.
> > > >
> > > > Next week I'll go through my logs to see where else the tests hang 
> > > > before and try to run just that test (and the one before it).
> > > >
> > > > >   There are some persistence unit properties for connection 
> > > > > configuration (including connection pools) in this section of the 
> > > > > doc that you could use to experiment.
> > > >
> > > > I was hoping you could tell me a place where I can hard-code them as 
> > > > I don't know what persistence.xml file is used (if any) when running 
> > > > the JPA test suite.
> > > >
> > > > >   You mention the following:
> > > > >
> > > > >  > On Symfoware, if you open a connection and create a table, you 
> > > > > can't use
> > > > >  > that table from other connections until the connection with 
> > > > > which the
> > > > >  > table was created is closed.
> > > > >
> > > > >   Is this just a "CREATE TABLE" issue or is it an issue with other 
> > > > > statements like UPDATE and DELETE?  What about SELECT?  Does Symfoware 
> > > >
> > > > My understanding was that it applies to any DDL followed by DML.
> > > >
> > > > > tend to work when used with Connection Pools on application servers?
> > > >
> > > > I think so, but we usually prepare the tables in advance so there is 
> > > > no DDL executed while in the J2EE container.
> > > >
> > > > Regards,
> > > > Dies
> > > >
> > > >
> > > > > Dies Koper wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > > 2. For the connection issues, some comments/questions
> > > > > > >
> > > > > > > - I wonder if some early test failures cause corruption of the 
> > > > > > > connections.  Do you see the connection problems as part of your 
> > > > > > > first error, or is it something else?
> > > > > >
> > > > > > If I run the FullRegressionTestSuite, it almost always hangs after 
> > > > > > it tried to drop sequence testSequenceObjectsDefinition. It failed 
> > > > > > in the deletion of this sequence because it was "locked by another 
> > > > > > user".
> > > > > >
> > > > > > > - Does your Symfoware database allow a limited number of 
> > > > > > > connections and if so, what is the limit?
> > > > > >
> > > > > > I think there is a limit on the client side (not sure what it's 
> > > > > > default is) and on the server side. On the server side I've 
> > > > > > increased it twice from 256 to 356 to 456 but it still hangs at 
> > > > > > the same spot.
> > > > > > When it is hanging, I cannot connect to the database through the 
> > > > > > web-based admin console either, so the limit on the client side 
> > > > > > seems irrelevant.
> > > > > >
> > > > > > Is there a way I can see how many connections are used at a 
> > > > > > certain time within EclipseLink? Some MBean, or a particular 
> > > > > > variable I can look at in the debugger?
> > > > > >
> > > > > > And is there a way to run the tests with EclipseLink's connection 
> > > > > > pooling off?
> > > > > > I've tried setting the following in ConnectionPool but saw no 
> > > > > > difference:
> > > > > >
> > > > > >     public static final int MAX_CONNECTIONS = 1;
> > > > > >     public static final int MIN_CONNECTIONS = 0;
> > > > > >     public static final int INITIAL_CONNECTIONS = 0;
> > > > > >     public static final int WAIT_TIMEOUT = 10000; // 3 minutes.
> > > > > >
> > > > > > Can they be specified in the test set somewhere in a single place?
> > > > > >
> > > > > > > - It sometimes helps to take the individual suites that are 
> > > > > > > listed in FullRegressionSuite and run them individually to reduce 
> > > > > > > the number of tests and see if the connectin problems still 
> > > > > > > occurs.  Are you running FullRegressionTestSuite as a whole, or 
> > > > > > > are you running individual suites that are components of 
> > > > > > > FullRegressionTestSuite?
> > > > > >
> > > > > > I have also tried running individual test suites.
> > > > > > EntityManagerJUnitTestSuite hangs in the same place as when I run 
> > > > > > FullRegressionTestSuite. The error messages before it complain 
> > > > > > about the temporary table being locked by another user 
> > > > > > (testSequenceObjectsDefinition).
> > > > > > All logged SQLExceptions before this are expected errors (DROP 
> > > > > > statements that fail because I start with a clean DB, sequence's 
> > > > > > next_val failing because the sequence hasn't been created yet).
> > > > > >
> > > > > > I have heard that Symfoware focussed on data integrity so it locks 
> > > > > > a lot. We have seen the following issue before:
> > > > > >
> > > > > > On Symfoware, if you open a connection and create a table, you 
> > > > > > can't use that table from other connections until the connection 
> > > > > > with which the table was created is closed.
> > > > > >
> > > > > > At least that was my understanding of it. So if I could ensure 
> > > > > > EclipseLink uses only one connection, it should stop complaining 
> > > > > > about other users using tables..
> > > > > >
> > > > > > Thanks,
> > > > > > Dies
> > > > > >
> > > > > > > Tom Ware wrote:
> > > > > > > > Hi Dies,
> > > > > > > >
> > > > > > > >   I am hoping to get chanes for the 
> > > > > > > > supportsPrimaryKeyConstraint() and the majority of the SQL 
> > > > > > > > reserved word issues checked in fairly soon.  There may be some 
> > > > > > > > SQL reserved word issues we have to deal with separately (mostly 
> > > > > > > > with SEQUENCE).  I'll update the bug when I check-in with the 
> > > > > > > > current state.
> > > > > > > >
> > > > > > > >   Does Symfoware support outer join in the Where clause?  We 
> > > > > > > > have support for writing SQL statements in an alternate way for 
> > > > > > > > database platforms that have that support.  Have a look at the 
> > > > > > > > operatorOuterJoin() and shouldUseJDBCOuterJoinSyntax() and 
> > > > > > > > shouldPrintOuterJoinInWhereClause() methods in various database 
> > > > > > > > platforms.  (e.g. OraclePlatform)
> > > > > > > >
> > > > > > > > -Tom
> > > > > > > >
> > > > > > > > Dies Koper wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > > >    Why does it check supportsPrimaryKeyConstraint() and not
> > > > > > > > > > > supportsUniqueKeyConstraint()?
> > > > > > > > > >
> > > > > > > > > > It is possible, this is a legacy issue.  (i.e. from before we 
> > > > > > > > > > had platform support for unique key constraints)  Please file 
> > > > > > > > > > a bug.
> > > > > > > > >
> > > > > > > > > Done!
> > > > > > > > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=289403
> > > > > > > > >
> > > > > > > > > > > 2. Check-in:  bug 289019, bug 289020, bug 289021 and bug 289023
> > > > > > > > > > >    You merged the patches into one and marked the bugs as 
> > > > > > > > > > > fixed. Have
> > > > > > > > > > > you committed the patch? I couldn't find it in trunk.
> > > > > > > > > >
> > > > > > > > > > I am not sure what happened to my check-in.  The changes 
> > > > > > > > > > should be there now.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > > > 3. INNER JOIN
> > > > > > > > > > >    Symfoware does not support (INNER) JOIN syntax.
> > > > > > > > > > >    "JOIN" is used in SQLSelectStatement twice, at least one 
> > > > > > > > > > > of which is
> > > > > > > > > > > used when using a self-referencing many to many relationship and
> > > > > > > > > > > performing a (JPQL) LEFT OUTER JOIN query. I can prevent the 
> > > > > > > > > > > syntax
> > > > > > > > > > > error by replacing the inner join by a left outer join for 
> > > > > > > > > > > Symfoware,
> > > > > > > > > > > but I'm worried that might select too many rows. I suppose a 
> > > > > > > > > > > better
> > > > > > > > > > > solution is to compare join keys in the where clause. What do 
> > > > > > > > > > > you think?
> > > > > > > > > > > Do you happen to know an easy way to do that (using existing 
> > > > > > > > > > > code)? If
> > > > > > > > > > > not, never mind, I'll do some more research myself.
> > > > > > > > > >
> > > > > > > > > > Could you please give an example of a statement that gives you 
> > > > > > > > > > an issue, what the resulting SQL is, and what part of it is 
> > > > > > > > > > not supported by Symfoware?
> > > > > > > > >
> > > > > > > > > Query query = em.createQuery("SELECT o from Person o LEFT OUTER 
> > > > > > > > > JOIN
> > > > > > > > > o.persons AS p");
> > > > > > > > >
> > > > > > > > > gives:
> > > > > > > > >
> > > > > > > > > SELECT t0.ID, t0.NAME, t1.ID, t1.NAME FROM {oj PERSON t0 LEFT 
> > > > > > > > > OUTER JOIN
> > > > > > > > > (PERSON_PERSON t2 JOIN PERSON t1 ON (t1.ID = t2.persons_ID)) ON
> > > > > > > > > (t2.Person_ID = t0.ID)}
> > > > > > > > >
> > > > > > > > > Symfoware complains about "JOIN", saying that the join type is not
> > > > > > > > > specified. Legal values are LEFT/RIGHT (OUTER).
> > > > > > > > >
> > > > > > > > > So I suppose I should change it to generate something like:
> > > > > > > > >
> > > > > > > > > SELECT t0.ID, t0.NAME, t1.ID, t1.NAME FROM {oj PERSON t0 LEFT 
> > > > > > > > > OUTER JOIN
> > > > > > > > > (SELECT PERSON_PERSON t2, PERSON t1 WHERE t1.ID = 
> > > > > > > > > t2.persons_ID) ON
> > > > > > > > > (t2.Person_ID = t0.ID)}
> > > > > > > > >
> > > > > > > > > > > 4. Patches.
> > > > > > > > > > Start by attaching the single patch to the Symfoware platform 
> > > > > > > > > > bug and describing the issues.  If any of the changes are 
> > > > > > > > > > large enough to merit their own bug, we can add the bug later.
> > > > > > > > >
> > > > > > > > > Done!
> > > > > > > > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=288715
> > > > > > > > >
> > > > > > > > > >  From your previous email:
> > > > > > > > > >
> > > > > > > > > > 1. --
> > > > > > > > > >
> > > > > > > > > > LANGUAGE, VALUE, DOMAIN, SEQUENCE
> > > > > > > > > >
> > > > > > > > > > - I am still hoping to find the time to deal with these issues.
> > > > > > > > >
> > > > > > > > > I have enclosed most if not all occurrences in double-quotes in 
> > > > > > > > > the patch I attached to the bug for SymfowarePlatform.
> > > > > > > > > That worked for me with Symfoware, and I have also confirmed 
> > > > > > > > > with Derby that none of my changes caused new failures on Derby.
> > > > > > > > >
> > > > > > > > > 56% of 1270 tests (I temporarily disabled some) are currently 
> > > > > > > > > passing.
> > > > > > > > > I will document some of the limitations in this platform I 
> > > > > > > > > found on the Wiki tomorrow.
> > > > > > > > >
> > > > > > > > > The biggest issue I still have is that I get errors saying that 
> > > > > > > > > the resource (table, etc.) is locked by another user, or 
> > > > > > > > > sometimes it even hangs on getConnection. I checked with the 
> > > > > > > > > JDBC driver's trace info and found the number of calls to 
> > > > > > > > > getConnection to be much higher than the number of calls to 
> > > > > > > > > close(). I assume that this could be related. I don't know 
> > > > > > > > > whether connections were not closed because cleanup processing 
> > > > > > > > > did not complete for failing JUnit tests, or something else is 
> > > > > > > > > going on, but if you have any suggestions I'm all ears.
> > > > > > > > > What I'll try first is to disable ElipseLink connection pooling 
> > > > > > > > > and see how that affects it.
> > > > > > > > >
> > > > > > > > > I also hope you can find some time to review the patches I 
> > > > > > > > > attached to the bug. You might see something obviously wrong or 
> > > > > > > > > missing that can help boost the passing rate when fixed.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Dies
>
> _______________________________________________
> eclipselink-dev mailing list
> eclipselink-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-dev