Re: [eclipselink-users] Plot thickens v. 1.0.2 EntityManagerFactoryImpl

[Gordon Ferguson <ferguson@xxxxxxx>]

Hi Tom,
Short answer seems to be: the difference is how we
force the diconnection.
I have attached some files to document this.

Upshot is
(1) I was able to replicate your successful restart after an orderly 
stop/start of MySQL running locally.
(2) The restart does not succeed when network connectivity to a
remote oracle instance is broken and restored.
(3) The EMF variant, closing and recreating the entire factory,
has different, perhaps more informative, symptoms.

Attached are: the java test file, console output from the success,
console output from the failure, and console output from the
ORACLE failure using the EMF variant. The only difference in the java
code between the two was to swap persistent unit names.
In the mysql case I used the database manager to stop/start.
In the ORACLE case I disconnected/reconnnected the VPN we use.

The EMF failure had problems with closing the EMF -- it apparently
tried to access the database as part of the close() and failed. And
it apparently did not clear out the old EM:
Caused by: Exception [EclipseLink-28009] (Eclipse Persistence Services - 
1.0.2 (Build 20081024)): 
org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named 
file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod without closing it.

Thanks,
Gordon



Tom Ware wrote:
> Hi Gordon,
>
>   FYI: By default, EclipseLink will create a connection pool for you.  
> The settings you have below affect the connection limits, but do not 
> affect whether or not there is a connection pool.
>
>   I cannot recreate the issue you are having.  Can you tell me what is 
> different about what I am doing.
>
> I am running two simple tests on MySQL.  Here is the pseudo code:
>
> // This is the case where the whole EMF is recreated
>     public void testEMFCloseAndOpen(){
>         Employee emp = null;
>
>         // creates an EMF for PU default and then creates an EM from it
>         EntityManager em = createEntityManager();
>         em.find(Employee.class, 1);
>         try{
>             em.getTransaction().begin();
>             // shut down DB prior to this line
>             emp = new Employee();
>             em.persist(emp);
>             em.getTransaction().commit();
>
>         } catch (Exception e){
>             // get exception here after a number of retries
>             e.printStackTrace();
>         }
>         getEntityManagerFactory().close();
>         // restart the db prior to this line
>         em = createEntityManager();
>         em.getTransaction().begin();
>         emp = new Employee();
>         em.persist(emp);
>         em.getTransaction().commit();
>     }
>
> // this is the case where just the EM is recreated
>     public void testEMCloseAndOpen(){
>         Employee emp = null;
>         EntityManager em = createEntityManager();
>         em.find(Employee.class, 1);
>         try{
>             em.getTransaction().begin();
>             // shut down DB prior to this line
>             emp = new Employee();
>             em.persist(emp);
>             em.getTransaction().commit();
>         } catch (Exception e){
>             // get exception here after a number of retries
>             e.printStackTrace();
>         }
>         em.close();
>         // restart the db prior to this line
>         em = createEntityManager();
>         em.find(Employee.class, 1);
>         em.getTransaction().begin();
>         emp = new Employee();
>         em.persist(emp);
>         em.getTransaction().commit();
>     }
>
> -Tom
>
> Gordon Ferguson wrote:
> > Tom Ware wrote:
> > > Hi Gordon,
> > >
> > >   It sounds like you are using the default EclipseLink connection pool.
> > No doubt. I did try with
> > snip ...
> > emfProperties.put(JDBC_READ_CONNECTIONS_MIN, "1");
> > emfProperties.put(JDBC_WRITE_CONNECTIONS_MIN, "1");
> > // Ensure that no server-platform is configured
> > emfProperties.put(TARGET_SERVER, TargetServer.None);
> > etc
> > But the problem remained.
> >
> > >   Is there any particular reason you have chosen to close your 
> > > EntityManagerFactory?  What happens if you simply acquire a new 
> > > entityManager from the same factory?  Do you have problems with the 
> > > connection?
> > I did try some variations. Closing and recreating everything seemed
> > the most likely way to get a fresh start. Shot in the dark in other 
> > words.
> >
> > I have just (re)tried: close the EntityManager and get a new one from
> > the existing factory; the problem appears unchanged.
> >
> > One, perhaps interesting, note about the stack trace: EL logs
> > a successful login before throwing the null pointer exception.
> > I don't see the login success when the underlying network
> > connection is unavailable.
> >
> > Thanks again,
> > Gordon
> >
> > > -Tom
> > >
> > > Gordon Ferguson wrote:
> > > > Tom Ware wrote:
> > > > > Hi Gordon,
> > > > >
> > > > >   Are you running in JavaSE or in some kind of container?
> > > > JavaSE
> > > > >   Are you using a connection pool?
> > > > not as far as I know. Should I be?
> > > > >   Is it external to EclipseLink or EclipseLink's own connection pool?
> > > > n/a?
> > > > >   How does your test obtain the connection and reestablish it?
> > > >
> > > >  >>   emf = 
> > > > Persistence.createEntityManagerFactory(persistenceUnitLabel);
> > > >  >>   entityManager = emf.createEntityManager();
> > > >
> > > >
> > > > >   I am not surprised that the scenario you are describing is 
> > > > > causing issues. EclipseLink has some strategies you can use to 
> > > > > reestablish connections, but doing it somewhere EclipseLink is 
> > > > > completely unaware of could cause an issue.
> > > > So, what is the most simple solution? The actual application runs in 
> > > > Tomcat; database is sometimes Oracle sometimes MySQL. (The JNDI stuff
> > > > seems to multiply code complexity, do I really have to go there? )
> > > >
> > > > Many thanks,
> > > > Gordon
> > > >
> > > >
> > > > > -Tom
> > > > >
> > > > > Gordon Ferguson wrote:
> > > > > > Hi Tom,
> > > > > > It has been a few weeks but I'm back to this question.
> > > > > >
> > > > > > The basic scenario I'm having trouble with is recovery after a
> > > > > > SQLException: Io exception: The Network Adapter could not 
> > > > > > establish the connection ...
> > > > > >
> > > > > > My test case forces that error and then restores the underlying
> > > > > > connection. I then want to get EclipseLink going again.
> > > > > >
> > > > > > Is there an api to get a (new?) working EntityMangager at this point?
> > > > > >
> > > > > > What I'm trying is invoke close() both the EntityManager and the
> > > > > > EntityManagerFactory. (Nothing thrown by this.) Then I repeat this 
> > > > > > code:
> > > > > >
> > > > > > protected  void privateInit(String persistenceUnitLabel, ...) {
> > > > > >   emf = 
> > > > > > Persistence.createEntityManagerFactory(persistenceUnitLabel);          
> > > > > > entityManager = emf.createEntityManager();
> > > > > >   ...
> > > > > > }
> > > > > >
> > > > > > and the first line creates the stack trace below.
> > > > > >
> > > > > > Thanks for any suggestions,
> > > > > > Gordon
> > > > > >
> > > > > > Tom Ware wrote:
> > > > > > > Hi Gordon,
> > > > > > >
> > > > > > >   If EclipseLink is following the expected code path, the 
> > > > > > > structConverters should be initialized inside the "if (state == 
> > > > > > > STATE_PREDEPLOYED) {" code block below where the structConverters 
> > > > > > > variable is being defined on the following line:
> > > > > > >
> > > > > > > structConverters = processor.getStructConverters();
> > > > > > >
> > > > > > >   If you are not getting into that if statement and still getting 
> > > > > > > to "addStructConverters(session, structConverters)" something 
> > > > > > > strange is occurring with your deployment. It is hard to say what 
> > > > > > > the problem is without more information about the symptoms of the 
> > > > > > > problem you are seeing.
> > > > > > >
> > > > > > > -Tom
> > > > > > >
> > > > > > > Gordon Ferguson wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I'm using the SE approach and testing my 'recovery after error' 
> > > > > > > > logic.
> > > > > > > > I see the exception listed below. I looked at the code and found
> > > > > > > > EntityManagerSetupImpl.deploy() method initializes
> > > > > > > > structConverters to null in l.189
> > > > > > > > Then, in l. 319 passes that null to addStructConverters(...) 
> > > > > > > > which tries to add to the (null) List.
> > > > > > > >  List<StructConverter> structConverters = null;
> > > > > > > >
> > > > > > > > SO, I'm wondering if this is just a coding typo or if I need to 
> > > > > > > > dig deeper
> > > > > > > > into why I'm in this state?
> > > > > > > >
> > > > > > > > Any suggestions would be appreciated.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Gordon
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Listing:
> > > > > > > >
> > > > > > > > [EL Info]: 2008.10.30 
> > > > > > > > 11:53:03.781--ServerSession(653967)--EclipseLink, version: 
> > > > > > > > Eclipse Persistence Services - 1.0.2 (Build 20081024)
> > > > > > > > Exception in thread "main" [EL Info]: 2008.10.30 
> > > > > > > > 11:53:10.328--ServerSession(653967)--file:/D:/projs/costar-1.3/jclass/-True-DevDev 
> > > > > > > > login successful
> > > > > > > > costar.dbaccess.DbAccessException: Could not access database.
> > > > > > > >     at 
> > > > > > > > costar.dbaccess.jpa.CoStarJpaAccess.privateInit(CoStarJpaAccess.java:82) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > costar.dbaccess.jpa.CoStarJpaAccess.<init>(CoStarJpaAccess.java:53)
> > > > > > > >     at 
> > > > > > > > costar.dbaccess.jpa.test.JpaConnectionCheckerChecker.main(JpaConnectionCheckerChecker.java:104) 
> > > > > > > >
> > > > > > > > Caused by: java.lang.NullPointerException
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.addStructConverters(EntityManagerSetupImpl.java:319) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:249) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.getServerSession(EntityManagerFactoryImpl.java:69) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:118) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:112) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:100) 
> > > > > > > >
> > > > > > > >     at 
> > > > > > > > costar.dbaccess.jpa.CostarJpaFacade.<init>(CostarJpaFacade.java:47)
> > > > > > > >     at 
> > > > > > > > costar.dbaccess.jpa.CoStarJpaAccess.privateInit(CoStarJpaAccess.java:80) 
> > > > > > > >
> > > > > > > >     ... 2 more
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > eclipselink-users mailing list
> > > > > > > > eclipselink-users@xxxxxxxxxxx
> > > > > > > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > > > > > > _______________________________________________
> > > > > > > eclipselink-users mailing list
> > > > > > > eclipselink-users@xxxxxxxxxxx
> > > > > > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > > > > > _______________________________________________
> > > > > > eclipselink-users mailing list
> > > > > > eclipselink-users@xxxxxxxxxxx
> > > > > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > > > > _______________________________________________
> > > > > eclipselink-users mailing list
> > > > > eclipselink-users@xxxxxxxxxxx
> > > > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > > > _______________________________________________
> > > > eclipselink-users mailing list
> > > > eclipselink-users@xxxxxxxxxxx
> > > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > > _______________________________________________
> > > eclipselink-users mailing list
> > > eclipselink-users@xxxxxxxxxxx
> > > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> > _______________________________________________
> > eclipselink-users mailing list
> > eclipselink-users@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/eclipselink-users
> _______________________________________________
> eclipselink-users mailing list
> eclipselink-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
[EL Info]: 2008.11.21 14:15:01.093--ServerSession(33341602)--EclipseLink, version: Eclipse Persistence Services - 1.0.2 (Build 20081024)
[EL Info]: 2008.11.21 14:15:08.171--ServerSession(33341602)--file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod login successful
Shut down db now
go
This should fail
[EL Warning]: 2008.11.21 14:15:24.859--ClientSession(16278782)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()
Now clear the blockage
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:322)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1358)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:755)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:823)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:557)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:500)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeCall(AbstractSession.java:855)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:204)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:190)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelectCall(DatasourceCallQueryMechanism.java:233)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelect(DatasourceCallQueryMechanism.java:213)
	at org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(DataModifyQuery.java:66)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:666)
	at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2207)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1178)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1162)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1136)
	at org.eclipse.persistence.sequencing.QuerySequence.update(QuerySequence.java:330)
	at org.eclipse.persistence.sequencing.QuerySequence.updateAndSelectSequence(QuerySequence.java:269)
	at org.eclipse.persistence.sequencing.StandardSequence.getGeneratedVector(StandardSequence.java:72)
	at org.eclipse.persistence.sequencing.Sequence.getGeneratedVector(Sequence.java:267)
	at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:455)
	at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:884)
	at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:86)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:258)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:405)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNotRegisteredNewObjectForPersist(UnitOfWorkImpl.java:3879)
	at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.registerNotRegisteredNewObjectForPersist(RepeatableWriteUnitOfWork.java:369)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNewObjectForPersist(UnitOfWorkImpl.java:3827)
	at org.eclipse.persistence.internal.jpa.EntityManagerImpl.persist(EntityManagerImpl.java:219)
	at costar.dbaccess.jpa.eltest.TomsTestCase.testEMCloseAndOpen(TomsTestCase.java:70)
	at costar.dbaccess.jpa.eltest.TomsTestCase.main(TomsTestCase.java:17)
Caused by: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:988)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
	at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:748)
	... 29 more
go
This should work
[EL Info]: 2008.11.21 14:16:01.562--UnitOfWork(12608088)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
Read id: 5
Exception in thread "main" Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:322)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1358)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:755)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:823)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:557)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:500)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeCall(AbstractSession.java:855)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:204)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:190)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelectCall(DatasourceCallQueryMechanism.java:233)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelect(DatasourceCallQueryMechanism.java:213)
	at org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(DataModifyQuery.java:66)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:666)
	at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2207)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1178)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1162)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1136)
	at org.eclipse.persistence.sequencing.QuerySequence.update(QuerySequence.java:330)
	at org.eclipse.persistence.sequencing.QuerySequence.updateAndSelectSequence(QuerySequence.java:269)
	at org.eclipse.persistence.sequencing.StandardSequence.getGeneratedVector(StandardSequence.java:72)
	at org.eclipse.persistence.sequencing.Sequence.getGeneratedVector(Sequence.java:267)
	at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:455)
	at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:884)
	at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:86)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:258)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:405)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNotRegisteredNewObjectForPersist(UnitOfWorkImpl.java:3879)
	at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.registerNotRegisteredNewObjectForPersist(RepeatableWriteUnitOfWork.java:369)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNewObjectForPersist(UnitOfWorkImpl.java:3827)
	at org.eclipse.persistence.internal.jpa.EntityManagerImpl.persist(EntityManagerImpl.java:219)
	at costar.dbaccess.jpa.eltest.TomsTestCase.testEMCloseAndOpen(TomsTestCase.java:92)
	at costar.dbaccess.jpa.eltest.TomsTestCase.main(TomsTestCase.java:17)
Caused by: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:988)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
	at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:748)
	... 29 more
[EL Warning]: 2008.11.21 14:16:12.046--ClientSession(9089167)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()

This trace replicated a successful recovery on local MySQL with an orderly database shutdown.

[EL Info]: 2008.11.21 14:06:41.250--ServerSession(13878877)--EclipseLink, version: Eclipse Persistence Services - 1.0.2 (Build 20081024)
[EL Info]: 2008.11.21 14:06:41.546--ServerSession(13878877)--file:/D:/projs/costar-1.3/jclass/-Cloaks-Mysql login successful
Shut down db now
a
This should fail
[EL Info]: 2008.11.21 14:07:05.812--ClientSession(13506221)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
[EL Warning]: 2008.11.21 14:07:06.843--ClientSession(13506221)--com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 0 ms ago.
[EL Info]: 2008.11.21 14:07:12.937--ClientSession(13506221)--Communication failure detected when attempting to create transaction on database.  Attempting to retry begin transaction.
[EL Warning]: 2008.11.21 14:07:14.062--ClientSession(13506221)--com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 0 ms ago.
[EL Info]: 2008.11.21 14:07:20.156--ClientSession(13506221)--Communication failure detected when attempting to create transaction on database.  Attempting to retry begin transaction.
[EL Warning]: 2008.11.21 14:07:21.062--ClientSession(13506221)--com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 0 ms ago.
[EL Info]: 2008.11.21 14:07:27.046--ClientSession(13506221)--Communication failure detected when attempting to create transaction on database.  Attempting to retry begin transaction.
[EL Info]: 2008.11.21 14:07:32.046--ClientSession(13506221)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
Error Code: 0
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:322)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1358)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicBeginTransaction(DatabaseAccessor.java:195)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.beginTransaction(DatasourceAccessor.java:234)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTransaction(AbstractSession.java:381)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTransaction(AbstractSession.java:423)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTransaction(AbstractSession.java:370)
	at org.eclipse.persistence.sessions.server.ClientSession.basicBeginTransaction(ClientSession.java:123)
	at org.eclipse.persistence.internal.sessions.AbstractSession.beginTransaction(AbstractSession.java:535)
	at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:447)
	at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:884)
	at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:86)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:258)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:405)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNotRegisteredNewObjectForPersist(UnitOfWorkImpl.java:3879)
	at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.registerNotRegisteredNewObjectForPersist(RepeatableWriteUnitOfWork.java:369)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNewObjectForPersist(UnitOfWorkImpl.java:3827)
	at org.eclipse.persistence.internal.jpa.EntityManagerImpl.persist(EntityManagerImpl.java:219)
	at costar.dbaccess.jpa.eltest.TomsTestCase.testEMCloseAndOpen(TomsTestCase.java:70)
	at costar.dbaccess.jpa.eltest.TomsTestCase.main(TomsTestCase.java:17)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
	at com.mysql.jdbc.Util.getInstance(Util.java:381)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
	at com.mysql.jdbc.ConnectionImpl.getMutex(ConnectionImpl.java:3032)
	at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4869)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicBeginTransaction(DatabaseAccessor.java:190)
	... 17 more
[EL Warning]: 2008.11.21 14:07:32.046--ClientSession(13506221)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
Error Code: 0
Now clear the blockage
go
This should work
[EL Info]: 2008.11.21 14:07:58.890--UnitOfWork(26577913)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
Read id: 5
[EL Info]: 2008.11.21 14:07:58.953--ClientSession(12514145)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
Persisted id: 22
[EL Info]: 2008.11.21 14:07:59.000--ClientSession(12514145)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query.
testEMCloseAndOpen() completed

package costar.dbaccess.jpa.eltest;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;

import costar.dbaccess.CodeEntry;
import costar.domain.Archive;
public class TomsTestCase {
	public static void main(String[] args) throws Exception {
		// try the EMF close and open varient first
		TomsTestCase test = new TomsTestCase();
	    test.testEMFCloseAndOpen();
		//test.testEMCloseAndOpen();
	}

	// This is the case where the whole EMF is recreated
	    public void testEMFCloseAndOpen() throws IOException /* IOException from console reads*/{
	        Archive emp = null;

	        // creates an EMF for PU default and then creates an EM from it
	        createEntityManagerFactory(); // gjf
	        EntityManager em = createEntityManager();
	        em.find(Archive.class, 5);
	        try{
	        	System.out.println("Shut down db now");
	    		System.in.read(inBuff);
	        	System.out.println("This should fail");
	            em.getTransaction().begin();
	            // shut down DB prior to this line
	            emp = new Archive();
	            em.persist(emp);
	            em.getTransaction().commit();

	        } catch (Exception e){
	            // get exception here after a number of retries
	        	try {
	        		getEntityManager().close(); // gjf moved this into the exception handler, then found it redundant
	        		System.out.println("Closed EM ");
	        		getEntityManagerFactory().close(); // gjf moved this into the exception handler, then found it redundant
	        		System.out.println("Closed EMF ");
	        	} catch (Exception e2) {
	        		// ignore this
	        		System.out.println("Ignoring "+e2.toString());
	        	}
	            e.printStackTrace();
	        }
	        // getEntityManagerFactory().close(); // gjf moved this into the exception handler

        	System.out.println("Now clear the blockage");
    		System.in.read(inBuff);
        	System.out.println("This should work");
	        // restart the db prior to this line
        	createEntityManagerFactory(); // gjf adds this
	        em = createEntityManager();
	        em.getTransaction().begin();
	        emp = new Archive();
	        em.persist(emp);
	        em.getTransaction().commit();
        	System.out.println("This should work");
	    }

	// this is the case where just the EM is recreated
	    public void testEMCloseAndOpen() throws IOException /* IOException from console reads*/ {
	    	Archive emp = null;
	    	createEntityManagerFactory(); // gjf added
	        EntityManager em = createEntityManager();
	        em.find(CodeEntry.class, 1);
	        try{
	        	System.out.println("Shut down db now");
	    		System.in.read(inBuff);
	        	System.out.println("This should fail");
	            em.getTransaction().begin();
	            // shut down DB prior to this line
	            emp = new Archive();
	            em.persist(emp);
	            em.getTransaction().commit();
	        } catch (Exception e){
	            // get exception here after a number of retries
	            e.printStackTrace();
	        }
	        // gjf adds try block, this close may be redundant
	        try {
	        	em.close();
	        } catch (Exception e) {
	        	System.out.println("Ignoring: "+e.toString());
	        }
        	System.out.println("Now clear the blockage");
    		System.in.read(inBuff);
        	System.out.println("This should work");
	        // restart the db prior to this line
	        em = createEntityManager();
	        // em.find(Archive.class, 5); //  gjf moved this into the trx (else threw)
	        em.getTransaction().begin();
	        Archive a = em.find(Archive.class, 5); //  gjf moved this into the trx
	        System.out.println("Read id: "+a.getId());
	        emp = new Archive();
	        em.persist(emp);
	        System.out.println("Persisted id: "+emp.getId());
	        em.getTransaction().commit();
        	System.out.println("testEMCloseAndOpen() completed");
	        
	    } 
	    
	    protected EntityManager createEntityManager() {
	    	
	    	entityManager = getEntityManagerFactory().createEntityManager(); // NB this is a possible failure point (eg DB not reachable will throw here)
	    	return entityManager;
	    }
	    

		protected  EntityManagerFactory createEntityManagerFactory() {
			String persistenceUnitLabel = "Cloaks-dev-prod"; // Remote ORACLE
			// String persistenceUnitLabel = "Cloaks-Mysql"; // Local MySQL
			String archiveLbl = "Costar";

			Map emfProperties = new HashMap();
			/*
			emfProperties.put(JDBC_READ_CONNECTIONS_MIN, "1");
			emfProperties.put(JDBC_WRITE_CONNECTIONS_MIN, "1");
			// Ensure that no server-platform is configured
			emfProperties.put(TARGET_SERVER, TargetServer.None);
			log.info("Using new explicitly pooled connection properties");
			*/
			entityManagerFactory = Persistence.createEntityManagerFactory(persistenceUnitLabel,emfProperties); 	
			
			return entityManagerFactory;
		}
//
// ATTRIBUTES AND ACCESSORS
//

		byte[] inBuff = new byte[10];
		
		EntityManagerFactory entityManagerFactory = null;
		/**
		 * @return the entityManagerFactory
		 */
		public EntityManagerFactory getEntityManagerFactory() {
			return entityManagerFactory;
		}

		/**
		 * @param entityManagerFactory the entityManagerFactory to set
		 */
		public void setEntityManagerFactory(EntityManagerFactory entityManagerFactory) {
			this.entityManagerFactory = entityManagerFactory;
		}

		/**
		 * @return the entityManager
		 */
		public EntityManager getEntityManager() {
			return entityManager;
		}

		/**
		 * @param entityManager the entityManager to set
		 */
		public void setEntityManager(EntityManager entityManager) {
			this.entityManager = entityManager;
		}

		EntityManager entityManager;
}

[EL Info]: 2008.11.21 15:05:51.812--ServerSession(33341602)--EclipseLink, version: Eclipse Persistence Services - 1.0.2 (Build 20081024)
[EL Info]: 2008.11.21 15:05:59.156--ServerSession(33341602)--file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod login successful
Shut down db now
down
This should fail
[EL Warning]: 2008.11.21 15:06:33.343--ClientSession(25515818)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()
Closed EM 
Ignoring Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Now clear the blockage
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
Error Code: 17002
Call: UPDATE CLOAKS_SEQS SET LAST_VALUE = LAST_VALUE + ? WHERE ENTITY = ?
	bind => [1, archives_seq]
Query: DataModifyQuery()
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:322)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1358)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:755)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:823)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:557)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:500)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeCall(AbstractSession.java:855)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:204)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:190)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelectCall(DatasourceCallQueryMechanism.java:233)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelect(DatasourceCallQueryMechanism.java:213)
	at org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(DataModifyQuery.java:66)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:666)
	at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2207)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1178)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1162)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1136)
	at org.eclipse.persistence.sequencing.QuerySequence.update(QuerySequence.java:330)
	at org.eclipse.persistence.sequencing.QuerySequence.updateAndSelectSequence(QuerySequence.java:269)
	at org.eclipse.persistence.sequencing.StandardSequence.getGeneratedVector(StandardSequence.java:72)
	at org.eclipse.persistence.sequencing.Sequence.getGeneratedVector(Sequence.java:267)
	at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:455)
	at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:884)
	at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:86)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:258)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:405)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNotRegisteredNewObjectForPersist(UnitOfWorkImpl.java:3879)
	at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.registerNotRegisteredNewObjectForPersist(RepeatableWriteUnitOfWork.java:369)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerNewObjectForPersist(UnitOfWorkImpl.java:3827)
	at org.eclipse.persistence.internal.jpa.EntityManagerImpl.persist(EntityManagerImpl.java:219)
	at costar.dbaccess.jpa.eltest.TomsTestCase.testEMFCloseAndOpen(TomsTestCase.java:35)
	at costar.dbaccess.jpa.eltest.TomsTestCase.main(TomsTestCase.java:16)
Caused by: java.sql.SQLException: Io exception: Connection reset by peer: socket write error
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:988)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
	at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:748)
	... 29 more
go
This should work
Exception in thread "main" javax.persistence.PersistenceException: Exception [EclipseLink-28019] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit [Cloaks-dev-prod] failed.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod without closing it.
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:227)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.getServerSession(EntityManagerFactoryImpl.java:69)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:118)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:112)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:100)
	at costar.dbaccess.jpa.eltest.TomsTestCase.createEntityManager(TomsTestCase.java:110)
	at costar.dbaccess.jpa.eltest.TomsTestCase.testEMFCloseAndOpen(TomsTestCase.java:58)
	at costar.dbaccess.jpa.eltest.TomsTestCase.main(TomsTestCase.java:16)
Caused by: Exception [EclipseLink-28019] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Deployment of PersistenceUnit [Cloaks-dev-prod] failed.
Internal Exception: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod without closing it.
	at org.eclipse.persistence.exceptions.EntityManagerSetupException.deployFailed(EntityManagerSetupException.java:218)
	... 8 more
Caused by: javax.persistence.PersistenceException: Exception [EclipseLink-28009] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod without closing it.
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.addSessionToGlobalSessionManager(EntityManagerSetupImpl.java:307)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.initServerSession(EntityManagerSetupImpl.java:1137)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:216)
	... 7 more
Caused by: Exception [EclipseLink-28009] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Attempted to redeploy a session named file:/D:/projs/costar-1.3/jclass/-Cloaks-dev-prod without closing it.
	at org.eclipse.persistence.exceptions.EntityManagerSetupException.attemptedRedeployWithoutClose(EntityManagerSetupException.java:74)
	... 10 more