Eclipse Community Forums: EclipseLink » Getting hung threads in Eclipselink 2.6.4

Home » Eclipse Projects » EclipseLink » Getting hung threads in Eclipselink 2.6.4(Getting hung threads in Eclipselink 2.6.4 while getting relation objects)

Show: Today's Messages :: Show Polls :: Message Navigator

Getting hung threads in Eclipselink 2.6.4 [message #1771626]

Mon, 28 August 2017 18:50

P Shetty

Messages: 7
Registered: August 2017

Junior Member

We migrated our code from using Toplink 9.0. to Eclipselink 2.6.4 thats running on Websphere App Server 8.5.5 and Java 7. We are using a Eclipselink project configuration xml which has all the mappings and we have the
corresponding Java objects for the mappings as well. We are not using EclipseLink JPA because migrating to JPA was a lot of rework with a ton of objects that we had. We changed the toplink-config.xml to eclipselink-config.xml maintaining the same mapping and making necessary modifications for the new eclipselink XSD.

After migrating the code, the application was tested in lower environments and all seemed okay, but in Production we keep getting transaction timeouts followed by several hung threads. One major issue here is we cannot reproduce it at will on another environment.
This eventually takes up all CPU and restarting the JVM is what we end up doing to restore normal operation. We did not have this issue with Toplink 9.0

I notice this happening while accessing a one-to-many relationship object.

AccessoryListType has a one-many relationship with AccessoryListTypeAccessoryLinks. If you look at AccessoryListType.java below, the second line in the getAccessories() method is where we get the AccessoryListTypeAccessoryLinks objects and results in the timeout/exception.

I am assuming we are doing a Lazy Fetch since ValueHolderInterface indirection is used. I am also attaching the eclipse-config.xml snippet which has the mapping for the objects.

ERROR :

[7/26/17 11:02:15:532 CDT] 00000072 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently,

associated was Thread[ORB.thread.p
ool : 3,5,main]. The stack trace of this thread when the timeout occurred was:
java.util.HashMap.getEntry(HashMap.java:477)
java.util.HashMap.get(HashMap.java:429)
org.eclipse.persistence.internal.identitymaps.FullIdentityMap.getCacheKey(FullIdentityMap.java:100)
org.eclipse.persistence.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:817)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:428)
org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:131)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3940)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3894)
org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:308)
org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:321)
org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:215)
org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:223)
org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60)
org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
com.harland.checks.server.businessobjects.product.AccessoryListType.getAccessoryListTypeAccessoryLinks(AccessoryListType.java:180)
com.harland.checks.server.businessobjects.product.AccessoryListType.getAccessories(AccessoryListType.java:260)
com.harland.checks.server.businessobjects.product.AccessoryList.getAccessoriesByType(AccessoryList.java:429)

AccessoryListType.java
---------------------------

public class AccessoryListType
{
private int uid;
private String accessoryListType = "";
private ValueHolderInterface accessoryListHolder = null;
private ValueHolderInterface accessoryListTypeAccessoryLinkHolder = null;

public AccessoryListType()
{
accessoryListTypeAccessoryLinkHolder = new ValueHolder(new Vector());
accessoryListHolder = new ValueHolder();
}

public void setUid(int id)
{
uid = id;
}

public int getUid()
{
return uid;
}

public String toString()
{
return getAccessoryListType();
}

public void setAccessoryListType(String type)
{
accessoryListType = type;
}

public String getAccessoryListType()
{
return accessoryListType;
}

public void setAccessoryListHolder(ValueHolderInterface holder)
{
accessoryListHolder = holder;
}

public ValueHolderInterface getAccessoryListHolder()
{
return accessoryListHolder;
}

public void setAccessoryList(AccessoryList al)
{
getAccessoryListHolder().setValue(al);
}

public AccessoryList getAccessoryList()
{
return (AccessoryList)getAccessoryListHolder().getValue();
}

public void setAccessoryListTypeAccessoryLinkHolder(
ValueHolderInterface holder)
{
accessoryListTypeAccessoryLinkHolder = holder;
}

public ValueHolderInterface getAccessoryListTypeAccessoryLinkHolder()
{
return accessoryListTypeAccessoryLinkHolder;
}

public void setAccessoryListTypeAccessoryLinks(Vector links)
{
getAccessoryListTypeAccessoryLinkHolder().setValue(links);
}

public Vector getAccessoryListTypeAccessoryLinks()
{
return (Vector)getAccessoryListTypeAccessoryLinkHolder().getValue();
}

public void addAccessoryListTypeLink(AccessoryListTypeAccLink altl)
{
getAccessoryListTypeAccessoryLinks().addElement(altl);
}

public void removeAccessoryListTypeLink(AccessoryListTypeAccLink altl)
{
getAccessoryListTypeAccessoryLinks().removeElement(altl);
}

public void addAccessory(Accessory accessory)
{
AccessoryListTypeAccLink link = new AccessoryListTypeAccLink();
link.setAccessory(accessory);
link.setAccessoryListType(this);
this.addAccessoryListTypeLink(link);
}

public void addAccessory(Accessory accessory, AccessoryListTypeAccLink link)
{
link.setAccessory(accessory);
link.setAccessoryListType(this);
this.addAccessoryListTypeLink(link);
}

public void removeAccessory(Accessory accessory)
{
Vector links = getAccessoryListTypeAccessoryLinks();

for (int loop = 0; loop < links.size(); loop++)
{
AccessoryListTypeAccLink link =
(AccessoryListTypeAccLink)links.elementAt(loop);

if (link.getAccessory().equals(accessory))
{
removeAccessoryListTypeLink(link);

break;
}
}
}

public Vector getAccessories()
{
Vector accessories = new Vector();

Vector links = this.getAccessoryListTypeAccessoryLinks(); ----/////// this is where we get the error
int size = links.size();

for (int i = 0; i < size; i++)
{
accessories.addElement(
((AccessoryListTypeAccLink)links.elementAt(i)).getAccessory());
}

return accessories;
}
}

AccessoryListTypeAccLink.java
----------------------------------

public class AccessoryListTypeAccLink
{
private int uid;
private ValueHolderInterface accessoryHolder = null;
private ValueHolderInterface accessoryListTypeHolder = null;

public AccessoryListTypeAccLink()
{
accessoryHolder = new ValueHolder();
accessoryListTypeHolder = new ValueHolder();
}

public void setUid(int id)
{
uid = id;
}

public int getUid()
{
return uid;
}

public void setAccessoryHolder(ValueHolderInterface holder)
{
accessoryHolder = holder;
}

public ValueHolderInterface getAccessoryHolder()
{
return accessoryHolder;
}

public void setAccessory(Accessory accessory)
{
getAccessoryHolder().setValue(accessory);
}

public Accessory getAccessory()
{
return (Accessory)getAccessoryHolder().getValue();
}

public void setAccessoryListTypeHolder(ValueHolderInterface holder)
{
accessoryListTypeHolder = holder;
}

public ValueHolderInterface getAccessoryListTypeHolder()
{
return accessoryListTypeHolder;
}

public void setAccessoryListType(AccessoryListType accessoryListType)
{
getAccessoryListTypeHolder().setValue(accessoryListType);
}

public AccessoryListType getAccessoryListType()
{
return (AccessoryListType)getAccessoryListTypeHolder().getValue();
}
}

eclipselink-config.xml (snippet)
----------------------------
<class-mapping-descriptor xsi:type="relational-class-mapping-descriptor">
<class>com.harland.checks.server.businessobjects.product.AccessoryListType</class>
<alias>AccessoryListType</alias>
<primary-key>
<field table="accessorylisttype" name="uid" xsi:type="column"/>
</primary-key>
<events xsi:type="event-policy"/>
<querying xsi:type="query-policy"/>
<attribute-mappings>
<attribute-mapping xsi:type="one-to-one-mapping">
<attribute-name>accessoryListHolder</attribute-name>
<get-method>getAccessoryListHolder</get-method>
<set-method>setAccessoryListHolder</set-method>
<reference-class>com.harland.checks.server.businessobjects.product.AccessoryList</reference-class>
<foreign-key>
<field-reference>
<source-field table="accessorylisttype" name="acclist_uid" xsi:type="column"/>
<target-field table="p_accessory_list" name="accessory_list_id" xsi:type="column"/>
</field-reference>
</foreign-key>
<foreign-key-fields>
<field table="accessorylisttype" name="acclist_uid" xsi:type="column"/>
</foreign-key-fields>
<indirection xsi:type="value-holder-indirection-policy"/>
<selection-query xsi:type="read-object-query"/>
</attribute-mapping>
<attribute-mapping xsi:type="direct-mapping">
<attribute-name>accessoryListType</attribute-name>
<get-method>getAccessoryListType</get-method>
<set-method>setAccessoryListType</set-method>
<field table="accessorylisttype" name="accessory_type" xsi:type="column"/>
</attribute-mapping>
<attribute-mapping xsi:type="one-to-many-mapping">
<attribute-name>accessoryListTypeAccessoryLinkHolder</attribute-name>
<get-method>getAccessoryListTypeAccessoryLinkHolder</get-method>
<set-method>setAccessoryListTypeAccessoryLinkHolder</set-method>
<reference-class>com.harland.checks.server.businessobjects.product.AccessoryListTypeAccLink</reference-class>
<private-owned>true</private-owned>
<target-foreign-key>
<field-reference>
<source-field table="acclisttypeacc_lk" name="acc_list_type_uid" xsi:type="column"/>
<target-field table="accessorylisttype" name="uid" xsi:type="column"/>
</field-reference>
</target-foreign-key>
<container xsi:type="list-container-policy">
<collection-type>java.util.Vector</collection-type>
</container>
<indirection xsi:type="value-holder-indirection-policy"/>
<selection-query xsi:type="read-all-query">
<container xsi:type="list-container-policy">
<collection-type>java.util.Vector</collection-type>
</container>
</selection-query>
</attribute-mapping>
<attribute-mapping xsi:type="direct-mapping">
<attribute-name>uid</attribute-name>
<get-method>getUid</get-method>
<set-method>setUid</set-method>
<field table="accessorylisttype" name="uid" xsi:type="column"/>
</attribute-mapping>
</attribute-mappings>
<descriptor-type>independent</descriptor-type>
<sequencing>
<sequence-name>accessorylisttype</sequence-name>
<sequence-field table="accessorylisttype" name="uid" xsi:type="column"/>
</sequencing>
<instantiation/>
<copying xsi:type="instantiation-copy-policy"/>
<change-policy xsi:type="deferred-detection-change-policy"/>
<tables>
<table name="accessorylisttype"/>
</tables>
</class-mapping-descriptor>
<class-mapping-descriptor xsi:type="relational-class-mapping-descriptor">
<class>com.harland.checks.server.businessobjects.product.AccessoryListTypeAccLink</class>
<alias>AccessoryListTypeAccLink</alias>
<primary-key>
<field table="acclisttypeacc_lk" name="uid" xsi:type="column"/>
</primary-key>
<events xsi:type="event-policy"/>
<querying xsi:type="query-policy"/>
<attribute-mappings>
<attribute-mapping xsi:type="one-to-one-mapping">
<attribute-name>accessoryHolder</attribute-name>
<get-method>getAccessoryHolder</get-method>
<set-method>setAccessoryHolder</set-method>
<reference-class>com.harland.checks.server.businessobjects.product.Accessory</reference-class>
<foreign-key>
<field-reference>
<source-field table="acclisttypeacc_lk" name="acc_uid" xsi:type="column"/>
<target-field table="p_accessory" name="accessory_id" xsi:type="column"/>
</field-reference>
</foreign-key>
<foreign-key-fields>
<field table="acclisttypeacc_lk" name="acc_uid" xsi:type="column"/>
</foreign-key-fields>
<indirection xsi:type="value-holder-indirection-policy"/>
<selection-query xsi:type="read-object-query"/>
</attribute-mapping>
<attribute-mapping xsi:type="one-to-one-mapping">
<attribute-name>accessoryListTypeHolder</attribute-name>
<get-method>getAccessoryListTypeHolder</get-method>
<set-method>setAccessoryListTypeHolder</set-method>
<reference-class>com.harland.checks.server.businessobjects.product.AccessoryListType</reference-class>
<foreign-key>
<field-reference>
<source-field table="acclisttypeacc_lk" name="acc_list_type_uid" xsi:type="column"/>
<target-field table="accessorylisttype" name="uid" xsi:type="column"/>
</field-reference>
</foreign-key>
<foreign-key-fields>
<field table="acclisttypeacc_lk" name="acc_list_type_uid" xsi:type="column"/>
</foreign-key-fields>
<indirection xsi:type="value-holder-indirection-policy"/>
<selection-query xsi:type="read-object-query"/>
</attribute-mapping>
<attribute-mapping xsi:type="direct-mapping">
<attribute-name>uid</attribute-name>
<get-method>getUid</get-method>
<set-method>setUid</set-method>
<field table="acclisttypeacc_lk" name="uid" xsi:type="column"/>
</attribute-mapping>
</attribute-mappings>
<descriptor-type>independent</descriptor-type>
<sequencing>
<sequence-name>acclisttypeacc_lk</sequence-name>
<sequence-field table="acclisttypeacc_lk" name="uid" xsi:type="column"/>
</sequencing>
<instantiation/>
<copying xsi:type="instantiation-copy-policy"/>
<change-policy xsi:type="deferred-detection-change-policy"/>
<tables>
<table name="acclisttypeacc_lk"/>
</tables>
</class-mapping-descriptor>

Report message to a moderator

Re: Getting hung threads in Eclipselink 2.6.4 [message #1771690 is a reply to message #1771626]

Tue, 29 August 2017 14:18

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

Indirection forces a query when it gets triggered, so this is the same issue as described here https://www.eclipse.org/forums/index.php/m/1771687/#msg_1771687 but from a different point.

Since the lock is occurring entirely when trying to clone objects within the associated UnitOfWork object, the only cause I can think of is two or more threads using the same UnitOfWork instance concurrently. Check that your application isn't reading objects through a UnitOfWork and then caching them or accessing them in other threads. A thread dump might show other threads getting stuck, and allow you to trace how/where they got the objects they are stuck on.

I'd also recommend contacting Oracle support if you have a support contract, as they might be able to help take a look at your usage both at the current app and the prior TopLink application to make suggestions.

Report message to a moderator

Re: Getting hung threads in Eclipselink 2.6.4 [message #1772132 is a reply to message #1771690]

Tue, 05 September 2017 21:40

P Shetty

Messages: 7
Registered: August 2017

Junior Member

Thank you for your response!

My understanding is is that we are not using the same UnitOfWork instance, but I am not a 100% sure as I have not been able to confirm that yet.

We do not have any Oracle support contract anymore.

From the coredumps, this is what I got for the blocking thread (holding the monitor) and for one of the blocked thread. See below -
Can you please advice or suggest some options? The objects, the classdescriptors and the location in the code where we get the error is pointed out above in the previous message.
Also, is there something else that I should look for in the coredump?

Blocking thread :

WebContainer 16 holds monitor

Owns Monitor Lock on org/eclipse/persistence/internal/indirection/UnitOfWorkQueryValueHolder@0x0000000713B47FE0

Stacktrace :

at java/util/HashMap.getEntry(HashMap.java:477(Compiled Code)) 
at java/util/HashMap.get(HashMap.java:429(Compiled Code)) 
at org/eclipse/persistence/internal/identitymaps/FullIdentityMap.getCacheKey(FullIdentityMap.java:100(Compiled Code)) 
at org/eclipse/persistence/internal/identitymaps/IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:817(Compiled Code)) 
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:428(Compiled Code)) 
at org/eclipse/persistence/internal/sessions/UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:131(Compiled Code)) 
at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3940(Compiled Code)) 
at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3894(Compiled Code)) 
at org/eclipse/persistence/mappings/ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:111(Compiled Code)) 
at org/eclipse/persistence/mappings/ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:73(Compiled Code)) 
at org/eclipse/persistence/internal/indirection/UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60(Compiled Code)) 
at org/eclipse/persistence/internal/indirection/UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173(Compiled Code)) 
at org/eclipse/persistence/internal/indirection/UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234(Compiled Code)) 
at org/eclipse/persistence/internal/indirection/DatabaseValueHolder.getValue(DatabaseValueHolder.java:89(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryListTypeAccLink.getAccessory(AccessoryListTypeAccLink.java:100(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryListType.getAccessories(AccessoryListType.java:265(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryList.getAccessoriesByType(AccessoryList.java:429(Compiled Code)) 
at com/harland/checks/server/bizservices/applicationcontrollers/AccessoryController.buildAccessoryListTableFromPGList(AccessoryController.java:388(Compiled Code)) 
at com/harland/checks/server/bizservices/applicationcontrollers/AccessoryController.getAccessoryInfoTable(AccessoryController.java:640(Compiled Code)) 
at com/harland/checks/server/EnhancementBuilder.validateEnhancements(EnhancementBuilder.java:365) 
at com/harland/checks/server/CheckService.validateEnhancement(CheckService.java:2190) 
at com/harland/checks/intf/EJSRemote0SLHCCheckService_31046456.validateEnhancement(EJSRemote0SLHCCheckService_31046456.java)

Blocked thread

Blocked thread - Webcontainer 0

Monitor
Waiting for Monitor Lock on org/eclipse/persistence/internal/indirection/UnitOfWorkQueryValueHolder@0x0000000713B47FE0

Stacktrace :

at org/eclipse/persistence/internal/indirection/DatabaseValueHolder.getValue(DatabaseValueHolder.java:86(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryListTypeAccLink.getAccessory(AccessoryListTypeAccLink.java:100(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryListType.getAccessories(AccessoryListType.java:265(Compiled Code)) 
at com/harland/checks/server/businessobjects/product/AccessoryList.getAccessoriesByType(AccessoryList.java:429(Compiled Code)) 
at com/harland/checks/server/bizservices/applicationcontrollers/AccessoryController.buildAccessoryListTableFromPGList(AccessoryController.java:388(Compiled Code)) 
at com/harland/checks/server/bizservices/applicationcontrollers/AccessoryController.getAccessoryInfoTable(AccessoryController.java:640(Compiled Code)) 
at com/harland/checks/server/EnhancementInfoBuilder.getEnhancementsForCodesNew(EnhancementInfoBuilder.java:476) 
at com/harland/checks/server/CheckInfoBuilder.getCheckInfo(CheckInfoBuilder.java:1558) 
at com/harland/checks/server/CheckService.getEnhancementInfoForOrderDetailLineItem(CheckService.java:316) 
at com/harland/checks/intf/EJSRemote0SLHCCheckService_31046456.getEnhancementInfoForOrderDetailLineItem(EJSRemote0SLHCCheckService_31046456.java) 
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) 
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95(Compiled Code)) 
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code)) 
at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code)) 
at com/ibm/CORBA/iiop/ClientDelegate$3.run(ClientDelegate.java:1165(Compiled Code)) 
at java/security/AccessController.doPrivileged(AccessController.java:420(Compiled Code)) 
at com/ibm/CORBA/iiop/ClientDelegate.invoke0(ClientDelegate.java:1162(Compiled Code)) 
at com/ibm/CORBA/iiop/ClientDelegate$ClientDelegate0.invoke(ClientDelegate.java:1399(Compiled Code)) 
at com/sun/proxy/$Proxy59.getEnhancementInfoForOrderDetailLineItem(Bytecode PC:26) 
at com/harland/checks/intf/_HCCheckServiceRemote_Stub.getEnhancementInfoForOrderDetailLineItem(_HCCheckServiceRemote_Stub.java) 
at com/harland/branch/eacsservices/adapters/ChecksAdapter.getCheckDetailsForOrderDetail(ChecksAdapter.java:1273)

Report message to a moderator

Re: Getting hung threads in Eclipselink 2.6.4 [message #1772198 is a reply to message #1772132]

Wed, 06 September 2017 14:24

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

WebContainer 16 is proceeding to build an object, so it isn't necessarily locked or deadlocked, but this shows you have multiple threads sharing the same object read from a single UnitOfWork. The UnitOfWork and objects read from it (managed objects) are not thread safe - other threads should read in their own copies of the instances when required.

This would always have been an issue, so I don't know what might have changed - possibly you were using some query options that got lost in the transition to EclipseLink, such as reading a read-only instance instead. You will have to evaluate what the AccessoryListTypeAccLink.getAccessory method is doing and how the AccessoryListTypeAccLink was read in to see if there is a simple fix, such as reading from the clientSession or using the read-only hint in JPA, or if something else is needed depending on how you intend to use the object.

Best Regards,
Chris

Report message to a moderator

Re: Getting hung threads in Eclipselink 2.6.4 [message #1772202 is a reply to message #1772198]

Wed, 06 September 2017 14:33

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

I'd also suggest looking at the AccessoryController.buildAccessoryListTableFromPGList method, as assuming the AccessoryList is an object pulled from the database, the buildAccessoryListTableFromPGList involvement in both threads seems suspect.

Report message to a moderator

Previous Topic:	Strange behaviour on Oracle 12 with escape character
Next Topic:	Migrating from Oracle Toplink to Eclipselink

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Tue Sep 25 04:55:54 GMT 2018

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter