Re: [eclipselink-dev] EclipseLink Stuck Threads on acquiring a lock

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [eclipselink-dev] EclipseLink Stuck Threads on acquiring a lock

From: Lukas Jungmann <lukas.jungmann@xxxxxxxxxx>
Date: Mon, 29 Sep 2014 15:12:59 +0200
Delivered-to: eclipselink-dev@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/eclipselink-dev>
List-help: <mailto:eclipselink-dev-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/eclipselink-dev>, <mailto:eclipselink-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/eclipselink-dev>, <mailto:eclipselink-dev-request@eclipse.org?subject=unsubscribe>

Hi John,

On Sep 24, 2014, at 8:11 PM, John Bedell <jbedell@xxxxxxxx> wrote:

We are running into a problem in our production environments that we have been unable to reproduce locally. I have posted on several bug reports and even opened my own bug report. I have also posted on the forums. None of those attempts have even yielded a response, let alone a fix. So, I’m trying the mailing list. The latest instance of this problem came with the stacktrace at the bottom of this message.

Thanks for all your reports, investigation, patience and all your work here. We will definitelly look into your bugs and findings within 2.6.0 timeframe and will get back to you and follow up as soon as we will have some questions.

Also, we are investigating paying for support to resolve this issue, but I cannot find any information on how to do this. Any links to that effect would be appreciated as well.

You can get support for EclipseLink through Oracle TopLink, see: http://www.oracle.com/technetwork/middleware/toplink/overview/index.html

WBR,

—lukas

Similar bug reports:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=413775
https://bugs.eclipse.org/bugs/show_bug.cgi?id=399528
https://bugs.eclipse.org/bugs/show_bug.cgi?id=305611
https://www.eclipse.org/forums/index.php/t/524575/
https://bugs.eclipse.org/bugs/show_bug.cgi?id=442913
https://bugs.eclipse.org/bugs/show_bug.cgi?id=294355
https://www.eclipse.org/forums/index.php/t/804977/

After looking at bug 294355 (above), we came up with the theory that we may be seeing what user Szymon Bochniak described in his response to that bug report. A thread may be releasing a lock that it did not acquire. We are in the process of adding some logging to verify this theory. Unfortunately, we cannot reliably reproduce the error, so we just have to wait for it to happen again. However, if we are right, then it begs that question asked in the link below: why does the acquireReadLock method verify that it is the active thread before locking, but the releaseReadLock does not have any similar verification before it releases?
https://wiki.eclipse.org/EclipseLink/Bugs/259993#The_activeThread_on_release.28.29_is_different_than_the_one_on_acquire.28.29

The stacktrace:

ERROR 24 Sep 2014 09:45:49,155 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] com.savi.ng.api.BaseApiController - API Controller Exception Handler
ERROR 24 Sep 2014 09:45:49,155 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] com.savi.ng.api.BaseApiController - Exception caught in rest service: Could not write JSON:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"])
org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"])
                at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:207)
                at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:179)
                at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:148)
                at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:90)
                at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:189)
                at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:69)
                at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122)
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
                at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
                at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
                at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
                at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
                at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
                at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
                at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at com.savi.ng.api.filter.CorsFilter.doFilterInternal(CorsFilter.java:196)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at com.savi.ng.api.filter.NoCacheFilter.doFilterInternal(NoCacheFilter.java:18)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at com.savi.ng.servlet.filter.UserProfileInjectorFilter.doFilter(UserProfileInjectorFilter.java:85)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:300)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74)
                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3288)
                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
                at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
                at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163)
                at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089)
                at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074)
                at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1513)
                at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: com.fasterxml.jackson.databind.JsonMappingException:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"])
                at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)
                at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:197)
                at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:184)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:605)
                at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)
                at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:186)
                at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597)
                at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142)
                at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597)
                at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)
                at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:186)
                at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597)
                at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142)
                at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597)
                at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142)
                at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:118)
                at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:1819)
                at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:204)
                ... 46 more
Caused by: Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.ConcurrencyException
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice.
                at org.eclipse.persistence.exceptions.ConcurrencyException.signalAttemptedBeforeWait(ConcurrencyException.java:84)
                at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:489)
                at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:392)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1019)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:933)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:193)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:121)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3906)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3861)
                at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
                at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
                at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:76)
                at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildClone(ForeignReferenceMapping.java:251)
                at org.eclipse.persistence.internal.descriptors.ObjectBuilder.populateAttributesForClone(ObjectBuilder.java:3574)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.populateAndRegisterObject(UnitOfWorkImpl.java:3657)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1004)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:933)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:193)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:121)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3906)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3861)
                at org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:296)
                at org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:309)
                at org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:214)
                at org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:222)
                at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:56)
                at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:161)
                at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:222)
                at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:88)
                at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:244)
                at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:415)
                at org.eclipse.persistence.indirection.IndirectList.isEmpty(IndirectList.java:490)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.isEmpty(IndexedListSerializer.java:51)
                at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.isEmpty(IndexedListSerializer.java:21)
                at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:556)
                at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597)
                ... 68 more

John Bedell
Senior Staff Software Engineer

Email jbedell@xxxxxxxx
Work +1-859-263-6309

<image001.png>

www.savi.com
Attention: The information contained in this email message may be legally privileged, confidential and protected from disclosure. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If you are not the intended recipient, you are on notice that any dissemination, distribution or copying of this message, including any attachments, in any form, is strictly prohibited. If you think you have received this email in error, please email the sender and delete/destroy any copy of this message.

_______________________________________________
eclipselink-dev mailing list
eclipselink-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/eclipselink-dev

References:
- [eclipselink-dev] EclipseLink Stuck Threads on acquiring a lock
  - From: John Bedell

Prev by Date: [eclipselink-dev] IP CQs waiting on your team
Next by Date: Re: [eclipselink-dev] Dynamic persistence unit
Previous by thread: [eclipselink-dev] EclipseLink Stuck Threads on acquiring a lock
Next by thread: [eclipselink-dev] Committer vote for Dmitry Kornilov has concluded successfully
Index(es):
- Date
- Thread

Breadcrumbs