[jgit-dev] Issues with JDK 11.0.2 and JGIT

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

[jgit-dev] Issues with JDK 11.0.2 and JGIT

From: Mike Bell <mbell@xxxxxxxxxxxxx>
Date: Fri, 29 Mar 2019 08:46:18 -0700
Delivered-to: jgit-dev@xxxxxxxxxxx
List-archive: <https://www.eclipse.org/mailman/private/jgit-dev>
List-help: <mailto:jgit-dev-request@eclipse.org?subject=help>
List-subscribe: <https://www.eclipse.org/mailman/listinfo/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://www.eclipse.org/mailman/options/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=unsubscribe>

Evidence still being gathered, suggestions gratefully received.

Versions: 5.1.3.201810200350-r, 5.2.1.201812262042-r, 5.3.0.201903130848-r, 5.1.0.201809111528-r, 5.1.3.201810200350-r

Scenario: 6 different deploys in java 11 containers (mesos)

We have used jgit for years for a library that we use to poll github for chhanges to repos successfully.

In this program every minute we re pull and check for changes, and pass affected files list and shas up to the caller. This is done in a single threaded scheduled executor.

We are finding that after several days. the synchronization stops. Neither the scheduled thread nor jgit-workqueue look "suspicious"

In fact they look very benign.

"JGit-WorkQueue" - Thread t@67

java.lang.Thread.State: WAITING

at jdk.internal.misc.Unsafe.park(Native Method)

- parking to wait for <5e502c44> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:

- None

config-update-0" - Thread t@87

java.lang.Thread.State: WAITING

at jdk.internal.misc.Unsafe.park(Native Method)

- parking to wait for <79fc6f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:

- None

However, heavy debugging shows we go into this code and never "return"

LOG.trace("Configuration of credentials completed, setting remote {}", remoteIndex);

pull.setRemote("remote" + remoteIndex);

PullResult result = pull.call();

LOG.trace("Got result {}", result);

(The first log shows, but not the second)

Additional debugging - a custom ProgressMonitor shows beginTask is called, and then no further

A try/catch/throwable around the code gleans no obvious failure of the thread itself.

Additional: We've turned on full net debugging and seen odd finalizer errors

"throwable" : {

java.net.SocketException: Socket is closed

at java.base/java.net.Socket.shutdownInput(Socket.java:1521)

at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:216)

at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:651)

at java.base/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:606)

at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:566)

at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:479)

at java.base/sun.net.www.http.HttpClient.closeServer(HttpClient.java:1058)

at java.base/sun.net.www.http.ClientVector.put(KeepAliveCache.java:252)

at java.base/sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:125)

at java.base/sun.net.www.protocol.https.HttpsClient.putInKeepAliveCache(HttpsClient.java:658)

at java.base/sun.net.www.http.HttpClient.finished(HttpClient.java:406)

at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:99)

at java.base/sun.net.www.MeteredStream.justRead(MeteredStream.java:93)

at java.base/sun.net.www.MeteredStream.skip(MeteredStream.java:154)

at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:89)

at java.base/sun.net.www.MeteredStream.finalize(MeteredStream.java:209)

at java.base/java.lang.System$2.invokeFinalize(System.java:2117)

at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87)

at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)}

Additional: We've tried turning off TLS 1.3 (openjdk has known issues) with -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2

and verified only TLSv1.2 is being used with tcpdump.

Michael Bell

Principal Software Engineer / Architect

1 Montgomery Tower, Suite 700

San Francisco, CA 94104

(415) 317-9559

mbell@xxxxxxxxxxxxx

Follow-Ups:
- Re: [jgit-dev] Issues with JDK 11.0.2 and JGIT
  - From: Matthias Sohn

Prev by Date: Re: [jgit-dev] Cannot garbage collect some objects
Next by Date: Re: [jgit-dev] CI Bot -1's too often due to flakiness
Previous by thread: [jgit-dev] Cannot garbage collect some objects
Next by thread: Re: [jgit-dev] Issues with JDK 11.0.2 and JGIT
Index(es):
- Date
- Thread

Breadcrumbs