[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] Issues with JDK 11.0.2 and JGIT

On Fri, Mar 29, 2019 at 4:46 PM Mike Bell <mbell@xxxxxxxxxxxxx> wrote:
Evidence still being gathered, suggestions gratefully received.

Versions: 5.1.3.201810200350-r, 5.2.1.201812262042-r, 5.3.0.201903130848-r, 5.1.0.201809111528-r, 5.1.3.201810200350-r
Scenario: 6 different deploys in java 11 containers (mesos)

We have used jgit for years for a library that we use to poll github for chhanges to repos successfully.Â

In this program every minute we re pull and check for changes, and pass affected files list and shas up to the caller. This is done in a single threaded scheduled executor.

We are finding that after several days. the synchronization stops. Neither the scheduled thread nor jgit-workqueue look "suspicious"
In fact they look very benign.Â

maybe you need to gc the local repository you are pulling to from time to time
Â
"JGit-WorkQueue" - Thread t@67
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <5e502c44> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:
- None

config-update-0" - Thread t@87
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <79fc6f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:
- None

However, heavy debugging shows we go into this code and never "return"

LOG.trace("Configuration of credentials completed, setting remote {}", remoteIndex);
pull.setRemote("remote" + remoteIndex);
PullResult result = pull.call();
LOG.trace("Got result {}", result);

(The first log shows, but not the second)

Additional debugging - a custom ProgressMonitor shows beginTask is called, and then no further
A try/catch/throwable around the code gleans no obvious failure of the thread itself.

Additional: We've turned on full net debugging and seen odd finalizer errors

avax.net.ssl|WARNING|03|Finalizer|2019-03-29 12:26:57.882 UTC|SSLSocketImpl.java:494|SSLSocket duplex close failed (
"throwable" : {
 java.net.SocketException: Socket is closed
 at java.base/java.net.Socket.shutdownInput(Socket.java:1521)
 at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:216)
 at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:651)
 at java.base/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:606)
 at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:566)
 at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:479)
 at java.base/sun.net.www.http.HttpClient.closeServer(HttpClient.java:1058)
 at java.base/sun.net.www.http.ClientVector.put(KeepAliveCache.java:252)
 at java.base/sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:125)
 at java.base/sun.net.www.protocol.https.HttpsClient.putInKeepAliveCache(HttpsClient.java:658)
 at java.base/sun.net.www.http.HttpClient.finished(HttpClient.java:406)
 at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:99)
 at java.base/sun.net.www.MeteredStream.justRead(MeteredStream.java:93)
 at java.base/sun.net.www.MeteredStream.skip(MeteredStream.java:154)
 at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:89)
 at java.base/sun.net.www.MeteredStream.finalize(MeteredStream.java:209)
 at java.base/java.lang.System$2.invokeFinalize(System.java:2117)
 at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87)
 at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)}


Additional: We've tried turning off TLS 1.3 (openjdk has known issues) with -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2
and verified only TLSv1.2 is being used with tcpdump.


--

Michael Bell

Principal Software Engineer / Architect

1 Montgomery Tower, Suite 700

San Francisco, CA 94104

(415) 317-9559

mbell@xxxxxxxxxxxxx

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJoAAAAoCAIAAACJn3Q+AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAABw1JREFUeNrsm09MVFcUxhUxRHkYIjUwxDQ24rjTiK40EVd0obiAhUpSVkRT66IVa1cGkYSkVGhXNhBXNvXPQrsQF3YjY0JXMi3dSWtijGEghpbIIAEb6Y855vT2zXvPN8MMQ+k9IZPHnfPuO/d85zvn3DszaxcWFtZYWS1SZF1g4bSyQqU4h3PNxkdexUdmh395nZj4a3xCxzfs2VUSrdlQu3tj7e6iMsc6PX+ydum188108s9bd14O3Dch9E4FTqlTd2Bza8v6SJV1/UqEc+rm7cmr194kZzK6q/xYY0Vri2XqCoITUo53ds88/CnLLF9VWd19qSS63WJQeDjB8vnptrnfniypDXNKKy+cJ/1aGAoJZ06wVNl6pYcuySJRMDiffXQqV1gKR7de6bVZtzD7ThqfHGK5yPXkzERnt0WiAHC+Toz/cfVazo0gPuiQLRjLDWc+sFTSU5ItHkuU4oyo+fLej3myg5SbfDi06fCH79QcHh6+fv06r8nkIvzRaPTQoUMnTpwoKytbHpe1tbXFYrEABczDqoB7HccZHBz0VBgdHW1ububi7NmzcpEvOGdiQ3l109TNO8FwTk9P9/T0DAwMuNaP4MH+/n4/J+ZWMCNrBXlLAjH43gCdUHCS66THKXIczz4zf9TUCooNAUdF6ViqsHhiOYAWOZSGhgYzmDRJwDkuSBLLE1VBcL68d//F11f0rG7Dnl3V3Zdcng3f0JYfayw/3rg+UgU8pFBz5mB5FR/xO1W4e/euYllXV0fWqq6ulnFgFp8yyL8S4zgat+JcLiTk5V/P7C0XLgVuBCGeMjY2lkgkFKqGlIjOyZMn4/G4PHrv3r3pc4Z5rnljAGuxx1NfFihPWYQzGRua6PzK1Jj9+dfnp9ve/67vn5H4SEgst3z6cfnxpreNVplD/izZUfOs5VRIgvrBSS6ViyNHjly8eNHkys6dO3EriOJ04GTkxo0b6IM6I+oFsMHpJrcoYEylac1UkBrGiMymt9TW1hI9AXUa7I8ePWqOMAlPocCbg/v27dPrSCSCQgCorIXEo3aijw2SA/pTIkGDzmJnC3s8PQtlMyZ7VaViqULq3nS4PuR5k5+PhB/icde7rES7BpMWNB2KpSTkjo4O7UG4OHfunFmiTAUtciaWCFx0jaSn/fQRvB9wC0tjUaaprhIDYOa06BO++ITrR48e4RMAFoOL8KDfB1uvExOZwunHrdKDoQ5m50Z/91twcOLS0JZFmsxob2+/fPkyrFLvyIVSHAWcQifpSgMqsLyvr49XjYOAJWBhe0q4RWs59rsMI8dgFWogIZCj7xnHEj0sBIUHDx5wo+iLnXiDa54icOby4+sVKJrlwBsGJ1IixUbiHW8yIq7BZQy6WIKnJAKYQTKkH43M2KLMazrRiJRi7womlCU5e25+tF3AjOGU6CSSh5gH80BdjCymvJEhPQm6PlKZqfsow1s+O+2xyXkYapNTEq3xHJcQ1rKfTlDNsabLJHj1QrwsiVT1FUvffOM4ZuGUxidAKN5k7PAew+Aw08ZTkr6lYV1mJ7HITgBIfNHu9uyO7WE29S4hLKZu3naVz7nRJyE3OX67FNYsNJI6ZC5AMCatvbNLNNOdxgeuNLsSF35ZiLBEazzovhMqLSUBQqqn43OlDe+NCgWv8sLn6RsVUy/8Z1gvvvmWouvaqIS8lxjye4viL1EPwwhMz40KIJmNq7krB3J1K2rKWlypJ0pQdom7RvOsSjtkTziJP2nfsF/g1AgTefz4sTnIQvCA7lgYd+Whf+07IaJz8EDwMQK+Drn1nLp1h78s3LHRP2jwjkZ6LCV+zDCF3lWaIPUp/4ojJMXhSmbG+3IgAKtwtKenwojmBmYmheB6v+PA3t5e4pLHqWGuQORGLGGQWoCRqImdEjHAybvpRafITHRQkD+/jx5DbjayFsIl+NtDoCV9nWeFo/HzJJZZdWTHprNpV4mCsCqZkqWsQi0EreCjXZO4WC7ki6ZEdzgAhp2S/wGVCcU8zwYCWeeqQwGyrmJzdpwLKe990urXCr3Fu6SENhUGsJjJycn5+XlZf1NTU1dX17Zt28ykJ80OFJSiK3kYNfUCszHCq8yGApPjDfEmak+fPq2oqJBjCs3MCwsLqO3fv98ckcllZszDMO7llanOnDkD15kHI3mWIMFT5LBJ9hhiv1pVX18vjyYbV6SEkemUCC8xwFyIKZl9G2GisztPJ7dFTukHP3yfq+/26VkJlA1zhLZqJLPPOze3tuTJDvs9zQLASbOaD0SpmulHg1aykOIsaDQTG8rtV78qL5zP7ar0vLuAn1UVROwXM//HyVa3NGAQsOUPz8vIlx0WywKzUzlqf9SweuB8ewBkf3K0muBcY38QuMrgVLE/111VcFr5T3a2ViycVpZD/hZgAP9XlxwC+KCNAAAAAElFTkSuQmCCAAAAAAAAAAA=


_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jgit-dev