Eclipse Community Forums: EGit / JGit » [solved]JGIT Push takes a long time to add an empty commit (no changes) when executed from cloud

Home » Eclipse Projects » EGit / JGit » [solved]JGIT Push takes a long time to add an empty commit (no changes) when executed from cloud

[solved]JGIT Push takes a long time to add an empty commit (no changes) when executed from cloud [message #1858798]

Sat, 22 April 2023 15:58

Eclipse User

Hello everyone and beware the ask is fuzzy and I would not have posted it here, but I'm really puzzled.

Context: I'm using JGIT 6.5.202303070854-r in a custom application that runs as a plugin inside another Java app inside Tomcat. For all intents and purposes it's a normal Java implementation. The end result is that methods like Push and Pull are exposed as webservices

Behavior: When this plugin is deployed in a cloud environment (which means the host app is deployed in a kubernetes architecture) what I observed is that even if Pull executes relatively ok, Push on the other hand executes very slow, making the webservice request timing out (socket timeout exception (read timeout)).

What I tried: The same Push in my own local Tomcat instance executes in 5-6 seconds, while in cloud I sat once 15 minutes and it was still executing.I observed that essentially the commit is being pushed successfully with all its content, as far as I checked (though I did not do a binary check on the file count/contents) All the tests I did were with empty commits and with a 22MB repo with 532 files with 5 commits. I did observe that during the execution time, I have the following error: ".git/refs/remotes/origin/.probe-99c3f223-6a2c-4535-8c39-0c34f8f44ff6: Operation not permitted" issued from JGit-FileStoreAttributeReader-2 (the thread name). However this error does not happen all the time - last time the service executed successfully but in 5,8 minutes without any error.

I am currently in the process of retrieving the Kubernetes node logs to see if there's anything there that might point to this, but if anyone encountered this I would be grateful to understand why that happens.

The Push implementation is this, in case it matters, wrapper in a try/catch: `

myGitFolder.add().addFilepattern(".").setUpdate(true).call();
myGitFolder.commit().setAll(true).setMessage(Message).setCommitter(str_CommitterName, str_CommitterEmail).call();

CredentialsProvider credentialsProvider = new UsernamePasswordCredentialsProvider(str_User, str_Password);

Iterable<PushResult> prList = myGitFolder.push().setRemote("origin")
.setCredentialsProvider(credentialsProvider).call();

`
I'm also in the process of creating a Java disk performance benchmark (not something very complex) to look at the disk read performance first, then other metrics, like file access time (?) since I suspect the Azure storage account might have some limitations.

Expectation: I'm expecting to have Push with the above implementation execute in under 10 seconds with an empty commit.
Question: since JGit does not seem to have internal TRACE logging I can active, I feel to ask very blindly if anyone has seen this kind of behavior in an infrastructure like mine (cloud).
I suspect the Azure storage account that persists the Git repository files has some sort of limitations (I'm not good with Azure, just aware). But, even with that, it seems that for 500 files jgit should not spend 15 minutes to push an empty commit. HTTP request speed does not seem to be the culprit, since pull works fast.

[Updated on: Mon, 08 May 2023 09:59] by Moderator

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858815 is a reply to message #1858798]

Mon, 24 April 2023 09:20

Eclipse User

Can you provide a stack trace for the operation not permitted error ? JGit tries to measure resolution of filesystem timestamps
by lazily writing to some .probe-<hash> files.

JGit uses slf4j 1.7. to write logs, if you bind it to some logging implementation you should be able to configure and gather logs and traces.

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858828 is a reply to message #1858815]

Tue, 25 April 2023 08:04

Eclipse User

Thanks for replying Matthias.
I will try to replicate the operation not permitted. It does not happen all time.

However, as the next best thing, I managed to create a stacktrace each 30 seconds from the system at hand for this operation.
As a result, I have 21 stacktraces (21stacktraces*30seconds = 630 seconds=10,5 minutes) for my service, which I'm hoping would allow you to point me where should I look.

This was an empty commit that I added.

I cleaned up the stacktraces until my custom service Push for clarity.

Attachment: StackTrace.txt
(Size: 23.85KB, Downloaded 70 times)

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858834 is a reply to message #1858828]

Tue, 25 April 2023 15:20

Eclipse User

In the same context, but another question to verify what I'm doing is right.

I'm running two times the add command, once without setUpdate and another time with setUpdate(true).

This is because I understand that their behavior is exclusive from the perspective of adding new files - and I want to track new, modified and deleted files and folders.

myGitFolder.add().addFilepattern(".").call();
myGitFolder.add().addFilepattern(".").setUpdate(true).call();

I assume that's the correct way based on my expectation, right? (eg: no way to optimize these calls as a single command?)

(the reason I'm asking this is because I see that both these Add executions add time to the overall process, so I was trying to see if I can optimize just a bit - even if the underlying problem still exists)

[Updated on: Tue, 25 April 2023 15:20] by Moderator

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858840 is a reply to message #1858834]

Wed, 26 April 2023 04:16

Eclipse User

I continued my investigation based on the stacktrace I provided earlier.
As a pattern in the 21 stacktraces I retrieved, the code seems to be staying in the AddCommand, specifically these lines:
call(AddCommand.java:200) - 4 times
call(AddCommand.java:206) - 3 times
call(AddCommand.java:207) -13 times
call(AddCommand.java:209) -1 time

The corresponding JGIT source code is:

                                        long len = f.getEntryContentLength(); - line 200
					// We read and filter the content multiple times.
					// f.getEntryContentLength() reads and filters the input and
					// inserter.insert(...) does it again. That's because an
					// ObjectInserter needs to know the length before it starts
					// inserting. TODO: Fix this by using Buffers.
					try (InputStream in = f.openEntryStream()) { - line 206
						ObjectId id = inserter.insert(OBJ_BLOB, len, in); - line 207
						entry.setObjectId(id);
					} - line 209

I'm suspecting the Azure Storage Account type used here, but, still, I feel the times spent here are way too high.

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858846 is a reply to message #1858840]

Wed, 26 April 2023 06:21

Eclipse User

For completeness, I was able to retrieve a stacktrace for the error I mentioned initially.
As a reminder, this happens sometimes, during the long execution time of my method. Usually once as I observed.

2023-04-24 12:57:04.054+0000 [L: ERROR] [O: o.e.j.u.FS] [I: ] [U: ] [S: ] [P: ] [T: JGit-FileStoreAttributeReader-1] /[redacted]/.git/.probe-47918e5d-1e22-4261-899a-6b6eb381b3aa: Operation not permitted
java.nio.file.FileSystemException: [redacted]/.git/.probe-47918e5d-1e22-4261-899a-6b6eb381b3aa: Operation not permitted
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.setTimes(UnixFileAttributeViews.java:125)
at java.base/java.nio.file.Files.setLastModifiedTime(Files.java:2355)
at org.eclipse.jgit.util.FS$FileStoreAttributes.getFsResolution(FS.java:608)
at org.eclipse.jgit.util.FS$FileStoreAttributes.measureFsTimestampResolution(FS.java:566)
at org.eclipse.jgit.util.FS$FileStoreAttributes.lambda$3(FS.java:430)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

I think my comment above will probably be able to point in the right direction.
I also found out the Azure Storage account type is Premium File Shares, so it's the high performance one (though I'm not sure which settings are enabled in it)

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1858848 is a reply to message #1858846]

Wed, 26 April 2023 07:07

Eclipse User

I'm investigating now if there's an antivirus installed in there that might interfere.

Re: JGIT Push takes a long time to add an empty commit (no changes) when executed from a Cloud syste [message #1859006 is a reply to message #1858848]

Fri, 05 May 2023 06:51

Eclipse User

Here's an update for whoever hits this issue in the future.
The slowness exhibited in my case is due to three compound aspects:
- the frequent file read/write operations executed by the Java JGIT library when pushing data (even with an empty commit)
- a large file count (started being visible after 80 files per repo)
- use of Azure Files for file persistence in the kubernetes architecture my app is hosted. They mention this issue here.

I performed a lot of tests, and if anyone would like to have a deep dive, please ping me with DM to discuss - simply too much to share here.

[Updated on: Fri, 05 May 2023 06:54] by Moderator

Previous Topic:	Possible to use a SSH Public Key for Commit Signing?
Next Topic:	planns for using jakarta EE 10

Goto Forum:

-=] Back to Top [=-

Current Time: Thu May 15 18:30:44 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter