[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] [egit-dev] Git LFS

I've implemented a workaround that passes remote-name instead of remote-location (URI) and the native pre-push hook is working as excpected with JGit.

I've filed a bug for EGit:Âhttps://bugs.eclipse.org/bugs/show_bug.cgi?id=546566

The Transport.open() has to be called with remote-name instead of remote-location in this case from EGit...


Am Do., 18. Apr. 2019 um 12:56ÂUhr schrieb Bossert, Andre <anb0s@xxxxxxxx>:
OK, i've the problem or difference taht cases native git-lfs to push alle objects.

native git trace output:
...
11:06:52.571531 trace git-lfs: exec: git 'version'
11:06:52.688537 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:06:52.785543 trace git-lfs: exec: git 'config' '-l'
11:06:52.832546 trace git-lfs: pre-push: refs/heads/jgit_lfs 73e7e71578ef06082156fcb967d521a712dcf092 refs/heads/jgit_lfs 4b6e83212e9e5caaa7a3bed34de2c35e0ef42d8a
11:06:53.615590 trace git-lfs: tq: running as batched queue, batch size of 100
11:06:53.615590 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
11:06:53.829603 trace git-lfs: tq: sending batch of size 1
11:06:53.829603 trace git-lfs: api: batch 1 files
...

jgit trace output:
...
11:05:12.289795 trace git-lfs: exec: git 'version'
11:05:12.426803 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:05:12.517808 trace git-lfs: exec: git 'config' '-l'
11:05:12.574811 trace git-lfs: pre-push: refs/heads/jgit_lfs 73e7e71578ef06082156fcb967d521a712dcf092 refs/heads/jgit_lfs 4b6e83212e9e5caaa7a3bed34de2c35e0ef42d8a
11:05:13.347856 trace git-lfs: tq: running as batched queue, batch size of 100
11:05:13.347856 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=https://myserver/myrepo.git --
11:05:13.408859 trace git-lfs: tq: sending batch of size 100
11:05:13.411859 trace git-lfs: api: batch 100 files
...

You see the difference that "git rev-list" is called with "--remotes=origin" parameter that uses in native git hook and "--remotes=https://myserver/myrepo.git" with repo-URL.
This causes to transfer "api: batch 1 files" in good case and "api: batch 100 files" in bad case.Â

If you try this commands at command line, e.g. here instead of --stdin "git rev-list 73e7e71578ef06082156fcb967d521a712dcf092 --objects --not --remotes=origin --" then it will return the right diff (sha1) with one file.
With "git rev-list 73e7e71578ef06082156fcb967d521a712dcf092ÂÂ--objects --not --remotes=https://myserver/myrepo.git" it will return the whole history with all commits!

The parameters are passen to the hook and inside hook it looks like:
Native Git hook call:"git lfs pre-push origin https://myserver/myrepo.git"
JGit Git hook call:Â Â "git lfs pre-push https://myserver/myrepo.git https://myserver/myrepo.git"ÂÂ

The question is why instead of remote name JGit passes the (double) URL to the hook ?




Am Do., 18. Apr. 2019 um 10:29ÂUhr schrieb Bossert, Andre <anb0s@xxxxxxxx>:
Hello,

i've added some code to support MSYS2 bash (Git for Windows) and now the hooks are executed that solves some issues. But now the pre-push hook is called in context of EGit push dialog with zero Old-Object-Id, e.g.:
refs/heads/jgit_lfs 73e7e71578ef06082156fcb967d521a712dcf092 refs/heads/jgit_lfs 0000000000000000000000000000000000000000

That is passed to "git lfs pre-push" in stdin and now native LFS logic uploads all objects, instead of only diff between local and remote! This goes wrong, because some of the LFS objects are not in local cache and missing.

This happens during "Trying pushing to remote repositories" and the real push will execute all the steps again and then push. So in case of LFS hook, it makes no sense to upload the binary data from my point of view.
If this can be skipped, i hope the Old-Object-Id in the second call be non zero and LFS push will upload only needed binary data...

Regards,
Andre

Am Do., 18. Apr. 2019 um 10:03ÂUhr schrieb Bossert, Andre <anb0s@xxxxxxxx>:
Hello,

my investigations so far:
- native git hooks are not supported at Windows (without cygwin)
- Git for Windows (MSYS2 = cygwin fork) is not supported
- this explains whyÂpost-commit and pre-push actions are not executed
- this explains why "git lfs status" shows different output to native git and binary data is not transferred to LFS server
- there is no error message and no abort of push --> breaks consistency between LFS and Git server --> fetch in another clones shows missing LFS objects
- JGit builtin LFS works with embedded hooks, but push to HTTPS enabled server (Artifactory) is not supported

From my point of view LFS is not usable for us at Windows and there is no clear documentation howto use and what are the pitfalls.

I will try to adress the main points we need to proceed with JGit LFS.

If somebody has some more input to clarify and show how it can work, please write some hints. Thank You!

Regards,
Andre

Am Mi., 17. Apr. 2019 um 13:38ÂUhr schrieb Bossert, Andre <anb0s@xxxxxxxx>:
Hi,

Thanks :)

I've found that in the Mehod "org.eclipse.jgit.hooks.GitHook.doRun()" the call "org.eclipse.jgit.util.ProcessResult.isExecutedWithError()" returns false and the status is ProcessResult.Status.NOT_SUPPORTED.
So there is no error, exception and no log if hook was not executed :(
...
if (result.isExecutedWithError()) {
throw new AbortedByHookException(
new String(errorByteArray.toByteArray(), UTF_8),
getHookName(), result.getExitCode());
}
...

Why the hook is not supported, but Filters are supported is the question here. May be related how MSYS2 Git Bash at Windows is detected...

Regards,
Andre



Am Mi., 17. Apr. 2019 um 13:17ÂUhr schrieb Duft Markus <Markus.Duft@xxxxxxxxxxxxxxxx>:

Hi,


Hook handling is generic and should be used whenever useJGitBuiltin is not set...â We had situations where the native GIT LFS was used (and working) because we forgot to enable this :)


Unfortunately I don't know locations in the code from the top of my head...


Cheers,

Markus



From: Bossert, Andre <anb0s@xxxxxxxx>
Sent: Wednesday, April 17, 2019 11:28
To: Duft Markus
Cc: Andre Bossert; JGit Developers list; EGit developer discussion (egit-dev@xxxxxxxxxxx); Andre Bossert
Subject: Re: [egit-dev] Git LFS
Â
Hello Markus,

Thank You for the answer!

If HTTPS is not implemented or not working can we use native Git LFS with JGit? So then the pre-push hooks should etc. should just work... Do you have hints where to search, set breakpoints etc. to analyse why they are not executed...

Regards,
Andre

Duft Markus <Markus.Duft@xxxxxxxxxxxxxxxx> schrieb am Mi., 17. Apr. 2019, 11:18:

Hey,


I implemented parts of this...


We are using JGit LFS in production for some time now (~1Âyear) with thousands of users.


We are using authentication through SSH (using Gerrit and the Gerrit LFS plugin, which supports obtaining authentication tokens using 'ssh server git-lfs-authenticate project').


If nothing changed in the meantime, I guess HTTP(S) authentication is simply not supported (yet). Matthias did some tests against GitHub back then, and AFAIR they were successful, but I don't know what was the exact setup for the tests...


HTH,ÂCheers,

Markus




From: egit-dev-bounces@xxxxxxxxxxx <egit-dev-bounces@xxxxxxxxxxx> on behalf of Bossert, Andre <anb0s@xxxxxxxx>
Sent: Wednesday, April 17, 2019 10:31
To: JGit Developers list (jgit-dev@xxxxxxxxxxx); EGit developer discussion (egit-dev@xxxxxxxxxxx)
Cc: Andre Bossert
Subject: Re: [egit-dev] Git LFS
Â
I've found the option "lfs.useJGitBuiltin = true" that enables builtin LFS support. The pre-push hooks shows an error / exception, because of the autorization
...
!STACK 0
org.eclipse.jgit.errors.TransportException: When trying to open a connection to https://[server]/artifactory/[project]/objects/batch the server responded with an error code. rc=401
at org.eclipse.jgit.transport.Transport.push(Transport.java:1365)
at org.eclipse.egit.core.op.PushOperation.run(PushOperation.java:191)
at org.eclipse.egit.ui.internal.push.ConfirmationPage$2.run(ConfirmationPage.java:211)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)
Caused by: java.io.IOException: When trying to open a connection to https://[server]/artifactory/[project]/objects/batchÂÂthe server responded with an error code. rc=401
at org.eclipse.jgit.lfs.LfsPrePushHook.requestBatchUpload(LfsPrePushHook.java:214)
at org.eclipse.jgit.lfs.LfsPrePushHook.call(LfsPrePushHook.java:123)
at org.eclipse.jgit.transport.Transport.push(Transport.java:1363)
...

There is no dialog to enter the username and password. How can we enable this?

I've found only this information about LFS:
...

Eclipse JGit: Java implementation of Git 4.2

Support git-lfs extension

  • Add support for pre-push hooks

  • Add support for smudge filters.ÂIf defined in .gitattributes call smudge filter during checkout.
  • Add support for clean filters.ÂIf defined in .gitattributes call clean filter during add to index.
...

The pre-push hook is not executed in the case native git lfs is used, but it looks like the filters are executed... Is there any deeper information about JGit LFS ?

Am Di., 16. Apr. 2019 um 10:30ÂUhr schrieb Bossert, Andre <anb0s@xxxxxxxx>:
Hello,

we are using native Git with LFS support. The LFS files are pushed to an Artifactory server (https).
Filters (smudge and clean) and Hooks (post-checkout, pre-push etc.) are executed.

Now i'm trying to use Eclipse EGIt / JGit and it looks different:

- after a binary file (LFS tracked) is added to staging area "git lfs status" shows it in "Git LFS objects to be committed:" and "Git LFS objects not staged for commit:", but expected and seen in native Git it should be in "Git LFS objects to be committed:" only.

- push does not send the file to Artifactory, but there is no exception / error etc. The "oid sha256" from the git pointer file cannot be found at Artifactory server.

- it's also not clear for us if native git lfs is used or the embedded JGit LFS - are both ways still supprted? If yes how can we choose?

How can we analyse the issues? Is there any deep documentation about how it works or some FAQ where to search or enable some debug log etc.?

Thanks and Regards,
Andre



--
Regards
Andre (anb0s)

SSI SchÃfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht fÃr Zivilrechtssachen Graz

SSI SchÃfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht fÃr Zivilrechtssachen Graz


--
Regards
Andre (anb0s)


--
Regards
Andre (anb0s)


--
Regards
Andre (anb0s)


--
Regards
Andre (anb0s)
eMail: anb0s@xxxxxxxx


--
Regards
Andre (anb0s)
eMail: anb0s@xxxxxxxx