Eclipse Community Forums: EGit / JGit » Unexpected reply from ssh-agent: SSH_AGENT

Home » Eclipse Projects » EGit / JGit » Unexpected reply from ssh-agent: SSH_AGENT_FAILURE(Eclipse on Windows 10 to pull a GitHub repo using a YubiKey Hardware Ke)

Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859947]

Thu, 06 July 2023 10:48

Eclipse User

I am trying to use Eclipse on Windows 10 to pull a GitHub repo using a YubiKey Hardware Key

I get the error "Unexpected reply from ssh-agent: SSH_AGENT_FAILURE"

Using Git from the command line to pull the repo on Windows 10 with the YubiKey works.

Using Eclipse on my Mac to pull the same repo with the YubiKey also works without error.

I have updated Eclipse to the latest version (but have had the error with older versions too).

Eclipse is:
Version: 2023-06 (4.28.0)
Build id: 20230608-1333

Git integration for Eclipse 6.6.0.202305301015-r org.eclipse.egit.feature.group Eclipse EGit

I am using the Win32 Open SSH from Microsoft. Eclipse Git Preferences are set to
Use SSH agent for SSH connections = true, and Default SSH agent = "Win32 OpenSSH". The ssh-agent is running.

I have found "SSH_AGENT_FAILURE" in the OpenSSH source code, so the error suggest to me that JGIT has successfully connected to the ssh-agent

The Stacktrace is:
org.eclipse.jgit.api.errors.TransportException: git@github.ibm.com:OTMS/current.git: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:249)
at org.eclipse.jgit.api.PullCommand.call(PullCommand.java:266)
at org.eclipse.egit.core.op.PullOperation$PullJob.run(PullOperation.java:256)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.eclipse.jgit.errors.TransportException: git@github.ibm.com:OTMS/current.git: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE
at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:263)
at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:281)
at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:153)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:153)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:105)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1462)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:238)
... 3 more
Caused by: org.apache.sshd.common.SshException: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE
at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:141)
at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:56)
at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:35)
at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:74)
at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:172)
at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:101)
at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:256)
... 11 more
Caused by: org.apache.sshd.common.SshException: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE
at org.eclipse.jgit.internal.transport.sshd.agent.SshAgentClient.sign(SshAgentClient.java:211)
at org.apache.sshd.client.auth.pubkey.KeyAgentIdentity.sign(KeyAgentIdentity.java:63)
at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.appendSignature(UserAuthPublicKey.java:446)
at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.processAuthDataRequest(UserAuthPublicKey.java:413)
at org.apache.sshd.client.auth.AbstractUserAuth.process(AbstractUserAuth.java:88)
at org.apache.sshd.client.session.ClientUserAuthService.processUserAuth(ClientUserAuthService.java:345)
at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:267)
at org.apache.sshd.common.session.helpers.CurrentService.process(CurrentService.java:109)
at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:592)
at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:523)
at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:522)
at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1649)
at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:483)
at org.eclipse.jgit.internal.transport.sshd.JGitClientSession.messageReceived(JGitClientSession.java:197)
at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:407)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:380)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:375)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859955 is a reply to message #1859947]

Thu, 06 July 2023 15:33

Eclipse User

Yes, JGit has successfully connected to the agent. In fact, I think it has at that point also successfully obtained the public key from the agent. I don't know why the Win32-OpenSSH agent then fails when asked to sign some data.

Are you sure that using git on the command line it is using Win32-OpenSSH and its agent? I'm not sure git bash would do so...

What is the key type? Probably some sk-* ... it's also possible that the Apache MINA sshd library used by JGit has a bug with these keys.

Personally, I use neither Windows nor U2F/FIDO keys, so I won't really be able to help.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859958 is a reply to message #1859955]

Thu, 06 July 2023 17:30

Eclipse User

HI Thomas
I don't normally use Windows myself very often, but sometimes I have to . ..

The key is a RSA key.

We are using YubiKey PIV.

The only "proof" that I have that Windows command line GIT is using Win32-OpenSSH and the agent is if I stop the agent service, GIT hangs.

If the ssh-agent service is running, I get prompted on the command line to enter the YubiKey pin.

Is there anything I can do to debug this?

Being a java developer I tried writing a test project calling JGIT. I got as far as being able to clone a simple repo from GithHub (e.g the JGIT repo at https://git.eclipse.org/r/jgit/jgit.git), but got stuck calling a SSH repo

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859978 is a reply to message #1859958]

Fri, 07 July 2023 11:06

Eclipse User

I think I have got Eclipse / JGIT working on Windows after hours of experimentation.

The preconditions seem to be:
a) Pageant (Putty SSH Agent) is NOT running: kill with Task Manger if running.
b) SSH_AUTH_SOCK is NOT set: Unset in env vars (e.g on reboot it is set by "something" for the pageant pipe"). This is even more important than a) above.
c) OpenSSH Authentication Agent service is running. (I am currently using v9.2.2.0 from https://github.com/PowerShell/Win32-OpenSSH, not the OpenSSH from Microsoft (Apps and Features / Optional Features),
d) YubiKey Hardware Key is inserted.
e) Eclipse / JGIt "Use SSH Agent for SSH Connections" = true, Default SSH_Agent = "Win32 OpenSSH",

Once all of that is checked and double-checked:

1) open a NEW cmd shell. Double check that SSH_AUTH_SOCK is not set for that session (for an old shell it may still be set....).
2) Ssh-add -L -> "The agent has no identities": Any other response / error implies one or more of the preconditions above are not met.
3) From Eclipse / JGIT, git pull -> pull fails "publicly: no keys to try": This is "OK" because the agent has no identities loaded.
4) ssh-add -s "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll" -> Enter Passphrase for PKCS#11" -> Card Added ..
5) ssh-add -L -> lists two sh-rsa keys
6) From Eclipse / JGIT, git pull -> pull is successful!!!!

If I perform similar tests with Git from the command line, at step 3) Git will prompt me for the Yubikey pin, and will succeed. i.e. I don't need to run step 4). But if I do run step 4, I no longer need to enter the pin.

Later this evening I will revert to the OpenSSH from Microsoft, to see if that caused the problem initially reported.

[Updated on: Fri, 07 July 2023 13:41] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859983 is a reply to message #1859978]

Fri, 07 July 2023 14:17

Eclipse User

I can now confirm that (at least on my Windows 10 workstation) part of the problem is down to the OpenSSH Client for Windows installed from Microsoft (Apps and Features / Optional Features). I was running version "OpenSSH_8.1p1 for Windows".

Using the alternative OpenSSH Client for Windows from https://github.com/PowerShell/Win32-OpenSSH, in my case, v9.2.0.0, I was able to get Eclipse / JGIt + OpenSSH + Yubikey PIV PCKS#11 working, if all the preconditions above are met.

Using the Microsoft version of OpenSSH Client the step 4) above failed with the error "Could not add card "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll": communication with agent failed"

Nevertheless less there are still some strange inconsistencies. If these are down to JGIT, or to the particular ssh-agent running I cannot say.

1) On Windows Eclipse / JGIT never prompts me for the YubiKey pin. This compares to:

2) On my Mac, Eclipse / JGIt does prompt me for the YubiKey pin via a popup dialog.

3) On Windows Eclipse / JGIT prompts me for a passphrase via a popup dialog when using a local key pair for ssh.

4) On Windows Git at the command line prompts me for the YubiKey pin.

On my Mac I am using this ssh-agent https://github.com/FiloSottile/yubikey-agent. It just works: having installed it I do not need to check any-preconditions like on Windows.

It looks like the popup dialog for the YubiKey pin on my Mac (which pops both from Eclipse and the macOS terminal) is from the yubikey-agent, because it says so at the top of the dialog.

In all my experimentation on Windows, I have so far disregarded the Putty ssh-agent Pageant. Possibly it might work in combination with something like https://github.com/bitlogik/PIVageant

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859984 is a reply to message #1859978]

Fri, 07 July 2023 14:19

Eclipse User

That's interesting. Maybe we are doing something wrong in JGit -- I don't think we've implemented anything for ssh-add -s "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll". Command-line git uses the normal SSH executable, and apparently that does this automatically. I'll have to look what that ssh-add -s does exactly, and whether normal SSH would do it automatically.

How does your ~/.ssh/config look like for the host? Are AddKeysToAgent and SecurityKeyProvider set? Or does this need PKCS11Provider?

But it's still a little bit strange; the stack trace you posted originally shows a failure during signing, using a KeyAgentIdentity, so JGit must have gotten a public key from the agent. How did that key get into the agent in the first place, if this ssh-add -s was not run?

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859985 is a reply to message #1859984]

Fri, 07 July 2023 15:10

Eclipse User

After a brief look into the OpenSSH code, it appears that SSH will, if PKCS11Provider is set, automatically start a little helper application that behaves like an ssh-agent and that communicates with the shared library to put the smartcard key into this little special agent. That might explain why it just works with command-line git.

We have definitely not implemented this. So with JGit, the way to go in this case is indeed to manually or otherwise add the smartcard key to an existing agent via ssh-add -s, and then set up JGit to use that agent. The agent must understand the SSH_AGENTC_ADD_SMARTCARD_KEY command. Don't know whether all do.

Implementing this in JGit or in Apache MINA sshd even only in the SSH client library would be a lot of work involving loading a shared native library given by name at runtime and then communicating with that library to figure out the interface methods and then call them. And for safety reasons, that dynamic library stuff should really be in a separate process. The code in OpenSSH for doing it doesn't look exactly trivial, and via Java it might be even more complicated. Possibly doable with JNA, but non-trivial, and certainly different for Unices or for Windows.

BTW: also interesting: https://www.exploit-db.com/exploits/40963 . (Gives also some insights into how this works in OpenSSH at all.) Looking into ssh-agent.c, it appears that the agent has a whitelist of wildcard glob patterns and only loads libraries that match a whitelisted pattern.

P.S.: just as an aside: JGit doesn't do agent forwarding. If you need to hop through servers to get to the git repo, use ProxyJump instead.

[Updated on: Fri, 07 July 2023 15:13] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859987 is a reply to message #1859985]

Sat, 08 July 2023 04:16

Eclipse User

Hi Thomas

Thanks.

Somewhat frustratingly I can no longer recreate the original error at the start of this mail. In my experimentation I have changed so many things back and forth, and many preconditions are required to get things working.

I suspect it was down to particular OpenSSH for Windows implementation. With the latest v9.2.0.0 it currently works.

In someways the original error is secondary: my real goal is to be able to use Eclipse / JGIT reliably on Windows with the YubiKey PIV.

My .shh/config looks like this:

Host github.acme.org
 
  Port 22
  User git
  #LogLevel DEBUG3
 
  #Config for YuiKey
  PKCS11Provider "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll"
  AddKeysToAgent yes

B.T.W on my Mac the ssh-agent I use is a dedicated agent for YubiKey from FiloSotille, not an OpenSSH agent, so that may account for some of the differences between Mac and Windows. Sadly the Windows implementation is "work-in-progress", probably with low priority.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1859989 is a reply to message #1859987]

Sat, 08 July 2023 08:44

Eclipse User

Looking a little more around in Java, it appears that all this smartcard and shared library interaction can be handled through the SunPKCS11 security provider. Instantiate such a provider dynamically, create a PKCS11 KeyStore with that provider and a password callback, and use that to get the key.

At that point, this starts looking to be possible to implement much more easily -- but still not exactly simple.

Sounds interesting. I'll try to whip up some code, and perhaps even test it with softhsm. And then figure out how to do it all in a separate process (such that it works also in an OSGi environment like Eclipse).

But don't hold your breath. If you'd like to give this a try yourself and come up with something nice: a Gerrit change (or changes) for this would be most welcome.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860014 is a reply to message #1859989]

Mon, 10 July 2023 17:37

Eclipse User

Here's my take on this: https://git.eclipse.org/r/c/jgit/jgit/+/203078 .

Didn't do the loading in a separate process, though: which library is loaded is totally under control of the user via ~/.ssh/config, so there's no great risk. A PKCS#11 HSM vendor's library also had better be trustworthy, and if that dynamic library loading were a security risk, Java should provide different and safer APIs in the first place. That it doesn't is IMO another indication that it's not really a problem.

I could test only with SoftHSM; it would be great if you could give this a try with a real YubiKey PIV. For me it worked fine in Eclipse with SoftHSM.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860016 is a reply to message #1860014]

Tue, 11 July 2023 02:32

Eclipse User

Hi Thomas

Thanks, I will look into that in the next few days (today I am "away" from my Windows laptop).

In the meantime, after more experimentation, I have discovered that the key obstacle that prevents the combination of JGIT + OpenSSH + YubiKey working is the incorrect setttng of the environment variable SSH_AUTH_SOCK.

In particular it must be correctly set in the Windows cmd session in which the OpenSSH ssh-add commands are executed. It does not matter if SSH_AUTH_SOCK is generally set to something else (e.g for Putty Pageant).

So that the ssh-add command works with OpenSSH,in the cmd session from which ssh-add is called, SSH_AUTH_SOCK must be set to either:
SET SSH_AUTH_SOCK=
or
SET SSH_AUTH_SOCK=//./pipe/openssh-ssh-agen

either are correct for OpenSSH on Windows.

Once so set:

ssh-add -s "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll"

should work with error: i.e. prompt for the YubiKey pin

After that, the YubiKey private keys should be loaded to the OpenSSH ssh-agent, and anything properly using that, including Eclipse / JGIT, should no longer prompt for the pin.

Curiously, once the keys have been successfully loaded to the OpenSSH agent, they seem to be persisted, and are loaded even after a reboot!

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860022 is a reply to message #1860016]

Tue, 11 July 2023 10:39

Eclipse User

Hi Thomas

Do you have an test code I could use to test your changes with the YubiKey? I'd rather not reinvent the wheel and write my own code if I don't have ....

I did find some test code here org.eclipse.jgit.ssh.apache.test/tst/org/eclipse/jgit/transport/sshd that might help.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860023 is a reply to message #1860022]

Tue, 11 July 2023 13:06

Eclipse User

Easiest way to test this:

Install Egit nightly in Eclipse. Use the update site https://download.eclipse.org/egit/updates-nightly . Restart Eclipse. This ensures that you have the newest EGit and that the next step actually works.
Download the zip of the target/repository directory listed at https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4016/artifact/ . That was created by the CI build of that change.
Unpack that zip to some directory. The directory contains a p2 update site.
Check what JGit bundles you have in your eclipse, and update them all from the unzipped p2 repository in that directory. (You might be prompted about unsigned bundles. I'm not sure the pre-merge CI build signs them.)
Restart Eclipse.
Ensure that you do not have your YubiKey in the SSH agent. (That should no longer be necessary.)
Try a git repository access (like fetching) from that eclipse, on a git server for which you have set PKCS11Provider. (And no other key specified.) You should be prompted for the PIN of your YubiKey PIV, and the fetch should proceed. (It might report "no changes", though.)

The repository in step 2 is temporary; we keep only build artifacts of the last five CI builds.

Oh, and please read the README.md. If your token is not in slot index 0, also set PKCS11SlotListIndex. PKCS11SlotListIndex is a temporary thing; I'll have to remove that again since it breaks compatibility with OpenSSH. If you need it, I'll have to figure out a better way.

[Updated on: Tue, 11 July 2023 16:39] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860055 is a reply to message #1860023]

Fri, 14 July 2023 09:21

Eclipse User

Hoi Thomas

The URL above https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4016/artifact/

give me a 404 Error

https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/lastSuccessfulBuild/artifact/

gives me Artifacts of "jgit.gerrit-pipeline.java11 #4045". However a quick look in the sources indicates that the new pkcs11 classes are not included. So I guess that won't help me much.

Looking here https://git.eclipse.org/r/c/jgit/jgit/+/203078 I see the build failing, which may explain the 404 error above.

mfg

Chris

[Updated on: Fri, 14 July 2023 10:02] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860056 is a reply to message #1860055]

Fri, 14 July 2023 10:24

Eclipse User

No, the build worked fine. I just left a -1 comment because this needs some cleanup before it can be merged.

The problem is that there were more than five builds since that one, so the artifacts are no longer there. I'll just retrigger the build, then there should be new artifacts in about 15 minutes.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860057 is a reply to message #1860056]

Fri, 14 July 2023 10:29

Eclipse User

Yes, I just saw that https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4047/ is building, triggered by your change. I will get that later this evening

Edit: just successfully downloaded, Thanks

[Updated on: Fri, 14 July 2023 10:43] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860058 is a reply to message #1860056]

Fri, 14 July 2023 11:01

Eclipse User

New artifacts available at https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4047/artifact/ .

In case they're already gone by the time you try again, you could also clone the JGit repo, fetch that change (click "Download patch" in the top-right menu at https://git.eclipse.org/r/c/jgit/jgit/+/203078 ) and then build JGit yourself. You need Java 17, then run first "mvn clean install -DskipTests", followed by "mvn -f org.eclipse.jgit.packaging/pom.xml clean install". Then you should have the repository at org.eclipse.jgit.packaging/org.eclipse.jgit.repository/target/repository at you should be able to install directly from there.

Edit: Oh, I see you already got it.

[Updated on: Fri, 14 July 2023 11:02] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860061 is a reply to message #1860058]

Fri, 14 July 2023 11:42

Eclipse User

Hi Thomas

As I have some time to kill on the train to Zürich, I have "practiced" the installation on my Mac, and then run some regression tests to verify that the patch does not interfere with the Filosottile YubiKey Agent I run on the Mac. So far it looks good.

Either later this evening, or sometime over the weekend I will repeat the installation and test on my Windows PC, where I had the problem first reported.

As a side detail the Filosottile YubiKey Agent on my Mac seems to be aggressive about loading the keys from the YubiKey, or does not allow them to be unloaded.

If I do "ssh-add -D", it reports "All identities removed.". If I do "ssh-add -L" immediately after, it shows the ssh-rsa key as loaded.

"ssh-add -e /usr/local/lib/libykcs11.dylib" gives "Could not remove card ... Agent refused cooperation".

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860062 is a reply to message #1860061]

Fri, 14 July 2023 12:06

Eclipse User

I suspect that this Filosottile agent just keeps the hardware key as long as it's plugged in. If you want to be sure Eclipse doesn't use that agent by mistake, just put "IdentityAgent none" in the ssh config.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860070 is a reply to message #1860062]

Sat, 15 July 2023 08:24

Eclipse User

I now have the patch installed on my Windows 10 machine as well. However I can see no change in behaviour.

If I unload the keys from the OpenSSH Agent ("ssh-add -D" or "ssh-add -e xxxx"), then attempt to git pull in JGIT, I get "publickey: ne keys to try". I am not prompted for the YubiKey pin.

I will double check that I am testing what I think I am testing: i.e that everything is the correct version,; and will experiment if any settings in .ssh/config make a difference.

Is there anything I can do to better debug?

[Updated on: Sat, 15 July 2023 08:49] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860071 is a reply to message #1860070]

Sat, 15 July 2023 10:02

Eclipse User

Verify that the YubiKey is indeed in slot index zero. If it's not in index 0, add PKCS11SlotListIndex with the right index.

Verify the version of the JGit bundles in Eclipse. Is it indeed the right version?

If you try the latest artifacts from https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4094/artifact/ : they have quite a bit of debug logging. You could also try starting Eclipse from the command line with debug logging switched on. How exactly debug logging can be switched on depends on which logging back-ends are present in your Eclipse. Maybe some logging back-end needs to be installed first.

Even with the artifacts from build 4047 having a logging backend in Eclipse and running it from the command-line may help; perhaps there will be some warnings logged. Perhaps the shared library cannot be loaded?

If you want to live debug this, set up an Eclipse for EGit/JGit development (easy via the installer, see https://wiki.eclipse.org/EGit/Contributor_Guide ). Fetch that Gerrit change and run a debug child Eclipse, then try to fetch from the git server inside that child Eclipse.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860072 is a reply to message #1860071]

Sat, 15 July 2023 10:31

Eclipse User

Thanks Thomas

These are the EGIT / JGIT versions I have installed on Windows 10

    Git integration for Eclipse	6.7.0.202307121016	org.eclipse.egit.feature.group	Eclipse EGit
      Java implementation of Git	6.7.0.202307141436	org.eclipse.jgit.feature.group	Eclipse JGit
      Java implementation of Git - GPG support using BouncyCastle	6.7.0.202307141436	org.eclipse.jgit.gpg.bc.feature.group	Eclipse JGit
      Java implementation of Git - optional Http support using Apache httpclient	6.7.0.202307141436	org.eclipse.jgit.http.apache.feature.group	Eclipse JGit
      Java implementation of Git - ssh support using Apache MINA sshd	6.7.0.202307141436	org.eclipse.jgit.ssh.apache.feature.group	Eclipse JGit

That should be ok for build 4047.

My .ssh/config is:
Host github.acme.com

Port 22
User git
LogLevel DEBUG3

  #Config for YubiKey
  PKCS11Provider "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll"
  # Line below does not work.
  ##IdentityFile "pkcs11:id=%01"
  # The line below ensures the OpenSSH ssh-agent is used.
  IdentityAgent //./pipe/openssh-ssh-agent
  ## The line below seems to trouble JGIT, but not Git on the commandline.
  #IdentitiesOnly yes
  AddKeysToAgent yes
  PKCS11SlotListIndex 9a

The YubiKey PIV key is in slot 9a.

Edit: I have just upgraded to 4094, and will attempt to debug with that patch level.
Edit 2: ... and I have a Eclipse for EGit/JGit development installing on Windows....

[Updated on: Sat, 15 July 2023 11:50] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860073 is a reply to message #1860072]

Sat, 15 July 2023 11:26

Eclipse User

Before I start debugging a child eclipse, I thought I would read through your code.

One detail I have noticed, is that YubiKey use alphanumeric names for the slots. e.g. the PIV key is in "9a" (see https://developers.yubico.com/PIV/Introduction/Certificate_slots.html).
I see slotListIndex is an int. Are we using "slot" with two different meanings here? of should slotListIndex be a String to store the value "9a"?

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860074 is a reply to message #1860073]

Sat, 15 July 2023 12:57

Eclipse User

Hi Thomas

Edit: With a bit of hacking to get around some errors, your change now works:

I am now happily debugging a Child Eclipse in which I am calling JGit to pull my repo with the YubiKey.

In the Parent Eclipse, I get this error:

[sshd-JGitSshClient[1dfee458]-nio2-thread-3] WARN org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication - HostConfig for host github.ibm.com (hostname github.ibm.com): could not instantiate PKCS11Provider C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll
java.security.InvalidParameterException: Error configuring SunPKCS11 provider
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.configure(SunPKCS11.java:122)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.lambda$0(Pkcs11Provider.java:145)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.getProvider(Pkcs11Provider.java:128)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.getPkcs11Keys(JGitPublicKeyAuthentication.java:516)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.getAgentIdentities(JGitPublicKeyAuthentication.java:411)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.initializeAgentIdentities(JGitPublicKeyAuthentication.java:348)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKeyIterator.<init>(UserAuthPublicKeyIterator.java:59)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.<init>(JGitPublicKeyAuthentication.java:320)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication.createPublicKeyIterator(JGitPublicKeyAuthentication.java:139)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.init(UserAuthPublicKey.java:108)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication.init(JGitPublicKeyAuthentication.java:125)
	at org.apache.sshd.client.session.ClientUserAuthService.tryNext(ClientUserAuthService.java:410)
	at org.apache.sshd.client.session.ClientUserAuthService.processUserAuth(ClientUserAuthService.java:331)
	at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:267)
	at org.apache.sshd.common.session.helpers.CurrentService.process(CurrentService.java:109)
	at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:592)
	at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:523)
	at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:522)
	at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1649)
	at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:483)
	at org.eclipse.jgit.internal.transport.sshd.JGitClientSession.messageReceived(JGitClientSession.java:208)
	at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:407)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:380)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:375)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
	at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.pkcs11.ConfigurationException: Unknown keyword 'Files\Yubico\Yubico', line 1
	at jdk.crypto.cryptoki/sun.security.pkcs11.Config.parse(Config.java:499)
	at jdk.crypto.cryptoki/sun.security.pkcs11.Config.<init>(Config.java:221)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:118)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:115)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.configure(SunPKCS11.java:115)
	... 35 more

The file that is being parsed has this content

name = JGit-0-C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll
library = C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll
slotListIndex = 0

I think paths with spaces are not being properly handled, hence the 'Files\Yubico\Yubico' reported in the error.

If I move the Yubico Dlls to a path with no spaces, and update the .ssh/config accordingly, then it works! I get a popup dialog requesting the Yubikey passphrase, and the Git pull succeeds.

On a second pull, I am not prompted again for the passphrase.

Curiously, ssh-add -L on the command line reports "The agent has no identities."

[Updated on: Sat, 15 July 2023 13:32] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860075 is a reply to message #1860074]

Sat, 15 July 2023 14:50

Eclipse User

Yes, I noticed this problem with spaces, too. But at least the basic mechanism appears to work; that's great.

Probably the name must not contain spaces. I can't believe that SunPKCS11 would have troubles with the library path having spaces. At least not with single spaces. Multiple consecutive spaces might be a problem. If that's the case, I have another idea.

I'll test some more with the library at paths with spaces, and push an update when I got it all sorted out.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860076 is a reply to message #1860075]

Sat, 15 July 2023 17:00

Eclipse User

Hoi Thomas

I agree, it is great that we have proved the basic mechanism works, at least on Windows.

Some thoughts at this stage are:
1) The problem with spaces is probably a minor issue, easily resolved. However this error seems to have been swallowed, and not re-thrown. I only caught it in the parent Eclipse instance. Had it been thrown to the child instance, we might have understood this error without debugging.
2) Having understood the "path with spaces problem*, I copied the YubiKey Dlls to a directory without spaces. Unfortunately, in my excitement, I made a typo (i.e. the entry in .ssh/config was not quite correct).. Similar to 1) above, this error was only reported in the parent Eclipse instance.
3) There is potential for confusion between "slot" and "slot index*. Using the YubiKey PIV, the Authentication key is stored in slot 9a. However the slotListIndex must be "0". Cleverly worded documentation should help clarify this.

I thank you for both your quick responses, and for your clear instructions. Only a few days ago I could not have imagined how easy it is to debug a child Eclipse instance. Once one knows how, it is child's play.

I have started installing a similar Eclipse / Jgit dbug setup on macOS. Hopefully that should be equally helpful on that OS.

mfg und bis bald,

Chris

[Updated on: Sat, 15 July 2023 17:01] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860077 is a reply to message #1860075]

Sat, 15 July 2023 17:31

Eclipse User

Yes, slot ID and slot list index are different things. "9a" would be the slot ID (and it's an unsigned long, just shown in hex). This ID may not be static; it may change when the token is removed and re-inserted, or possibly also when the library is loaded again. (Some libraries may provide stable slot IDs, though.) The slot list index is always an integer, and the index of the first slot is always zero. (Though I don't know what the "first" slot is if you have two YubiKeys plugged in. The one inserted first?)

Unfortunately the Java SunPKCS11 provider is rather limited in its ways to select a token. It can be done via slot ID or slot list index only, and the latter appears to be preferred. I see nothing to select a slot by token manufacturer or serial number or the like.

That the agent still reports it had no identities is expected. The YubiKey identity is not cached in the agent, it is cached in the YubiKey library. There is no agent involved at all in all this. As long as Eclipse runs, and the YubiKey is not removed, you should be prompted only once for the PIN. When the token is removed and re-inserted, you might be asked (in fact, should be asked) once again for the PIN.

JGit also ignores AddKeysToAgent for PKCS11Provider keys. With built-in PKCS11 support, that is not needed at all in JGit. It would make sense only for agent forwarding, but JGit doesn't do that anyway. If you have to hop through servers to get to the repository, use ProxyJump instead.

Anyway, the real problem was the name containing spaces, and indeed the Java SunPKCS11 provider has trouble with paths containing multiple successive spaces.

https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline.java11/4098/artifact/ has fixes for that and should work also with library paths that contain spaces.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860078 is a reply to message #1860077]

Sat, 15 July 2023 17:38

Eclipse User

Re: "error seems to have been swallowed, and not re-thrown."

Eclipse still has no real integration of slf4j or Java logging into the OSGi logging. Probably the code in JGitPublicKeyAuthentication should only catch and log UnsupportedOperationException, but propagate all others. Or perhaps propagate any.

Edit: fixed now.

[Updated on: Sun, 16 July 2023 04:13] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860088 is a reply to message #1860078]

Sun, 16 July 2023 09:49

Eclipse User

This afternoon I did a quick retest on Windows 10 with 4099, both in my JGIT Debug Parent + Child Eclipse, and then as a artifact added to my normal Eclipse. I can confirm that the paths with spaces now work, and I am prompted for the YubiKey pin on first use as expected.

Thanks.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860107 is a reply to message #1860088]

Tue, 18 July 2023 06:09

Eclipse User

Thanks a lot for your testing; the change is merged in JGit now and is available in the EGit nightly repository at https://download.eclipse.org/egit/updates-nightly/ .

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860139 is a reply to message #1860107]

Wed, 19 July 2023 07:48

Eclipse User

Hi Thomas

Some more testing:

If I try to do a SSH from the commandline, on both the Mac and Windows I get the error

/Users/christopherlamb/.ssh/config: line 20: Bad configuration option: pkcs11slotlistindex
/Users/christopherlamb/.ssh/config: terminating, 1 bad configuration options

It looks like ssh is objecting to the key "pkcs11slotlistindex". Is that still used in the latest code? I think you said it would be temporary.

As promised I set up an Eclipse / JGIT debug setup on my Mac as well, with patch 4103.

This got as far as popping the dialog requesting the YubiKey passphrase / pin, but then failed with this error "Secure storage was unable to retrieve the master password from the OS keyring. Make sure that this application has access to the OS keyring". (full stack a the end of this mail"

I suspect that the error is an Apple security thing ... down to quaranting or signing ...

Pulling 1 repository
git@github.ibm.com:OTMS/w21.7.git: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
git@github.ibm.com:OTMS/w21.7.git: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR

WARNING: Using incubator modules: jdk.incubator.foreign, jdk.incubator.vector
2023-07-16 15:53:47.946 java[34836:577917] +[CATransaction synchronize] called within transaction
2023-07-16 15:58:12.303 java[34836:577917] +[CATransaction synchronize] called within transaction
2023-07-16 15:58:12.462 java[34836:577917] +[CATransaction synchronize] called within transaction
2023-07-16 16:01:03.210 java[34836:577917] +[CATransaction synchronize] called within transaction
[Worker-26: Pulling origin from git@github.ibm.com:OTMS/w21.7.git] INFO org.apache.sshd.common.util.security.eddsa.EdDSASecurityProviderRegistrar - getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
[Worker-26: Pulling origin from git@github.ibm.com:OTMS/w21.7.git] INFO org.apache.sshd.common.io.DefaultIoServiceFactoryFactory - No detected/configured IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
!SESSION 2023-07-16 15:51:31.458 -----------------------------------------------
eclipse.buildId=unknown
java.version=17.0.7
java.vendor=Azul Systems, Inc.
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_GB
Framework arguments:  -product org.eclipse.sdk.ide
Command-line arguments:  -product org.eclipse.sdk.ide -data /Users/christopherlamb/Projects/OTMS/tools/eclipse/eclipse4jgit/egit-master/ws/../JGitDebugChildEclipseWorkspace -dev file:/Users/christopherlamb/Projects/OTMS/tools/eclipse/eclipse4jgit/egit-master/ws/.metadata/.plugins/org.eclipse.pde.core/JGit Debug Child Eclipse/dev.properties -os macosx -ws cocoa -arch x86_64 -consoleLog

!ENTRY org.eclipse.equinox.security 4 0 2023-07-16 16:05:47.317
!MESSAGE Secure storage was unable to retrieve the master password from the OS keyring. Make sure that this application has access to the OS keyring. If the error persists, the password recovery feature could be used, or secure storage can be deleted and re-created.
!STACK 0
java.lang.SecurityException: Could not obtain password.  Result: -25300
	at org.eclipse.equinox.internal.security.osx.OSXProvider.getPassword(Native Method)
	at org.eclipse.equinox.internal.security.osx.OSXProvider.getPassword(OSXProvider.java:49)
	at org.eclipse.equinox.internal.security.storage.PasswordProviderModuleExt.getPassword(PasswordProviderModuleExt.java:44)
	at org.eclipse.equinox.internal.security.storage.SecurePreferencesRoot.getModulePassword(SecurePreferencesRoot.java:259)
	at org.eclipse.equinox.internal.security.storage.SecurePreferencesRoot.getPassword(SecurePreferencesRoot.java:220)
	at org.eclipse.equinox.internal.security.storage.SecurePreferences.put(SecurePreferences.java:229)
	at org.eclipse.equinox.internal.security.storage.SecurePreferencesWrapper.put(SecurePreferencesWrapper.java:128)
	at org.eclipse.egit.core.internal.credentials.EGitSecureStore.putCredentials(EGitSecureStore.java:76)
	at org.eclipse.egit.core.internal.EGitSshdSessionFactory$EGitFilePasswordProvider.keyLoaded(EGitSshdSessionFactory.java:294)
	at org.eclipse.jgit.transport.sshd.IdentityPasswordProvider.keyLoaded(IdentityPasswordProvider.java:286)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.SecurityCallback.passwordTried(SecurityCallback.java:113)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.load(Pkcs11Provider.java:218)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.getKeys(Pkcs11Provider.java:295)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.getPkcs11Keys(JGitPublicKeyAuthentication.java:515)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.getAgentIdentities(JGitPublicKeyAuthentication.java:410)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.initializeAgentIdentities(JGitPublicKeyAuthentication.java:348)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKeyIterator.<init>(UserAuthPublicKeyIterator.java:59)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication$KeyIterator.<init>(JGitPublicKeyAuthentication.java:320)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication.createPublicKeyIterator(JGitPublicKeyAuthentication.java:139)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.init(UserAuthPublicKey.java:108)
	at org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication.init(JGitPublicKeyAuthentication.java:125)
	at org.apache.sshd.client.session.ClientUserAuthService.tryNext(ClientUserAuthService.java:410)
	at org.apache.sshd.client.session.ClientUserAuthService.processUserAuth(ClientUserAuthService.java:331)
	at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:267)
	at org.apache.sshd.common.session.helpers.CurrentService.process(CurrentService.java:109)
	at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:592)
	at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:523)
	at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:522)
	at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1649)
	at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:483)
	at org.eclipse.jgit.internal.transport.sshd.JGitClientSession.messageReceived(JGitClientSession.java:208)
	at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:407)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:380)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:375)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
	at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
[sshd-JGitSshClient[5acd91bb]-nio2-thread-7] WARN org.eclipse.jgit.internal.transport.sshd.JGitPublicKeyAuthentication - processAuthDataRequest(JGitClientSession[git@github.ibm.com/169.60.70.162:22])[ssh-connection][publickey] sent algorithm rsa-sha2-512 but got back ssh-rsa from SSH-2.0-babeld-9e792ef
[sshd-JGitSshClient[5acd91bb]-nio2-thread-7] WARN org.eclipse.jgit.internal.transport.sshd.JGitClientSession - exceptionCaught(JGitClientSession[git@github.ibm.com/169.60.70.162:22])[state=Opened] ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR

!ENTRY org.eclipse.egit.core 4 0 2023-07-16 16:05:47.547
!MESSAGE Pulling 1 repository
!SUBENTRY 1 org.eclipse.egit.core 4 0 2023-07-16 16:05:47.547
!MESSAGE git@github.ibm.com:OTMS/w21.7.git: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
!STACK 0
org.eclipse.jgit.api.errors.TransportException: git@github.ibm.com:OTMS/w21.7.git: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:249)
	at org.eclipse.jgit.api.PullCommand.call(PullCommand.java:266)
	at org.eclipse.egit.core.op.PullOperation$PullJob.run(PullOperation.java:256)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.eclipse.jgit.errors.TransportException: git@github.ibm.com:OTMS/w21.7.git: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
	at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:265)
	at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
	at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:279)
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:152)
	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:153)
	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:105)
	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1465)
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:238)
	... 3 more
Caused by: org.apache.sshd.common.SshException: [ssh-connection]: Failed (ProviderException) to execute: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
	at org.apache.sshd.common.future.AbstractSshFuture.lambda$verifyResult$2(AbstractSshFuture.java:146)
	at org.apache.sshd.common.future.AbstractSshFuture.formatExceptionMessage(AbstractSshFuture.java:206)
	at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:145)
	at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:56)
	at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:35)
	at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:74)
	at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:172)
	at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:101)
	at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:258)
	... 11 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_SignFinal(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.engineSign(P11Signature.java:621)
	at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1423)
	at java.base/java.security.Signature.sign(Signature.java:712)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.sign(Pkcs11Provider.java:248)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider$Pkcs11Identity.sign(Pkcs11Provider.java:368)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.appendSignature(UserAuthPublicKey.java:446)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.processAuthDataRequest(UserAuthPublicKey.java:413)
	at org.apache.sshd.client.auth.AbstractUserAuth.process(AbstractUserAuth.java:88)
	at org.apache.sshd.client.session.ClientUserAuthService.processUserAuth(ClientUserAuthService.java:345)
	at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:267)
	at org.apache.sshd.common.session.helpers.CurrentService.process(CurrentService.java:109)
	at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:592)
	at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:523)
	at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:522)
	at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1649)
	at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:483)
	at org.eclipse.jgit.internal.transport.sshd.JGitClientSession.messageReceived(JGitClientSession.java:208)
	at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:407)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:380)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:375)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
	at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860141 is a reply to message #1860139]

Wed, 19 July 2023 09:06

Eclipse User

Regarding PKCS11SlotListIndex: first, if it's zero, you don't need to set it. Second, see the README: if you set PKCS11SlotListIndex, also set IgnoreUnknown PKCS11SlotListIndex. Since this mechanism for making OpenSSH ignore this option exists, I left this non-standard option in.

As for the exception regarding the OS X keyring: that appears to be a problem with Eclipse trying to store the PIN in the Eclipse secure storage. Did you check the checkbox in the dialog? I guess so, because otherwise EGit should not even have tried to do that. Apparently you have OS X keyring integration enabled, and that one fails. This is unrelated to PKCS#11. I did try that, and for me it worked, but I don't think I have this OS X keychain integration enabled. (Can't check right now; I'm not at my development machine.)

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860142 is a reply to message #1860141]

Wed, 19 July 2023 09:12

Eclipse User

This storing passwords in the Eclipse secure store could probably be improved such that is doesn't fail the whole operation. After all, the actual password had worked already. It would perhaps be sufficient if the EGitFilePasswordProvider just logged (via Eclipse OSGi logging) the failure to save that password in the secure store.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860143 is a reply to message #1860142]

Wed, 19 July 2023 09:56

Eclipse User

I just did a quick retest on Windows, and can confirm
a) PKCS11SlotListIndex is not required for my YubiKey
b) If PKCS11SlotListIndex is set, then "IgnoreUnknown PKCS11SlotListIndex" also works.

On macOS I now think the "OS keyring" is a follow on error. The real problem is probably "sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR"
If I disable "OS X Keystore Integration", then I still get the second error stack in my post above.
It may be down to the nss install on my mac https://stackoverflow.com/questions/47230371/sunpkcs11-provider-in-macos-for-firefox

i.e. the real error is here:

Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DEVICE_ERROR
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_SignFinal(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.engineSign(P11Signature.java:621)
	at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1423)
	at java.base/java.security.Signature.sign(Signature.java:712)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider.sign(Pkcs11Provider.java:248)
	at org.eclipse.jgit.internal.transport.sshd.pkcs11.Pkcs11Provider$Pkcs11Identity.sign(Pkcs11Provider.java:368)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.appendSignature(UserAuthPublicKey.java:446)
	at org.apache.sshd.client.auth.pubkey.UserAuthPublicKey.processAuthDataRequest(UserAuthPublicKey.java:413)
	at org.apache.sshd.client.auth.AbstractUserAuth.process(AbstractUserAuth.java:88)
	at org.apache.sshd.client.session.ClientUserAuthService.processUserAuth(ClientUserAuthService.java:345)
	at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:267)
	at org.apache.sshd.common.session.helpers.CurrentService.process(CurrentService.java:109)
	at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:592)
	at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:523)
	at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:522)
	at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1649)
	at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:483)
	at org.eclipse.jgit.internal.transport.sshd.JGitClientSession.messageReceived(JGitClientSession.java:208)
	at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:407)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:380)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:375)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
	at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

It looks like the CKR_DEVICE_ERROR might be coming from Yubico code .... https://github.com/Yubico/yubico-piv-tool/blob/master/ykcs11/ykcs11.c

[Updated on: Wed, 19 July 2023 16:17] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860157 is a reply to message #1860143]

Thu, 20 July 2023 02:56

Eclipse User

Yes, looks like the Eclipse secure storage error is caught and just logged. If that YubiKey library fails in C_SignFinal then perhaps there's a bug in that library? Or maybe the slot index is different? Try to list all slots and keys, and run Eclipse with debug logging on. The Java code should write out all keys it finds. Perhaps the first one isn't the right one?

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860161 is a reply to message #1860157]

Thu, 20 July 2023 05:19

Eclipse User

How do I run Eclipse / JGit with debug logging on? I am sure it should be obvious once one knows how, but somehow it has escaped me....
I did try starting Eclipse from the command line "Eclipse.app/Contents/MacOS/eclipse -debug -consoleLog", but I don't think that is what you meant.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860163 is a reply to message #1860161]

Thu, 20 July 2023 07:38

Eclipse User

Check what logging back-end is installed in your Eclipse. If there isn't one, then include one. Then configure that logging back-end to have debug logging on for org.eclipse.jgit.ssh.apache. How to do this depends on the logging back-end.

Or live debug and step through the Pkcs11Provider.getKeys() method.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860166 is a reply to message #1860163]

Thu, 20 July 2023 09:43

Eclipse User

With a bit of hacking, I have success on macOS.
To see what keys are being loaded, I temporarily added some System.out.println() calls in the Pkcs11Provider.getKeys() method. This gave me:

SunPKCS11-JGit-0-libykcs11.dylibLoaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid'.
SunPKCS11-JGit-0-libykcs11.dylibLoaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid'.

That gave me the Eureka! moment. It is using the wrong key (the "Attestation key" instead of the "Authentication key".
If I do

result.remove(0),

leaving only the Authentication key in the result, then the subsequent Git Pull succeeds!

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860171 is a reply to message #1860166]

Thu, 20 July 2023 11:24

Eclipse User

So now we have the problem of how to select the right key. I'll see what I can do. The idea was that one could export the public key of the wanted certificate, for instance to ~/.ssh/mykey.pub (steps in file manual_tests.txt in the change) and then set, in addition to PKCS11Provider,

IdentityFile ~/.ssh/mykey
IdentitiesOnly yes

but I just realized that I had completely forgotten to test that.

Edit: just tested that, and it seems to work fine.

Interesting that you didn't run into this problem on Windows. Do the libraries return the keys in different order on the two platforms?

[Updated on: Thu, 20 July 2023 12:40] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860178 is a reply to message #1860171]

Thu, 20 July 2023 15:36

Eclipse User

Actually, many moons ago I had a similar issue on Windows, and even have an issue open on the Yubico/yubico-piv-tool Github https://github.com/Yubico/yubico-piv-tool/issues/421.

When I opened that issue I had only one YubiKey. It worked on the Mac, but gave errors on Windows, because on Windows it was presenting the Attestation Key first.

Sometime later I purchased a second YubiKey for my Windows PC. The new key on Windows presented the Authentication key first (and thus gave no error).

i.e. on Windows the two YubiKeys seem to present the keys in different order.

I will now retest on Windows and Mac with both keys...

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860181 is a reply to message #1860178]

Thu, 20 July 2023 17:20

Eclipse User

Can you maybe export the two certificates and check whether they have the KeyUsage extension set? If so, I would expect the Attestation certificate to have signing = false, and the Authentication certificate signing=true. If they set that extension, then we'd have a way in Java to skip such certificates for non-signing keys altogether.

But if the IdentityFile/IdentitiesOnly works, then maybe that's better. OpenSSH explicitly considers non-signing keys because of https://bugzilla.mindrot.org/show_bug.cgi?id=1736 . Don't know how relevant that is.

[Updated on: Thu, 20 July 2023 17:23] by Moderator

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860183 is a reply to message #1860181]

Thu, 20 July 2023 17:48

Eclipse User

I will try exporting the certs tomorrow.

In the meantime I have repeatedly tested / debugged with both YubiKeys on my Mac.
JGit seems to load the keys from both YubiKeys in the same order, but in Pcks11Provider.sign() use different keys. I have not yet worked out why.

"New Windows" YubiKey on Mac (OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid'.
[SshAgentClient.getIdentities] Got SSH Agent Key with comment 'Public key for PIV Authentication'.
[SshAgentClient.getIdentities] Got SSH Agent Key with comment 'Public key for PIV Attestation'.
[JGitPublicKeyAuthentication.initializeAgentIdentities] 1st Key ='X.509 Certificate for PIV Attestation'.
[Pcks11Provider.sign] SunPKCS11-JGit-0-libykcs11.dylib: Signing with PKCS#11 key 'X.509 Certificate for PIV Authentication', algorithm 'SHA512withRSA' (attempt 1).

"Old Mac" YubiKey on Mac (NOT OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid'.
[SshAgentClient.getIdentities] Got SSH Agent Key with comment 'Public key for PIV Authentication'.
[SshAgentClient.getIdentities] Got SSH Agent Key with comment 'Public key for PIV Attestation'.
[JGitPublicKeyAuthentication.initializeAgentIdentities] 1st Key ='X.509 Certificate for PIV Attestation'.
[Pcks11Provider.sign] SunPKCS11-JGit-0-libykcs11.dylib: Signing with PKCS#11 key 'X.509 Certificate for PIV Attestation', algorithm 'SHA512withRSA' (attempt 1).

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860189 is a reply to message #1860183]

Fri, 21 July 2023 04:42

Eclipse User

Below is a dump of the certificate extensions for each of the two certificates on the two YubiKeys.

"New Windows" YubiKey on Mac (OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] [
[
  Version: V3
  Subject: CN=Yubico PIV Attestation
 
Certificate Extensions: 2
[1]: ObjectId: 1.3.6.1.4.1.41482.3.3 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 03 05 04 03                                     .....


[2]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] [
[
  Version: V3
  Subject: CN=SSH key with YubiKey

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 1F B1 16 C8 65 D8 83 5C   2F 00 02 56 2D 01 09 0C  ....e..\/..V-...
0010: 4E 00 39 FF                                        N.9.
]
]

[2]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen: no limit
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 1F B1 16 C8 65 D8 83 5C   2F 00 02 56 2D 01 09 0C  ....e..\/..V-...
0010: 4E 00 39 FF                                        N.9.
]

"Old Mac" YubiKey on Mac (NOT OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] [
[
  Version: V3
  Subject: CN=Yubico PIV Attestation
 
Certificate Extensions: 2
[1]: ObjectId: 1.3.6.1.4.1.41482.3.3 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 03 05 02 07                                     .....


[2]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid'.
[Pcks11Provider.getKeys] [
[
  Version: V3
  Subject: CN=SSH key with YubiKey

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 63 C9 E2 43 60 71 DC B1   CB 1F 59 13 C1 61 40 3B  c..C`q....Y..a@;
0010: 97 FD 05 5A                                        ...Z
]
]

[2]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen: no limit
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 63 C9 E2 43 60 71 DC B1   CB 1F 59 13 C1 61 40 3B  c..C`q....Y..a@;
0010: 97 FD 05 5A                                        ...Z
]

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860190 is a reply to message #1860183]

Fri, 21 July 2023 04:44

Eclipse User

I suggest to switch off SSH agent usage for testing (IdentityAgent none in ~/.ssh/config, or globally in the EGit preferences). Perhaps also write out the public key hash (KeyUtils.getFingerPrint(PublicKey)) in Pkcs11Provider.getKeys.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860191 is a reply to message #1860190]

Fri, 21 July 2023 04:49

Eclipse User

What does certificate.getKeyUsage() give for these certificates?

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860194 is a reply to message #1860191]

Fri, 21 July 2023 07:57

Eclipse User

x509.getKeyUsage() is null for both YubiKeys / Certificates

You can see the 4 publickey fingerprints below:

"New Windows" YubiKey on Mac (OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid, 
Public key fingerprint: 'SHA256:54W0/9rkv84M3pwsFa7qvWkCeQGbkWlwSkvk1fcsrV0'.
[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid, 
Public key fingerprint: 'SHA256:8j8Emi1UAWJ/oZyYZi6gqW3exfTarbbuQ2hQ3yEFh7I'.

"Old Mac" YubiKey on Mac (NOT OK)

[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Attestation', key type 'RSA'. 'Certificate is valid, 
Public key fingerprint: 'SHA256:BmZFU+D2c8opTkyt4zO0JHpUNr5Oh8u3AfbkkhRpOrk'.
[Pcks11Provider.getKeys] SunPKCS11-JGit-0-libykcs11.dylib: Loaded X.509 certificate 'X.509 Certificate for PIV Authentication', key type 'RSA'. 'Certificate is valid, 
Public key fingerprint: 'SHA256:n9Uotcz9uQVJKpSj3h0GnTuVT1fd8g7CCTUWn+1McMA'.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860195 is a reply to message #1860194]

Fri, 21 July 2023 08:17

Eclipse User

So it's indeed different keys (I was a bit worried that maybe something went wrong and we ended up with the same key), and we cannot use Certificate.getKeyUsage() to skip non-signing keys. Nasty. The token does have flags for the keys, but the Java KeyStore does not give us any access to them. So to skip keys that cannot sign we'd actually have to try to sign something and only consider certificates that give a private key that we can successfully sign with. That'd be most unfortunate.

I don't get why it's using different keys for signing if it loaded them both in the same order in both cases. If it loads the attestation key before the authentication key, it should try the attestation key first in both cases, and fail in both cases.

Did you try the IdentityFile/IdentitiesOnly thing? If you did so for the "New Windows" YubiKey, that might explain the behavior (having the attestation key first, but still trying only the authentication key), and if you did not do so for the "Old Mac" key, then that would also explain the difference.

Other than that I have no clue why it would use different keys in the two cases. It's all just Iterables/Iterators, no HashMaps or anything that might make the order indeterminate involved.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860199 is a reply to message #1860195]

Fri, 21 July 2023 13:05

Eclipse User

I am gradually circling in on the cause of the problem:

When using my Mac YubiKey:
UserAuthPublicKey.resolveAttemptedPublicKeyIdentity() is called once before via Pkcs11Provider.sign() is called.
i.e. when Pkcs11Provider.sign() is called, the Attestation key is used and fails.

When using with my Windows YubiKey:
UserAuthPublicKey.resolveAttemptedPublicKeyIdentity() is called twice before Pkcs11Provider.sign() is called.

i.e. when Pkcs11Provider.sign() is called, the Attestation key has been "skipped", and the Authentication key is used and succeeds.

My next step is to find out why / where the Attestation key is "skipped" with the Windows YubiKey.

So far I have not tried the IdentityFile/IdentitiesOnly thing, but I plan to soon.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860200 is a reply to message #1860199]

Fri, 21 July 2023 15:18

Eclipse User

Check the key types of the two attestation keys. (Result of KeyUtils.getKeyType(PublicKey).) Apache MINA sshd skips a key if it cannot determine a signature algorithm for it.

A breakpoint at UserAuthPublicKey, line 154, and then stepping through that loop, should show soon enough what's going on.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860206 is a reply to message #1860200]

Sat, 22 July 2023 09:08

Eclipse User

Good news: I can confirm that the IdentityFile/IdentitiesOnly configuration works!
On my mac, .ssh/config now looks like this:

Host github.acme.com
  PKCS11Provider /usr/local/lib/libykcs11.dylib
  Port 22
  User git
  #LogLevel DEBUG3
  #ConnectTimeout 600
  #IdentityAgent /private/tmp/com.apple.launchd.RT4X3BfB3J/Listeners
  IdentityFile ~/.ssh/yubiKey5cNanoAuthentication
  IdentitiesOnly true

I took me a while to "click" that the entry IdentityFile should be IdentityFile ~/.ssh/yubiKey5cNanoAuthentication, and NOT IdentityFile ~/.ssh/yubiKey5cNanoAuthentication.pub

Before trying the IdentityFile config I spent some hours debugging with both YubiKey.s on my Mac So far I have established
The keys on both YubiKeys are loaded in the same order.

With the New/Windows YubiKey, JGit starts with the Attestation Key, then at some point it drops that key, and moves to the Authentication Key, and only attempts to sign with the Authentication Key.

So far I could not work out why the first key was dropped (I was fighting against timeouts), but it was deep in the Apache mina ssh message handling code, which implies some ssh communication was taking place.

With the Old/Mac YubiKey, JGit also starts with the Attestation Key, and keeps using that key up to the point of signing, where it errors out.

I have tried putting breakpoints in UserAuthPublicKey, but nothing stops. For the class on my Mac line 154 is a commented out System.err.println() call.

I will have another go at debugging this evening when it gets dark.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860208 is a reply to message #1860206]

Sat, 22 July 2023 11:35

Eclipse User

OK... SSH public key authentication with Apache MINA sshd sends two messages:

Cient sends to Server: "Want to authenticate with this public key: <public key>".
Server replies whether it would accept that key, provided a correct signature is presented.
Client sends to Server: "Want to authenticate with this public key: <public key>, and BTW, here's the signature: <signature>".
Server checks key and signature and replies OK or NOK.

So it appears that for the attestation key of the "Old/Mac" key, the server somehow replies in step 2 "Yes, I know that key, go ahead". But for the attestation key in the "New/Windows" token, the server replies "No, I don't know that key, so I won't accept it for log-in". At that point the client doesn't even try steps 3 and 4 but goes on with step 1 with the next key in the list, which would then be the correct authentication key.

This was already mentioned at https://github.com/Yubico/yubico-piv-tool/issues/421 , but in less detail. Somehow your git server has that Old/Mac attestation key.

Glad that the IdentityFile/IdentitiesOnly works for you. I hope it all also works fine on the command line using OpenSSH.

Re: breakpoint: Apache MINA sshd has two classes named UserAuthPublicKey, one for client part and one for the server part. I meant the client implementation. But the idea about it not being able to figure out a signature algorithm is unlikely anyway. Much more likely is the explanation above.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860209 is a reply to message #1860208]

Sat, 22 July 2023 12:49

Eclipse User

... and I tried the 3rd instance of the class UserAuthPublicKey: the one in the package "com.jcraft.jsch", because by chance I happened to have the class UserAuthPassword already open.

As to Attestation Keys, I have been formulating a theory myself. The "Old/Mac" YubiKey is a key issued by my employer, the "New/Windows" YubiKey privately purchased. My theory is that the Attestation keys on the employer issued YubiKeys may be the same, and that "somebody" registered the Attestation Key with GitHub instead of or in addition to the Authentication Key. This ties in nicely with your explanation above. i.e. GitHub recognises the Employer issued YubiKey Attestation Key, but not the Attestation Key on my privately purchased YubiKe

Git Pull on the Mac Command line works fine with the "IdentityFile/IdentitiesOnly"

The only downside of the "IdentityFile/IdentitiesOnly" config is that if I switch between multiple YubiKeys, I will need to remember to change .ssh/config. But using multiple YubiKeys is an edge case. In practice I have a dedicated YubiKey per laptop: I have only been switching the keys from laptop to laptop for debugging purposes.

Thanks for all your support and explanations.

One final thought: If I understand correctly your excellent change is always active if there is a PKCS11Provider entry in the .ssh/config file. I have a nagging doubt at the back of my head: Could this be a breaking change for other users? Would it be safer if PKCS11Provider support was configurable by a checkbox in the Eclipse git ssh preferences?

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860210 is a reply to message #1860208]

Sat, 22 July 2023 13:01

Eclipse User

Here's the New & Noteworthy entry for this. Feel free to improve.

Re: Unexpected reply from ssh-agent: SSH_AGENT_FAILURE [message #1860211 is a reply to message #1860209]

Sat, 22 July 2023 13:14

Eclipse User

Christopher Lamb wrote on Sat, 22 July 2023 16:49

The only downside of the "IdentityFile/IdentitiesOnly" config is that if I switch between multiple YubiKeys, I will need to remember to change .ssh/config.

Export both public keys from both devices, and use two IdentityFile options. Like

Host git.acme.com
PKCS11Provider /path/to/library
IdentityFile ~/.ssh/key_from_device_1
IdentityFile ~/.ssh/key_from_device_2
IdentitiesOnly yes

The only prerequisite is that the library should be able to work with both devices.
Christopher Lamb wrote on Sat, 22 July 2023 16:49

.One final thought: If I understand correctly your excellent change is always active if there is a PKCS11Provider entry in the .ssh/config file. I have a nagging doubt at the back of my head: Could this be a breaking change for other users? Would it be safer if PKCS11Provider support was configurable by a checkbox in the Eclipse git ssh preferences?

Assuming that someone already has PKCS11Provider in his SSH config for a host entry that will be used to access a git repository, I think it is safe to assume that the user is happy if that key is used to access the git repository. Command-line git would have used the PKCS#11 keys already, and for JGit the user would, until now, have had to either add that key to the SSH agent, like you did initially, or use a separate "normal" key and have that configured additionally. If EGit now suddenly also can use that PKCS#11 key, that should not cause any trouble.

Previous Topic:	Create a main branch, not master
Next Topic:	Confused about repository with git submodule

Goto Forum:

-=] Back to Top [=-

Current Time: Sat May 24 20:03:41 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter