Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit / JGit » Eclipse JGit(Different SHA1Id generated for Windows and Linux file due to EOL difference )
Eclipse JGit [message #1799391] Thu, 06 December 2018 06:26 Go to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
I am using JGit jar to identify if file is updated or not depending upon SHA1Id generation. Observed, if file contents are same but file is generated on different OS (here windows and Linux) due to EOL difference SHA1Id is different.
I would like to know, is there any way to ignore the EOL while computing the SHA1Id in JGit jar pro-grammatically.
Jar version is : org.eclipse.jgit-5.1.3.201810200350-r
Re: Eclipse JGit [message #1799400 is a reply to message #1799391] Thu, 06 December 2018 08:53 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 264
Registered: July 2009
Senior Member
How about setting core.autocrlf to true and asking with JGIt Status command whether a file has changed?

Ciao
Chris
Re: Eclipse JGit [message #1799406 is a reply to message #1799400] Thu, 06 December 2018 09:22 Go to previous messageGo to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
Hi Christian,
Thank you for reply. I am using JGit jar to verify if customer has done any customization in provided code. Hence, is there any way through programming.
Re: Eclipse JGit [message #1799408 is a reply to message #1799406] Thu, 06 December 2018 10:40 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 264
Registered: July 2009
Senior Member
To compute a sha1 of a local file in a way you can compare it to sha1 which you find in git commit/blobs you would have to follow:
https://stackoverflow.com/a/24283352/3967955

The code in JGit which does this can be looked up here: https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/v5.1.3.201810200350-r/org.eclipse.jgit/src/org/eclipse/jgit/treewalk/WorkingTreeIterator.java#1176

If you want to compute a SHA1 which has autoCRLF semantics then you should feed to the method above not a FileInputStream but JGits AutoCRLFInputStream

like
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

import org.eclipse.jgit.api.errors.GitAPIException;
import org.eclipse.jgit.api.errors.JGitInternalException;
import org.eclipse.jgit.lib.Constants;
import org.eclipse.jgit.lib.ObjectId;
import org.eclipse.jgit.util.sha1.SHA1;

public class CalcSHA1 {

	public static void main(String args[]) throws IOException, GitAPIException, JGitInternalException {
		try (InputStream in = new FileInputStream(args[0])) {
			System.out.println(
					"Git hash for file " + args[0] + " is " + ObjectId.fromRaw(computeHash(in, 1172)).getName());
		}
	}

	private static final byte[] digits = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' };
	protected static final byte[] zeroid = new byte[Constants.OBJECT_ID_LENGTH];
	private static final byte[] hblob = Constants.encodedTypeString(Constants.OBJ_BLOB);

	private static byte[] computeHash(InputStream in, long length) throws IOException {
		SHA1 contentDigest = SHA1.newInstance();
		final byte[] contentReadBuffer = new byte[4096];
		contentDigest.update(hblob);
		contentDigest.update((byte) ' ');
		long sz = length;
		if (sz == 0) {
			contentDigest.update((byte) '0');
		} else {
			final int bufn = contentReadBuffer.length;
			int p = bufn;
			do {
				contentReadBuffer[--p] = digits[(int) (sz % 10)];
				sz /= 10;
			} while (sz > 0);
			contentDigest.update(contentReadBuffer, p, bufn - p);
		}
		contentDigest.update((byte) 0);
		for (;;) {
			final int r = in.read(contentReadBuffer);
			if (r <= 0)
				break;
			contentDigest.update(contentReadBuffer, 0, r);
			sz += r;
		}
		if (sz != length)
			return zeroid;
		return contentDigest.digest();
	}
}





Ciao
Chris
Re: Eclipse JGit [message #1799409 is a reply to message #1799408] Thu, 06 December 2018 10:46 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 264
Registered: July 2009
Senior Member
sorry, example code was not cleaned. See here

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

import org.eclipse.jgit.api.errors.GitAPIException;
import org.eclipse.jgit.api.errors.JGitInternalException;
import org.eclipse.jgit.lib.Constants;
import org.eclipse.jgit.lib.ObjectId;
import org.eclipse.jgit.util.sha1.SHA1;

public class CalcSHA1 {

	public static void main(String args[]) throws IOException, GitAPIException, JGitInternalException {
		File file = new File(args[0]);
		file.length();
		
		try (InputStream in = new FileInputStream(file)) {
			System.out.println(
					"Git hash for file " + args[0] + " is " + ObjectId.fromRaw(computeHash(in, file.length())).getName());
		}
	}

	private static final byte[] digits = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' };
	protected static final byte[] zeroid = new byte[Constants.OBJECT_ID_LENGTH];
	private static final byte[] hblob = Constants.encodedTypeString(Constants.OBJ_BLOB);

	private static byte[] computeHash(InputStream in, long length) throws IOException {
		SHA1 contentDigest = SHA1.newInstance();
		final byte[] contentReadBuffer = new byte[4096];
		contentDigest.update(hblob);
		contentDigest.update((byte) ' ');
		long sz = length;
		if (sz == 0) {
			contentDigest.update((byte) '0');
		} else {
			final int bufn = contentReadBuffer.length;
			int p = bufn;
			do {
				contentReadBuffer[--p] = digits[(int) (sz % 10)];
				sz /= 10;
			} while (sz > 0);
			contentDigest.update(contentReadBuffer, p, bufn - p);
		}
		contentDigest.update((byte) 0);
		for (;;) {
			final int r = in.read(contentReadBuffer);
			if (r <= 0)
				break;
			contentDigest.update(contentReadBuffer, 0, r);
			sz += r;
		}
		if (sz != length)
			return zeroid;
		return contentDigest.digest();
	}
}


Ciao
Chris
Re: Eclipse JGit [message #1799415 is a reply to message #1799409] Thu, 06 December 2018 11:43 Go to previous messageGo to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
Hi Christian ,
I have used above code with AutoCRLFInputStream and w/o AutoCRLFInputStream but, not getting expected result. I attaching some dummy files to reproduce the issue.
With your code, the SHA1Ids are:
linux.txt SHA1Id "7594b8c63dc99a37e9b27e6f27e405701413d09f"
windows.txt SHA1Id "0be486bd774c11a47e2476daaaf082a91d7877c4"

I am expecting, both files having same content but EOL is different (Windows : CRLF and Linux: LF) then SHA1Id should be same in such cases.
  • Attachment: windows.txt
    (Size: 0.02KB, Downloaded 39 times)
  • Attachment: linux.txt
    (Size: 0.02KB, Downloaded 36 times)
Re: Eclipse JGit [message #1799525 is a reply to message #1799415] Sat, 08 December 2018 12:25 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
Yes, it's not that easy because of the length. You can't just take file.length() here but have to read the file a first time to determine its length after the EOL translation has been done, and then pass that length into computeHash().
Re: Eclipse JGit [message #1799563 is a reply to message #1799525] Mon, 10 December 2018 10:58 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 264
Registered: July 2009
Senior Member
I added crlf conversion to the java example and added a script which shows which sha1's to expect (should be 7594b8c63dc99a37e9b27e6f27e405701413d09f for both files):

> git init
Initialized empty Git repository in /tmp/init/.git/
> ls
linux.txt	windows.txt
> git config core.autocrlf input
> git add linux.txt windows.txt
warning: CRLF will be replaced by LF in windows.txt.
The file will have its original line endings in your working directory
> git ls-files -sv
H 100644 7594b8c63dc99a37e9b27e6f27e405701413d09f 0	linux.txt
H 100644 7594b8c63dc99a37e9b27e6f27e405701413d09f 0	windows.txt
> git rm --cached linux.txt windows.txt
rm 'linux.txt'
rm 'windows.txt'
> git config core.autocrlf false
> git add .
> git ls-files -sv
H 100644 7594b8c63dc99a37e9b27e6f27e405701413d09f 0	linux.txt
H 100644 0be486bd774c11a47e2476daaaf082a91d7877c4 0	windows.txt

Java Code:

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

import org.eclipse.jgit.api.errors.GitAPIException;
import org.eclipse.jgit.api.errors.JGitInternalException;
import org.eclipse.jgit.lib.Constants;
import org.eclipse.jgit.lib.ObjectId;
import org.eclipse.jgit.util.io.AutoLFInputStream;
import org.eclipse.jgit.util.sha1.SHA1;

public class CalcSHA1 {
	static byte[] buffer = new byte[4096];
	private static final byte[] digits = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' };
	protected static final byte[] zeroid = new byte[Constants.OBJECT_ID_LENGTH];
	private static final byte[] hblob = Constants.encodedTypeString(Constants.OBJ_BLOB);

	public static void main(String args[]) throws IOException, GitAPIException, JGitInternalException {
		for (String a : args) {
			File file = new File(a);
			long l = 0, length = 0;
			try (InputStream in = new AutoLFInputStream(new FileInputStream(file), true)) {
				while ((l = in.read(buffer)) != -1) {
					length += l;
				}
			}
			try (InputStream in = new AutoLFInputStream(new FileInputStream(file), true)) {
				System.out.println(
						"Git hash for file " + a + " with length "+length+ " is " + ObjectId.fromRaw(computeHash(in, length)).getName());
			}
		}
	}

	private static byte[] computeHash(InputStream in, long length) throws IOException {
		SHA1 contentDigest = SHA1.newInstance();

		contentDigest.update(hblob);
		contentDigest.update((byte) ' ');
		long sz = length;
		if (sz == 0) {
			contentDigest.update((byte) '0');
		} else {
			final int bufn = buffer.length;
			int p = bufn;
			do {
				buffer[--p] = digits[(int) (sz % 10)];
				sz /= 10;
			} while (sz > 0);
			contentDigest.update(buffer, p, bufn - p);
		}
		contentDigest.update((byte) 0);
		for (;;) {
			final int r = in.read(buffer);
			if (r <= 0)
				break;
			contentDigest.update(buffer, 0, r);
			sz += r;
		}
		if (sz != length)
			return zeroid;
		return contentDigest.digest();
	}
}


Ciao
Chris
Re: Eclipse JGit [message #1799652 is a reply to message #1799563] Wed, 12 December 2018 05:18 Go to previous messageGo to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
Thank you,
Now, SHA1 Id is same for both the files. I am verifying the new implementation and let you know, if any thing went wrong.

Thank you very much!!!
Re: Eclipse JGit [message #1802687 is a reply to message #1799652] Thu, 14 February 2019 05:12 Go to previous messageGo to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
Hi Christian,
provided solution is working fine with java 1.8. we are trying on java 1.6, but its not working due to major minor version. So, i used jgit 3.3.2 version but getting compilation error for AutoLFInputStream and SHA1 and other stuff. Do we have any alternate way to resolve this or can you backport the jgit jar for java1.6.

[Updated on: Thu, 14 February 2019 10:51]

Report message to a moderator

Re: Eclipse JGit [message #1802699 is a reply to message #1802687] Thu, 14 February 2019 09:18 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 264
Registered: July 2009
Senior Member
No, I think java 6 is too old. Check https://en.wikipedia.org/wiki/Java_version_history to find "Java 6 reached the end of its supported life in February 2013". I would switch to java 8

Ciao
Chris
Re: Eclipse JGit [message #1802707 is a reply to message #1802699] Thu, 14 February 2019 10:51 Go to previous messageGo to next message
Tester Tester is currently offline Tester TesterFriend
Messages: 12
Registered: December 2018
Junior Member
Hi Christian,
Thank you confirming. What about the java 7, i have tried using 7 but same error.
Can you please help me.
Re: Eclipse JGit [message #1802968 is a reply to message #1802707] Tue, 19 February 2019 20:48 Go to previous message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
Hey, we have 2019 and the current Java version is Java 11 and Java 12 is about to be shipped next month.
Why should a JGit committer waste time to backport the code he wrote for you to Java 7 which is out of maintenance as well?
Maybe that's something you could try on your own ?
Previous Topic:Error when try to use ftp protocol remote
Next Topic:JGit clone call fails because it does not hold the lock
Goto Forum:
  


Current Time: Sat Dec 14 04:32:53 GMT 2019

Powered by FUDForum. Page generated in 0.03625 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top