Eclipse Community Forums: Hudson

Help

Home

Home » Archived » Hudson » .nfs file lock(.nfs file lock issue )

Show: Today's Messages :: Show Polls :: Message Navigator

.nfs file lock [message #1430486]

Wed, 24 September 2014 12:34

Nimi Sukumaran

Messages: 35
Registered: August 2013
Location: Bangalore

Member

Hi All,

I have faced a .nfs file lock issue in Hudson

Details:

I was running one of my hudson job of its 12th build. SVN check out has failed with below error .

svn: E175002: SSL peer shut down incorrectly
svn: E175002: REPORT request failed on '/svn/Wallet-Client-2-0-Android/!svn/vcc/default'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:106)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:90)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:662)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:286)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:274)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:696)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:328)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1287)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:835)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:216)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:100)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:756)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1221)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:292)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:781)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:99)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:408)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:75)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1742)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanContent(XMLEntityScanner.java:916)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanContent(XMLDocumentFragmentScannerImpl.java:1591)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2902)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:742)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:218)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:368)

Then I tried to wipe out the workspace , but it was saying that unabel to delete because of below error.

java.io.IOException: Unable to delete Build_job_name_path/workspace/.svn/.nfs00000000012cfbb90000545c
at hudson.Util.deleteFile(Util.java:263)
at hudson.Util.deleteRecursive(Util.java:305)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.Util.deleteRecursive(Util.java:304)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.cleanupBeforeCheckout(CheckoutUpdater.java:137)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:73)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)

Then I had to restart to get rid of this .nfs file lock.

But even after restart Hudson, the issue was persisting. For the temporary solution, we had to create a new job with the same configuration and run that job and it ran successfully.Please let me know why this behaviour and how to resolve this issue permanently.

kindly let me know if you need more information about this issue.

Report message to a moderator

Re: .nfs file lock [message #1430490 is a reply to message #1430486]

Wed, 24 September 2014 12:35

Denis Roy

Messages: 483
Registered: October 2004
Location: Ottawa, Ontario, Canada

Senior Member

We've had similar results with Hudson on NFS here at Eclipse. We've reverted to using local disks for the workspace (as the workspace is generally disposable) and using NFS to store build artifacts.

Denis Roy
Eclipse Webmaster -- webmaster@eclipse.org

Report message to a moderator

Re: .nfs file lock [message #1430494 is a reply to message #1430490]

Wed, 24 September 2014 12:45

Nimi Sukumaran

Messages: 35
Registered: August 2013
Location: Bangalore

Member

Okay. But in my case, I cannot make it local Sad

Report message to a moderator

Re: .nfs file lock [message #1430506 is a reply to message #1430494]

Wed, 24 September 2014 12:55

Denis Roy

Messages: 483
Registered: October 2004
Location: Ottawa, Ontario, Canada

Senior Member

Nimi Sukumaran wrote on Wed, 24 September 2014 08:45

Okay. But in my case, I cannot make it local

If you have a large NFS SAN, you can also use Linux tricks to create "local" storage, such as creating an image file, formatting it and mounting it locally using the loopback device.

# create 200G file
hudsonserver:/path/to/nfsmount$ dd of=hudson-disk.img bs=1M seek=200000 count=0
hudsonserver:/path/to/nfsmount$ mkfs.ext4 hudson-disk.img
hudsonserver:/path/to/nfsmount$ mount -o loop hudson-disk.img /path/to/local/mount

It's sloppy, but it works.

Denis Roy
Eclipse Webmaster -- webmaster@eclipse.org

Report message to a moderator

Re: .nfs file lock [message #1430508 is a reply to message #1430506]

Wed, 24 September 2014 12:59

Nimi Sukumaran

Messages: 35
Registered: August 2013
Location: Bangalore

Member

No , here I need to know why this .nfs file is getting locked.

I would like to know the root cause of this issue first. is this due to any memory or disk space issue?

Report message to a moderator

Re: .nfs file lock [message #1430717 is a reply to message #1430508]

Wed, 24 September 2014 18:11

Winston Prakash

Messages: 534
Registered: August 2011
Location: Fremont, CA USA

Senior Member

As Denis pointed out we don't recommend NFS mounted Hudson home. I have limited NFS knowledge, but if my understanding is correct, the .nfsxxxx file is created when one or more process have a file handle open on a file. The tricky part is, unless you have some tool to find out who is holding the file handle, it is very difficult to say why that .nfsxxxx file exists. Unless the file handle is released, the .nfsxxxx won't go away. You may have to search all machines that mounted that partition and figure out who is holding the file handle and release it.

Winston Prakash
Eclipse Hudson team

Report message to a moderator