Eclipse Community Forums: Hudson

Home » Archived » Hudson » .nfs file lock(.nfs file lock issue )

Wed, 24 September 2014 08:34

Eclipse User

Hi All,

I have faced a .nfs file lock issue in Hudson

Details:

I was running one of my hudson job of its 12th build. SVN check out has failed with below error .

svn: E175002: SSL peer shut down incorrectly
svn: E175002: REPORT request failed on '/svn/Wallet-Client-2-0-Android/!svn/vcc/default'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:106)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:90)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:662)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:286)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:274)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:696)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:328)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1287)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:835)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:216)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:100)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:756)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1221)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:292)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:781)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:99)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:408)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:75)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1742)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanContent(XMLEntityScanner.java:916)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanContent(XMLDocumentFragmentScannerImpl.java:1591)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2902)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:742)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:218)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:368)

Then I tried to wipe out the workspace , but it was saying that unabel to delete because of below error.

java.io.IOException: Unable to delete Build_job_name_path/workspace/.svn/.nfs00000000012cfbb90000545c
at hudson.Util.deleteFile(Util.java:263)
at hudson.Util.deleteRecursive(Util.java:305)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.Util.deleteRecursive(Util.java:304)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.cleanupBeforeCheckout(CheckoutUpdater.java:137)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:73)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)

Then I had to restart to get rid of this .nfs file lock.

But even after restart Hudson, the issue was persisting. For the temporary solution, we had to create a new job with the same configuration and run that job and it ran successfully.Please let me know why this behaviour and how to resolve this issue permanently.

kindly let me know if you need more information about this issue.

Re: .nfs file lock [message #1430490 is a reply to message #1430486]

Wed, 24 September 2014 08:35

Eclipse User

We've had similar results with Hudson on NFS here at Eclipse. We've reverted to using local disks for the workspace (as the workspace is generally disposable) and using NFS to store build artifacts.

Re: .nfs file lock [message #1430494 is a reply to message #1430490]

Wed, 24 September 2014 08:45

Eclipse User

Okay. But in my case, I cannot make it local Sad

Re: .nfs file lock [message #1430506 is a reply to message #1430494]

Wed, 24 September 2014 08:55

Eclipse User

Nimi Sukumaran wrote on Wed, 24 September 2014 08:45

Okay. But in my case, I cannot make it local

If you have a large NFS SAN, you can also use Linux tricks to create "local" storage, such as creating an image file, formatting it and mounting it locally using the loopback device.

# create 200G file
hudsonserver:/path/to/nfsmount$ dd of=hudson-disk.img bs=1M seek=200000 count=0
hudsonserver:/path/to/nfsmount$ mkfs.ext4 hudson-disk.img
hudsonserver:/path/to/nfsmount$ mount -o loop hudson-disk.img /path/to/local/mount

It's sloppy, but it works.

Re: .nfs file lock [message #1430508 is a reply to message #1430506]

Wed, 24 September 2014 08:59

Eclipse User

No , here I need to know why this .nfs file is getting locked.

I would like to know the root cause of this issue first. is this due to any memory or disk space issue?

Re: .nfs file lock [message #1430717 is a reply to message #1430508]

Wed, 24 September 2014 14:11

Eclipse User

As Denis pointed out we don't recommend NFS mounted Hudson home. I have limited NFS knowledge, but if my understanding is correct, the .nfsxxxx file is created when one or more process have a file handle open on a file. The tricky part is, unless you have some tool to find out who is holding the file handle, it is very difficult to say why that .nfsxxxx file exists. Unless the file handle is released, the .nfsxxxx won't go away. You may have to search all machines that mounted that partition and figure out who is holding the file handle and release it.

Re: .nfs file lock [message #1431334 is a reply to message #1430717]

Thu, 25 September 2014 10:43

Eclipse User

Thanks Prakash for the reply ...

Previous Topic:	Multi configuration project for two different builds
Next Topic:	3.2.1?

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 16:44:10 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter