Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Hudson » .nfs file lock(.nfs file lock issue )
.nfs file lock [message #1430486] Wed, 24 September 2014 12:34 Go to next message
Nimi SukumaranFriend
Messages: 35
Registered: August 2013
Location: Bangalore
Member
Hi All,

I have faced a .nfs file lock issue in Hudson

Details:

I was running one of my hudson job of its 12th build. SVN check out has failed with below error .

svn: E175002: SSL peer shut down incorrectly
svn: E175002: REPORT request failed on '/svn/Wallet-Client-2-0-Android/!svn/vcc/default'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:106)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:90)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:662)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:286)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:274)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:696)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:328)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1287)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:835)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:216)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:100)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:756)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1221)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:292)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:781)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:99)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:408)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:75)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1742)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanContent(XMLEntityScanner.java:916)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanContent(XMLDocumentFragmentScannerImpl.java:1591)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2902)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:742)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:218)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:368)



Then I tried to wipe out the workspace , but it was saying that unabel to delete because of below error.

java.io.IOException: Unable to delete Build_job_name_path/workspace/.svn/.nfs00000000012cfbb90000545c
at hudson.Util.deleteFile(Util.java:263)
at hudson.Util.deleteRecursive(Util.java:305)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.Util.deleteRecursive(Util.java:304)
at hudson.Util.deleteContentsRecursive(Util.java:224)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.cleanupBeforeCheckout(CheckoutUpdater.java:137)
at hudson.scm.subversion.CheckoutUpdater$UpdateTaskImpl.perform(CheckoutUpdater.java:73)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:152)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:807)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:790)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:771)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:763)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:706)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)

Then I had to restart to get rid of this .nfs file lock.

But even after restart Hudson, the issue was persisting. For the temporary solution, we had to create a new job with the same configuration and run that job and it ran successfully.Please let me know why this behaviour and how to resolve this issue permanently.

kindly let me know if you need more information about this issue.
Re: .nfs file lock [message #1430490 is a reply to message #1430486] Wed, 24 September 2014 12:35 Go to previous messageGo to next message
Denis Roy is currently offline Denis RoyFriend
Messages: 451
Registered: October 2004
Location: Ottawa, Ontario, Canada
Senior Member

We've had similar results with Hudson on NFS here at Eclipse. We've reverted to using local disks for the workspace (as the workspace is generally disposable) and using NFS to store build artifacts.

Denis Roy
Eclipse Webmaster -- webmaster@eclipse.org
Re: .nfs file lock [message #1430494 is a reply to message #1430490] Wed, 24 September 2014 12:45 Go to previous messageGo to next message
Nimi SukumaranFriend
Messages: 35
Registered: August 2013
Location: Bangalore
Member
Okay. But in my case, I cannot make it local Sad
Re: .nfs file lock [message #1430506 is a reply to message #1430494] Wed, 24 September 2014 12:55 Go to previous messageGo to next message
Denis Roy is currently offline Denis RoyFriend
Messages: 451
Registered: October 2004
Location: Ottawa, Ontario, Canada
Senior Member

Nimi Sukumaran wrote on Wed, 24 September 2014 08:45
Okay. But in my case, I cannot make it local Sad


If you have a large NFS SAN, you can also use Linux tricks to create "local" storage, such as creating an image file, formatting it and mounting it locally using the loopback device.

# create 200G file
hudsonserver:/path/to/nfsmount$ dd of=hudson-disk.img bs=1M seek=200000 count=0
hudsonserver:/path/to/nfsmount$ mkfs.ext4 hudson-disk.img
hudsonserver:/path/to/nfsmount$ mount -o loop hudson-disk.img /path/to/local/mount

It's sloppy, but it works.


Denis Roy
Eclipse Webmaster -- webmaster@eclipse.org
Re: .nfs file lock [message #1430508 is a reply to message #1430506] Wed, 24 September 2014 12:59 Go to previous messageGo to next message
Nimi SukumaranFriend
Messages: 35
Registered: August 2013
Location: Bangalore
Member
No , here I need to know why this .nfs file is getting locked.

I would like to know the root cause of this issue first. is this due to any memory or disk space issue?
Re: .nfs file lock [message #1430717 is a reply to message #1430508] Wed, 24 September 2014 18:11 Go to previous messageGo to next message
Winston Prakash is currently offline Winston PrakashFriend
Messages: 534
Registered: August 2011
Location: Fremont, CA USA
Senior Member
As Denis pointed out we don't recommend NFS mounted Hudson home. I have limited NFS knowledge, but if my understanding is correct, the .nfsxxxx file is created when one or more process have a file handle open on a file. The tricky part is, unless you have some tool to find out who is holding the file handle, it is very difficult to say why that .nfsxxxx file exists. Unless the file handle is released, the .nfsxxxx won't go away. You may have to search all machines that mounted that partition and figure out who is holding the file handle and release it.

Winston Prakash
Eclipse Hudson team
Re: .nfs file lock [message #1431334 is a reply to message #1430717] Thu, 25 September 2014 14:43 Go to previous message
Nimi SukumaranFriend
Messages: 35
Registered: August 2013
Location: Bangalore
Member
Thanks Prakash for the reply ...
Previous Topic:Multi configuration project for two different builds
Next Topic:3.2.1?
Goto Forum:
  


Current Time: Wed Nov 14 22:06:10 GMT 2018

Powered by FUDForum. Page generated in 0.01833 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top