Re: [geclipse-dev] Transferring local input files during job submission

[Harald Kornmayer <harald.kornmayer@xxxxxxxxxxxx>]

Hi all

For me not clear yet is the time/moment when the local file will be
uploaded to the WMS (or Grid??) ? 

Does the WMS upload the localfile immediately after the jobsubmission? 
(when specified as a part of the input sandbox). 

Or does the upload start after the execution of the job happens? 

Or is the problem the way, that the user specifies the file to be local,
but the WN does not have a mean to access the local file during
execution ("because I switched my laptop of")? 

One solution might be a check if the "local file" are really
"transferred" as a part of the InputSandBox? 

Then the question pops up, how the output will be brought back to the
local disk?? 

May be we discuss this afternoon?? 

Harald 



 
Did I understand the WMS System right, when I conclude the following: 

1. User defines a JSDL file. (including the usage of local files). 
2. User submits that JSDL to the WMS. 
3. The WMS looks for resources, but did  

On Mon, 2007-11-12 at 17:50 +0100, Ariel Garcia wrote:
> Hi Mariusz,
> 
> > I have to make changes in JDL, because original JDL refer to files on
> > user's local machine, which is not accessible from worker node.
> 
> of course the local machine cannot (and shouldn't, we don't want to have a 
> gridftp server on our geclipse!!) be accessed from RB/CE/WN
> 
> but that is why you write:
>    InputSandbox = { <SOME-FILES> };
> in your JDL, those files are uploaded from the local machine (or possibly 
> also SE servers) to the WMS/RB. From there the CE fetches the files, and 
> the WN fetches them from the CE (a gLite "optimization" ;-)
> 
> If you wouldn't specify 
>    InputSandboxDestFileName
> at all, all those files would keep their original name but land in the 
> working-dir of the job in the WN
> 
> Actually a JDL with 
>   Executable      = "/bin/cat";
>   Arguments       = "input.txt";
>   InputSandbox = { "file:///tmp/input.txt" };
> works without any problems even if "file:///tmp/input.txt" is on the local 
> machine (using the cli, for the API it should work too).
> 
> On the other hand, the file renaming when uploading using 
> InputSandboxDestFileName  doesn't seem to work at all for me from the 
> command line (my UI in a glite 3.0 one, perhaps it would work in 3.1?)
> the inputsandbox file is uploaded with the original name...
> 
> > Submission will look like:
> > 1. Remove from JDL (in g-Eclipse memory) all references to local files
> > 2. Register job on WMS
> > 3. Send local files to sandbox, into location returned during job
> > registration
> > 4. Start job on WMS
> 
> Ok, i am trying to argue that  point 1. shouldn't be required
> 
> > We cannot change JSDL->JDL transformation file (because of license), so
> 
> that is fine
> 
> > I have to transform JSDL to JDL (in g-Elipse memory), change JDL (still
> > in g-Eclipse memory) and submit it.
> > I'm not sure if after sending local files to WN sandbox, should I refer
> > to those files in submitted JDL?
> > I guess not, because those files are in right place (sandbox).
> 
> i think you can leave in the JDL all entries which are not relevant to the 
> current submission step, they shouldn't matter!
> 
> Are you using the WMProxy API provided by glite, right?
> so what you mean (or what i'd expect) is the following method calls:
> 
> myJobID = jobRegister( myJDL, delegationId )
> getTransferProtocols()
> getSandboxDestURI( myJobID, choosenProtocol )
>   [ Upload files here ]
> jobStart( myJobID )
> 
> There the JDL is given only once and it is stored by the instance of 
> WMProxyAPI you are working with. I don't see the need to change the JDL 
> anywhere... 
> 
> 
> Cheers, Ariel
> _______________________________________________
> geclipse-dev mailing list
> geclipse-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/geclipse-dev
-- 
==============================================
Dr. Harald Kornmayer
Principal Researcher
NEC Laboratories Europe, IT Research Division
NEC Europe Ltd.
Rathausallee 10, D-53757 St. Augustin, Germany
Phone:       +49 / 2241-92 52 57
Fax:         +49 / 2241-92 52 99
E-mail:      harald.kornmayer@xxxxxxxxxxxx
 
NEC Europe Ltd., Registered Office: NEC House, 
1 Victoria Road, London W3 6BL, 
Registered in England 2832014 
==============================================