Re: [geclipse-dev] Transferring local input files during job submission

[Ariel Garcia <garcia@xxxxxxxxxx>]

Hi Mariusz,

> I have to make changes in JDL, because original JDL refer to files on
> user's local machine, which is not accessible from worker node.

of course the local machine cannot (and shouldn't, we don't want to have a 
gridftp server on our geclipse!!) be accessed from RB/CE/WN

but that is why you write:
   InputSandbox = { <SOME-FILES> };
in your JDL, those files are uploaded from the local machine (or possibly 
also SE servers) to the WMS/RB. From there the CE fetches the files, and 
the WN fetches them from the CE (a gLite "optimization" ;-)

If you wouldn't specify 
   InputSandboxDestFileName
at all, all those files would keep their original name but land in the 
working-dir of the job in the WN

Actually a JDL with 
  Executable      = "/bin/cat";
  Arguments       = "input.txt";
  InputSandbox = { "file:///tmp/input.txt" };
works without any problems even if "file:///tmp/input.txt" is on the local 
machine (using the cli, for the API it should work too).

On the other hand, the file renaming when uploading using 
InputSandboxDestFileName  doesn't seem to work at all for me from the 
command line (my UI in a glite 3.0 one, perhaps it would work in 3.1?)
the inputsandbox file is uploaded with the original name...

> Submission will look like:
> 1. Remove from JDL (in g-Eclipse memory) all references to local files
> 2. Register job on WMS
> 3. Send local files to sandbox, into location returned during job
> registration
> 4. Start job on WMS

Ok, i am trying to argue that  point 1. shouldn't be required

> We cannot change JSDL->JDL transformation file (because of license), so

that is fine

> I have to transform JSDL to JDL (in g-Elipse memory), change JDL (still
> in g-Eclipse memory) and submit it.
> I'm not sure if after sending local files to WN sandbox, should I refer
> to those files in submitted JDL?
> I guess not, because those files are in right place (sandbox).

i think you can leave in the JDL all entries which are not relevant to the 
current submission step, they shouldn't matter!

Are you using the WMProxy API provided by glite, right?
so what you mean (or what i'd expect) is the following method calls:

myJobID = jobRegister( myJDL, delegationId )
getTransferProtocols()
getSandboxDestURI( myJobID, choosenProtocol )
  [ Upload files here ]
jobStart( myJobID )

There the JDL is given only once and it is stored by the instance of 
WMProxyAPI you are working with. I don't see the need to change the JDL 
anywhere... 


Cheers, Ariel