Re: [geclipse-dev] Replicable File Systems - proposal tothe corechanges

[Mateusz Pabis' <uranium@xxxxxxxxxxxxx>]

Harald Kornmayer pisze:
> Hi Mateusz
>
> First of all, maybe we concentrate a few more minutes on the use cases for
> the LFC. 
>
> The LFC EFS implementatin should be used by the user. The
> LogicalFileCatalogue was introduced to hide the details from the data grid
> from the user. LFC provides a "file system" which spans over many
> distributed Storage resources. So the LFC file system is "virtual". The
> replication mechanism is just one of the features of the LFC. The main
> feature is to hide complexity and to enable a user to find his distributed
> data much easier. 

I fully agree with you!

> But the LFC looks like a real file system. And at the end of each directory
> tree is a "file" with a name. But LFC provides here only the link to the
> original endpoint of the file (cardinality [1, inf.)). As a user I just want
> to get the file open by clicking on it. It should not matter where the file
> resides. From a performance view point, the closest file should be opened. 

I agree with you! With one but:
please define "closest", and would be nice to define it by measure, it 
would be nice if this measurement is light, so the measurement time << 
transfer of a 1MB file.

> So from my point of view, the file system in the project view resp. Grid
> connection view looks like: 
>
> - LFC
>   - directory 1
>   - directory 2
>     - subdirectory 3
>       - file1.txt
>       - file2.txt 
>     - subdirectory 4
>     ...
>
> The user should not see anything about the locations of the file's replicas
> in these views.
> But I propose to show some details in the propoerties view like:
>
> filename	file1.txt
> checksum    1234567
> closest rep augustin.de
> replicas
>   - URL     gsiftp://poznan/xx/xx/xx/file1.txt
>   - URL     s3://xx/xx/xx/file1.txt
>   - URL     gsiftp://cyprus/zz//xx/xx/file1.txt
>
> So the user can get more information from the properties view.

I fully agree with you!

Good news: It's already done, it's already in HEAD, it's already in 
Nightly Builds, it was like this on the review!
Even the closest replica was opened (*)

> So now coming to actions: 
>
> 1. Drag'n'drop: 
> The user must be able to drag a file from the LFC connection to any other
> connection. Before the real transfer starts, 
> our framework looks for the "closest SE" and uses that to make the transfer.

Already done, with the remark of (*)!

> It work for the "import drag'n'drop" correspondingly. Look for the closest
> SE and transfer the data there and put the information in the Cataloque. 

This is what I was about to implement. I had in mind method:
URI getDefaultReplicaURI; // get default location where replicas will be 
created
This method is dynamic, so it can decide during runtime what host, port, 
protocol, path is a default place for replicas.

I fully agree with your point! I've foreseen this usecase, that's why I 
proposed this method to include in my extension.

> 2. Actions: 
> If the users wants to manage some aspects of LFC, he gots the corresponding
> entries in the context menues: 
> Here we need a much better understanding of the actions a Grid user wants to
> perform.
> We must provide the support for the most common use cases. And not for every
> use case, that is possible. 

I fully agree, please bring me more users. I have one, he's asking for 
replica management view, but he's a pro, so he fears no replica 
management ;-).

> If the file is already in the LFC, actions might be: 
> - replicate...
  low level already done, still needs work on the UI side,

> - delete all replicas
  done

> - delete replicas ...
  low level already done, still needs work on the UI side,

> - manage replicas ...
  here comes ReplicaView! Or do you want it to be a dialog?


> IF the file is in another file system type (srm, gsiftp,..) and there is a
> LFC connection open, the following actions might
> appear in the context menu: 
> - register in LFC
  low level already done, UI is cumbersome for user, work pending

> - register in LFC ...
  doppelganger? I don't imagine how to register a file in LFC without a 
dialog.

> - unregister from LFC
  this is a SE we are talking of - how do you know if this file is 
already a replica registered in LFC for an entry?

> - unregister from LFC ...
  same as above,

> - delete from LFC (deleting all replicas)
  same as above,

> You see that there are already different possibilities, but we need to find
> the most useful once, and that ones we implement. 


> So for me the LFC filesystem ends with the "virtual" leaves. If I understood
> you right, your LFC ends with the "real replica" leaves.

No, you got me wrong.
There is this "hide-it-all" already done, but since I have this one user 
who also asks for "show-me-all" I decided to provide it _as_well_, not 
_instead_.

> I don't want to see them in in the filesystem/connection. That is complexity
> we want to hide!!!

I agree. I don't want to see them there as well, nobody does (I hope). 
But I want to see them somewhere, new dedicated view for sophisticated 
replica management is a nice place.

> Maybe it is worth to define the useful processes and the GUI interactions,
> before we start to implement the modell. 

Hiding it from the user does not mean hiding it in the model.
I would like to have replica support in our model, to make the things go 
smoother. We stick to the MVC pattern, my propose is as follows:
Model: supports replicas,
Controller: hides replicas, but show them on the user's demand,
View: two different: one based on output-with-hidden-replica and another 
on output-with-all-the-replica-cream from the Controller.

So I would like to have a screwdriver for a user, and a swissarmy knife 
for an advanced user.


And now a big question: why I want these changes in the model?
Right now, the model does not support replicas at all! Current 
implementation is UI+core LFC functionality in one package. When I 
separate them, there will be UI LFC and core LFC. So the replica will be 
only gLite specific. I think replicas are not the invention of gLite, 
and _not_having_ such a support in gEclipse can disclose us as a 
platform for the middleware we don't know yet.

(*) closest replica is defined like this: first replica which does not 
throw an exception while performing getInputStream().

Further comments are welcome.

--
Mateusz Pabis