Re: [flux-dev] follow up on sync mechanism collaboration

[Martin Lippert <mlippert@xxxxxxxxx>]

Hey Sun,

comments line…

> So I've tested the flux demo, checking out the prototype and tested the file sync and the node.js server. It works really well ! wow :) I'm impressed

Great to hear that. Thanks!!! :-)

> So I'm very excited to use in with Codenvy, for file system sync and maybe more in the future ! I've also listen to the pod cast on se-radio, I liked it.
> So our first step would be, starting from the prototype, write a simple CLI command that would run a sync process on the file system and synchronize against a workspace/or a project in Codenvy.
> 
> We would like to write a first implementation with flux and codenvy, I was wondering what was the node server doing ?
> From my understanding,
>   - it hosts the Orion editor + the simple browser resource navigator
>   - it serves the project resources, and act like a Flux client. If any message requesting a resource, it would managed to reply with the right file attached.
>   - it serves the websocket "server" localhost:3000
>   - it keeps the project files + metadata (timestamp, etc ...) in a MongoDB
> 
> Am I right ?

Let me explain this in a bit more detail:
In general the node server is doing all those different things at the moment for your convenience. It is composed out of independent parts that could be used and run separately from each other.

The “startup-all-in-one.js” is the entry point to get everything up and running from a single node process, but in the end you should be able to have a process/startup script for each component separately from each other.

>   - it hosts the Orion editor + the simple browser resource navigator


yes, that is being served as static web content (and could be served by a separate process as well)


>   - it serves the project resources, and act like a Flux client. If any message requesting a resource, it would managed to reply with the right file attached.


there is a repository implementation that works on an in-memory base, listens to messages, and keeps a copy of files in memory. This is a prototype for a cloud backup repository that is able to keep files in sync with other repositories. There is also a different implementation available that stores the files and resources in a MongoDB instead of in-memory, but serves the same purpose.


>   - it serves the websocket "server" localhost:3000


This is the central part where the messaging backbone is located. This will be run independently of all the other parts as the message backbone system


>   - it keeps the project files + metadata (timestamp, etc ...) in a MongoDB



As I said, this is an additional repository implementation that keeps files in a MongoDB instead of in-memory. Those repositories can also run in a separate process from the messaging backbone. They connect via a websocket channel already.

In the end, there can be an infinite number of repositories running in the cloud totally independent of each other. So you could run the in-memory repository as well as the MongoDB repo - as well as future repo implementations as well.


> Server side, Codenvy has a rest api to get the project resources and is using a custom VFS (with ACL) for storage. So I can see 2 options:
> - run the node server next to the Codenvy main server, the Codenvy main server would work like a Flux client, it will also need to be able to provide resources and request for modified resources.
> - re implement what we need in Codenvy main server as a Java module, don't use the nodejs server, if it would do nothing else than what i've listed before


I am not very familiar with the Codenvy structure and API, but I could imagine something like an add-on to Codenvy that acts like a repository (from the flux point of view). This component would need to connect to the Flux websocket messaging channel, listens to the file sync messages, and send out the corresponding file sync messages. This could be something like the MongoDB repo implementation that we have, but instead of using MongoDB to store the resources it uses the Codenvy storage.

Maybe this could be implemented using the Codenvy rest api. But I am not sure whether it would be possible for this component to be notified when a resource gets changed.


> As I didn't find any, i'll write some sequence diagram to explain how the file system sync work, what messages are sent and by who, will share it with you.


That would be great.
I also expect the file sync protocol to change in the future (as I said), but we can work on this.


> WDYT ?


See my comments above. Would be great to work on this with you guys.

Cheers,
-Martin






> 
> Cheers,
> 
> Sun
> 
> Le 02/07/2014 18:51, Martin Lippert a écrit :
>> Hey!
>> 
>> In the call last week between the Flux team and the Codenvy folks, we discussed a possible collaboration around the basic syncing mechanisms.
>> The call got recorded and is on YouTube: https://www.youtube.com/watch?v=TJWsawVkpjs, but it is 1h17m long, so be prepared… ;-)
>> 
>> As discussed I started to document a few general ideas about the architecture of the prototype on the wiki:
>> https://wiki.eclipse.org/Flux/Prototype
>> 
>> This page contains a few pointers into the code and explains the general ideas behind the mechanism of the prototype. There are a lot of open questions and things that need to be addressed in the future, but we can start a discussion around that here, if you like.
>> 
>> Looking forward to working on the more robust and bullet-proof synching mechanism.
>> 
>> Cheers,
>> -Martin
>> 
>> 
>> _______________________________________________
>> flux-dev mailing list
>> flux-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/flux-dev
> 
> _______________________________________________
> flux-dev mailing list
> flux-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/flux-dev