Re: [ecf-dev] E-intro [Was Efficient downloads]

[Scott Lewis <slewis@xxxxxxxxxxxxx>]

Hi Filip,

A couple of comments interspersed.

Filip Hrbek wrote:
> Hi Scott,
>
> thanks for your quick reply.
>
> We need a very robust file transfer implementation for our 
> application. I believe that if we cooperate on this we'll create 
> something really good.
>
> Purpose:
> - have downloads as fast as possible (optimize downloads for case that 
> the download capacity is bigger than single location's upload capacity)
> - have downloads reliable (download everything or nothing, avoid 
> saving corrupted files if possible)
> - have downloads convenient (automatic recovery, discovery of 
> alternatives)
> - let downloads have good UI (verbose enough, good cancel/pause/resume 
> response)
>
> Let me list features we are looking for (just a brainstorming, no 
> specification yet):
> - multiple protocol support (http/https/ftp/file...)

This is what ECF is fundamentally about, so the filetransfer API should 
help with this.

> - pause/resume using partial content requests (connection can be 
> closed/reopened before an attempt to resume)

This will be up to the implementation to provide, of course.  I *think* 
that the IFileTransferPausable as currently defined gives sufficient 
semantics for what you want to do here, but if not please let us know.

> - resume from a different location (e.g. different mirror)

Hmm.  Don't know how you are going to accomplish that without something 
quite different from normal http, but sounds interesting.

> - multi threaded download of one artifact from one location (starting 
> at different content offsets)
> - multi threaded download of one artifact from several locations at 
> the same time
> - multi threaded download of several artifacts at the same time (using 
> either single threaded or multi threaded single artifact downloads)

ECF's structure is based upon asynchronous download...i.e. each request 
has a job/thread associated with it that is responsible for 
asynchronously sending back events (data received, failed, completed, 
etc) to a listener (IFileTransferListener).  So this shouldn't be a 
problem as it's built into the fabric of ECF.

> - retrieving information from special headers (like Content-Disposition)
> - detecting URL redirections to final mirrors

I'm not sure what you are going to use to implement this, but would be 
curious to find out.

> - detecting approximate user's geographical position
> - collecting download statistics (slow/unreliable mirror discovery)
> - dynamic switching between mirrors in case of slow download
> - automatic recovery after connection timeout

I don't think these have any API implications (except perhaps the 
collecting download stats), but if I'm wrong about that please let me know.

> - good API for verbose progress bars (ability to create a good UI)

The asynchronous structure of ECF's filetransfer API should be 
well-suited to this, as progress bars can simply be updated in listener 
code...

> - good API for statistics processing

I think you would need to describe what statistics are desired here.  We 
can easily add adapter interfaces for collecting statistics associated 
with a given file retrieval/all to ecf or individual providers, but 
would need to know what stats are of interest.

> - configurable behavior (enable configuring features described above)
> - ...and perhaps many more (?)
>
> I don't know how much from this you already cover. I'll study what you 
> currently provide first. All the features don't have to be implemented 
> at the same time, we can define development stages to start with more 
> important issues and continue with less important ones.

Yes, I agree that's certainly the right approach.  Please feel free to 
ask any questions as you study what's there.

Thanks,

Scott

> Best regards
>  Filip Hrbek
>
> Note: I'll be offline from this Friday until next Sunday, so don't 
> worry if I don't answer your emails during next week. I'll be 
> definitely back from Monday (11th June).
>
>
>
> Scott Lewis napsal(a):
> > Hi Thomas and Filip,
> >
> > Thanks!  The help is most welcome.
> > The filetransfer API is in plugin:  org.eclipse.ecf.filetransfer.
> > The bundle docs (with pointers to javadocs) is:  
> > http://wiki.eclipse.org/index.php/ECF_API_Docs#File_Transfer_API
> >
> > There are also pointers in wiki to project sets source and test code.
> >
> > There is a base provider plugin:  
> > org.eclipse.ecf.provider.filetransfer.  This plugin has an extension 
> > point that allows other plugins to insert implementations associated 
> > with various protocols.  See the extension point 
> > org.eclipse.ecf.provider.filetransfer.fileTransferProtocolFactory.   
> > The httpclient implementation uses this extension point (and is 
> > defined in another plugin:  org.eclipse.ecf.provider.filetransfer.http).
> >
> > The basic implementation that uses jobs for doing pause/resume 
> > (IFileTransferPausable) is in 
> > org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.  
> > This can/could be used (and appropriate methods overidden) or 
> > replaced...up to you as to what makes sense.
> >
> > Filip please let me and everyone know if you have any 
> > questions/comments/suggested or needed changes...and 
> > thanksinadvance!  I would/will be curious myself about what 
> > http/https impl you are looking to use and what specific http/https 
> > protocol features you will be using as well.
> >
> > Scott
> >
> > Thomas Hallgren wrote:
> > > Hi Scott,
> > > We, (the Buckminster team) would like to offer our help in 
> > > implementing protocol-level pause/resume and recovery capabilities 
> > > for the ECF http/https file transfer. I'd like you to meet Filip 
> > > Hrbek. He is the Buckminster guy who currently works on improving 
> > > our materialization. He will be our main resource for this, should 
> > > you accept our help. So Filip, meet Scott. Scott meet Filip.
> > >
> > > Regards,
> > > Thomas Hallgren
> > >
> > >
> > >
> > > Scott Lewis wrote:
> > > > Hi Thomas,
> > > >
> > > > Thomas Hallgren wrote:
> > > > > Hi Scott,
> > > > > Thanks for a very elaborate answer. This looks very promising. 
> > > > > Well structured, low overhead. Only thing missing if I read you 
> > > > > correctly is true pause/resume on protocol level (and the failure 
> > > > > recovery that I think will come as a bonus). Do you have any sense 
> > > > > what it would take to implement that?
> > > >
> > > > It depends upon each protocol and it's corresponding support in 
> > > > (some) protocol library.  I assume you are most interested in 
> > > > http/https.  I'll take a look at the httpclient 3.0.1 support for 
> > > > pause/resume and failure recovery.
> > > >
> > > > Scott
> > > >
> > > >
> > > > _______________________________________________
> > > > ecf-dev mailing list
> > > > ecf-dev@xxxxxxxxxxx
> > > > https://dev.eclipse.org/mailman/listinfo/ecf-dev
> > >
> > > _______________________________________________
> > > ecf-dev mailing list
> > > ecf-dev@xxxxxxxxxxx
> > > https://dev.eclipse.org/mailman/listinfo/ecf-dev
> >
> > _______________________________________________
> > ecf-dev mailing list
> > ecf-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/ecf-dev
> _______________________________________________
> ecf-dev mailing list
> ecf-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/ecf-dev