Re: [ecf-dev] E-intro [Was Efficient downloads]

[Filip Hrbek <filip.hrbek@xxxxxxxxxxxxxx>]

Hi Scott,

thanks for your quick reply.

We need a very robust file transfer implementation for our application. 
I believe that if we cooperate on this we'll create something really good.

Purpose:
- have downloads as fast as possible (optimize downloads for case that 
the download capacity is bigger than single location's upload capacity)
- have downloads reliable (download everything or nothing, avoid saving 
corrupted files if possible)
- have downloads convenient (automatic recovery, discovery of alternatives)
- let downloads have good UI (verbose enough, good cancel/pause/resume 
response)

Let me list features we are looking for (just a brainstorming, no 
specification yet):
- multiple protocol support (http/https/ftp/file...)
- pause/resume using partial content requests (connection can be 
closed/reopened before an attempt to resume)
- resume from a different location (e.g. different mirror)
- multi threaded download of one artifact from one location (starting at 
different content offsets)
- multi threaded download of one artifact from several locations at the 
same time
- multi threaded download of several artifacts at the same time (using 
either single threaded or multi threaded single artifact downloads)
- retrieving information from special headers (like Content-Disposition)
- detecting URL redirections to final mirrors
- detecting approximate user's geographical position
- collecting download statistics (slow/unreliable mirror discovery)
- dynamic switching between mirrors in case of slow download
- automatic recovery after connection timeout
- good API for verbose progress bars (ability to create a good UI)
- good API for statistics processing
- configurable behavior (enable configuring features described above)
- ...and perhaps many more (?)

I don't know how much from this you already cover. I'll study what you 
currently provide first. All the features don't have to be implemented 
at the same time, we can define development stages to start with more 
important issues and continue with less important ones.

Best regards
 Filip Hrbek

Note: I'll be offline from this Friday until next Sunday, so don't worry 
if I don't answer your emails during next week. I'll be definitely back 
from Monday (11th June).



Scott Lewis napsal(a):
> Hi Thomas and Filip,
>
> Thanks!  The help is most welcome.
> The filetransfer API is in plugin:  org.eclipse.ecf.filetransfer.
> The bundle docs (with pointers to javadocs) is:  
> http://wiki.eclipse.org/index.php/ECF_API_Docs#File_Transfer_API
>
> There are also pointers in wiki to project sets source and test code.
>
> There is a base provider plugin:  
> org.eclipse.ecf.provider.filetransfer.  This plugin has an extension 
> point that allows other plugins to insert implementations associated 
> with various protocols.  See the extension point 
> org.eclipse.ecf.provider.filetransfer.fileTransferProtocolFactory.   
> The httpclient implementation uses this extension point (and is 
> defined in another plugin:  org.eclipse.ecf.provider.filetransfer.http).
>
> The basic implementation that uses jobs for doing pause/resume 
> (IFileTransferPausable) is in 
> org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.  
> This can/could be used (and appropriate methods overidden) or 
> replaced...up to you as to what makes sense.
>
> Filip please let me and everyone know if you have any 
> questions/comments/suggested or needed changes...and thanksinadvance!  
> I would/will be curious myself about what http/https impl you are 
> looking to use and what specific http/https protocol features you will 
> be using as well.
>
> Scott
>
> Thomas Hallgren wrote:
> > Hi Scott,
> > We, (the Buckminster team) would like to offer our help in 
> > implementing protocol-level pause/resume and recovery capabilities 
> > for the ECF http/https file transfer. I'd like you to meet Filip 
> > Hrbek. He is the Buckminster guy who currently works on improving our 
> > materialization. He will be our main resource for this, should you 
> > accept our help. So Filip, meet Scott. Scott meet Filip.
> >
> > Regards,
> > Thomas Hallgren
> >
> >
> >
> > Scott Lewis wrote:
> > > Hi Thomas,
> > >
> > > Thomas Hallgren wrote:
> > > > Hi Scott,
> > > > Thanks for a very elaborate answer. This looks very promising. Well 
> > > > structured, low overhead. Only thing missing if I read you 
> > > > correctly is true pause/resume on protocol level (and the failure 
> > > > recovery that I think will come as a bonus). Do you have any sense 
> > > > what it would take to implement that?
> > >
> > > It depends upon each protocol and it's corresponding support in 
> > > (some) protocol library.  I assume you are most interested in 
> > > http/https.  I'll take a look at the httpclient 3.0.1 support for 
> > > pause/resume and failure recovery.
> > >
> > > Scott
> > >
> > >
> > > _______________________________________________
> > > ecf-dev mailing list
> > > ecf-dev@xxxxxxxxxxx
> > > https://dev.eclipse.org/mailman/listinfo/ecf-dev
> >
> > _______________________________________________
> > ecf-dev mailing list
> > ecf-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/ecf-dev
>
> _______________________________________________
> ecf-dev mailing list
> ecf-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/ecf-dev