Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [ecf-dev] Efficient downloads

Hi Scott,
Thanks for a very elaborate answer. This looks very promising. Well structured, low overhead. Only thing missing if I read you correctly is true pause/resume on protocol level (and the failure recovery that I think will come as a bonus). Do you have any sense what it would take to implement that?

- thomas

Scott Lewis wrote:
Hi Thomas,

Thomas Hallgren wrote:
I'm looking for ways to improve Buckminsters download capabilities and I have some questions regarding the ECF filetransfer provider. As you might tell by my questions, I'm fairly new to the whole thing :-)

What is the current status for implementations (http, ftp, and bittorrent in particular)?

We use apache httpclient 3.0.1 for implementing http/https support, the jre URLConnection impl for ftp, and we have a working EPL bittorrent implementation (created by committer Remy Suen as part of the Google SOC project 2006). The URLConnection impl also supports file and jar URLs (of course). These are all for file retrieval of course...I assume this is the use case you are interested in (rather than peer-to-peer file sending which the ECF file transfer API also supports). The top-level container adapter for file retrieval is: IRetrieveFileTransferContainerAdapter[1].

Is it possible to resume a download from a certain position (e.g. using the http Range header attribute)?

We have an adapter interface called IFileTransferPausable, which exposes semantics for pausing/resuming a file transfer. This is currently implemented (with the protocols listed above except for bittorrent) via the Job.sleep() and Job.wakeUp() methods. It doesn't have to be implemented this way for a given protocol (i.e. it could use protocol specific mechanisms where available), but we haven't done that yet.

Does ECF provide any resume or recovery after failure capabilities out-of-the-box?

Resume, yes via IFileTransferPausable adapter. The adapter is optionally implemented by a given provider implementation.

Recovery from

Do you have any abstractions for this?
How much overhead does ECF add in terms of size in a headless environment assuming an eclipse runtime is active already?

It depends upon which ECF plugins are used. If you are using just the file transfer api and impls then the filetransfer API bundle (jar) is currently ~25k and it depends upon the ECF core API (61k), and the ECF identity API (39k). For file transfer, these are small by comparison with the size of the apache httpclient implementation bundle (from Orbit...~290kb).

Just FYI, we went to some significant trouble to reduce the dependencies of the ECF core APIs to the bare minimum and make them as small as possible, meaning that for all of these (filetransfer API, core API, identity API) they require only the CDC 1.0/Foundation 1.0 runtime + the equinox jobs bundle. Unfortunately, I don't yet have detailed numbers of memory usage by bundle...or for the runtime behavior of httpclient under various load conditions separate from other analyses that people have done of httpclient (I haven't searched to find these, but I imagine they do exist...if you find them please make them known to us).

Also...FYI, here's some new subsystem-level docs on ECF that might prove useful:

There's also some pointers on those pages to test code for filetransfer.

Let us know if we can be of more help.



Thomas Hallgren

ecf-dev mailing list

ecf-dev mailing list

Back to the top