[equinox-dev] [prov] processing steps, restartable downloads and ECF

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

Thanks to Stefan we have been introducing
 the notion of ProcessingSteps to munge the content as it is downloaded
from an artifact repository.  This allows for things like inline MD5
digest checking, unpack200 processing, delta merging, signature checking,
...  All great stuff.  Pascal just raised a very interesting
question.  How do we handle restarting?  Some background.  In
the current prototype (in my workspace, not yet in CVS) there is a chain
of ProcessingStep objects.  Each step is actually an OutputStream
that knows about the next step (output stream) in the chain.  When
a byte is written to step/stream N, it is processed (counted, transformed,
...) and then the result passed on to step N+1.  This repeats until
finally the content gets to the last stream in the chain which is usually
a FileOutputStream of some sort and so the content is then written to disk.
 All is well.

Now, what happens if we crash or the
user somehow pauses the download?  The content is partially processed/transformed
but it would likely be too costly for each step to persist its intermediate
results.  It would be more likely that somehow the raw content coming
in to the head of the chain of steps is cached and then when the download
is restarted after a crash/exit, the chain is recreated and the download
is effectively replayed through the chain from the cache.  When that
is done, the further content from the source would then be pushed through
the chain.

So, two questions.  Does this make
sense?  and if so, how should we implement this?  I wonder if
ECF has some technology/support/designs in this area since it seems they
support restartable downloads.  Scott?

Jeff