| [january-dev] FW: January: Streams for slicing after lunch |
|
Hello, I proposed a PR (https://github.com/eclipse/january/pull/155 which should not be merged as API is not nice enough) to add stream support for slicing lazy
datasets. However Peter and Jake have proposed that I add this support to dawnsci because of the following reasons: 1.
I would have to change January to 1.8 from 1.7 2.
I could use Jake’s SliceViewIterator if the functionality was in dawnsci. 3.
[The eventual API should probably be a stream of ILazyDataset and work with .parallel()] I am inclined to do what they say because it is the path of least resistance. My question is: Does anyone actually want slice streams in January? I would have thought them the most useful feature slice sliced
bread because everything for me seems to involve iterating images out of nD data. So the lambdas would truncate everyone’s code nicely. What are your thoughts and requirements? Matt From: Chang, Peter (DLSLtd,RAL,LSCI)
The implementation is not parallelizable too whereas the ones I placed in FunctionalUtils are. I recommend adding to said class. We could add chunking metadata to January to improve parallel streaming. Peter From: Filik, Jacob (DLSLtd,RAL,LSCI)
A simple method that calls getSlice() on the lazydataset returning the IDataset, catching the DatasetException and throwing it as a runtime exception. You would have to write it yourself. Calling getSlice in the stream is not pretty because of the Exception, From: Gerring, Matt (DLSLtd,RAL,LSCI)
Thanks. What is safeSlice(…) ? Matt From: Filik, Jacob (DLSLtd,RAL,LSCI)
Current code needed to do what you want. From what I see you can't easily deal with the DatasetException that could come from getSlice, so you would need to wrap
it in a method that throws a Runtime Exception instead. From: Gerring, Matt (DLSLtd,RAL,LSCI) Hello,
final ILazyDataset
lz = Random.lazyRand(64, 64, 100, 100);
I hacked up streams for slicing so you can replace something like:
final PositionIterator
it =
new PositionIterator(new
int[]{64, 64});
final List<Number>
maxes =
new ArrayList<Number>();
while(it.hasNext())
{
int[]
pos =
it.getPos(); Slice[]
slice =
new Slice[lz.getRank()];
for (int
i = 0;
i <
pos.length;
i++) {
slice[i]
= new Slice(pos[i],
pos[i]+1); } IDataset
image =
lz.getSlice(slice);
maxes.add(image.max()); } With this: List<Number>
maxes =
lz.sliceStream(64, 64).map(set->set.max()).collect(Collectors.toList()); I find this desirable because I can never remember how to slice correctly, streams give a great alternative. If I do a PR to January I hope that you could help massage this into something more generic? (While still being simple to use…) Best Regards, Matt
-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. |