Re: [science-iwg] Data Structures Project Proposal

[Erwin de Ley <erwin.de.ley@xxxxxxxxxx>]

I would also like to add Triquetrum as interested party/project.

This might add a slightly different use-case for the data structures, compared to the other mentioned projects that are often building scientific workbenches.

For Triquetrum, we should be able to run big loads of "head-less" workflows in different runtimes (workbench, web servers, batch systems, ...).

E.g. we would need to be able to support some kind of lightweight "URIs" or other kinds of references to data items, subsets, slices, ... within multiple datasets.
Such URIs may need to be passed between workflow actions, serialized and sent around on a cluster etc.
(there is/was some specific support for this in DAWN's dataset API during a pilot project with Passerelle workflows some time ago, to identify slices)

It would also be required that the data structure implementations can be used in non-eclipse applications (but still OSGi).
E.g. the implementation should not assume the presence of workspaces, projects etc and other things that may seem natural when using the eclipse resources APIs.

And I'm sure we will discover other pecularities ;-)


cheers
erwin


Op 11/12/2015 om 15:48 schreef Tracy Miranda:

Jay,

Great, thanks for the update, especially clarity on the scope of the project.  

Peter, Jonah & I were just discussing the project proposal on Wednesday. Jonah will go ahead and create it and distribute for feedback from the science working group. We will liase with all parties who have expressed interest to ensure this is captured in the proposal, committer list, etc. (So far this list looks like Oakridge/ICE, Diamond/DAWN, EAVP, Marintek, Werner/UOMo). Please let me know if I've forgotten anyone, or if you want to be on the proposal.

Regards,

Tracy 

On Fri, Dec 11, 2015 at 2:24 PM, Jay Jay Billings <jayjaybillings@xxxxxxxxx> wrote:

Tracy,

Yes, we discussed it and we think it should move forward. As you suggest in your email, we think Jonah and Peter would be good co-project leads. We would like to see this project contain the IDataSet packages, the data structures packages from ICE and the data structures from EAVP so that we start building a single data structures project for Science. (N.B. - the initial contribution for EAVP hasn't been added to the repo yet.)

The list of committers will need to include people from all three of those projects as well as any others that want to add to the initial contribution.

Although we like the idea of contributing to the Collections project, we felt that the scope of work for this project was different enough that it warranted its own project.

One possible future direction that we discussed which interests Greg and I quite a bit is figuring out how we can make these data structures architecturally aware so that for very large data sets the work can be processed on multiple cores or a GPU.

I forget what we thought the name should be.

Greg and Torkild, what did I miss from our discussion? Matt please chime in too since you weren't at the meeting.

Jay

On Fri, Dec 11, 2015 at 7:49 AM, Tracy Miranda <tracy@xxxxxxxxxxxxxxxx> wrote:

Picking up the data structures thread again - did the steering committee discuss this - what were the outcomes?

On Fri, Nov 6, 2015 at 9:43 PM, Jay Jay Billings <jayjaybillings@xxxxxxxxx> wrote:

Torkild,

With respect to the agenda, consider it done.

Jay

On Nov 6, 2015 4:40 PM, "Torkild Ulvøy Resheim" <torkildr@xxxxxxxxx> wrote:

From what I gather there is absolutely some mutual benefit gained from working with the Collections project. If we were to place the SWG data structures in this (project), I guess it would only be a matter of organizational issues – which I think can be easily resolved.

I’d like to put this on the agenda for the next SWG steering committee meeting.

Best regards,
Torkild

> 6. nov. 2015 kl. 22.27 skrev Mike Milinkovich <mike.milinkovich@xxxxxxxxxxx>:
>
>
> To be clear, I didn't mean to imply any obligation to merge with Collections. That is entirely up to you. However, I do know that the people in that project are very interested in community collaboration, so I'm sure that they would welcome any and all conversations about mutual interests.
>
> Mike Milinkovich
> mike.milinkovich@xxxxxxxxxxx
> +1.613.220.3223
>   Original Message
> From: Torkild Ulvøy Resheim
> Sent: Friday, November 6, 2015 4:21 PM
> To: Science Industry Working Group
> Reply To: Science Industry Working Group
> Subject: Re: [science-iwg] Data Structures Project Proposal
>
> Hi,
>
> I had to google “Five Guys bag”, those images are a bit scary ;-)
>
> I think makes sense to put "data structures" together with “collections” – from a user’s perspective we just need to make sure that these APIs can work together in a good way.
>
> On the other side, do we want the SWG data structures API in the Technology top level project? What implications would that have for a potential top level science project?
>
> Back to helping my kids with Minecraft.
>
> Best regards,
> Torkild
>
>> 6. nov. 2015 kl. 21.36 skrev Jay Jay Billings <jayjaybillings@xxxxxxxxx>:
>>
>> Everyone,
>>
>> I'm glad to see this discussion is already happening. I have a Five Guys bag in front of me, so I'll say more later. Here are two points:
>>
>> I would like to include ICE's org.eclipse.ice.datastructures as part of this project too. No reason to keep them separate from the rest of our data structures.
>>
>> We might consider contributing all of these to the project that Mike mentioned instead of starting our own. Just a thought for now as I review that project further.
>>
>> Back to my Five Guys,
>> Jay
>>
>> On Nov 6, 2015 9:36 AM, "Campbell, Stuart I." <campbellsi@xxxxxxxx> wrote:
>> +1
>>
>> Dr Stuart Campbell
>> Data Analysis Team Lead
>> Neutron Data Analysis & Visualization Division
>> Oak Ridge National Laboratory
>> 1 Bethel Valley Road
>> Oak Ridge, TN 37831-6475, USA
>>
>>
>> From: science-iwg-bounces@xxxxxxxxxxx <science-iwg-bounces@xxxxxxxxxxx> on behalf of Peter.Chang@xxxxxxxxxxxxx <Peter.Chang@xxxxxxxxxxxxx>
>> Sent: 06 November 2015 8:58 AM
>> To: science-iwg@xxxxxxxxxxx
>> Subject: Re: [science-iwg] Data Structures Project Proposal
>>
>> Hi all,
>>
>>
>> I will throw “January” into the ring (JAva NUmerical ARraY).
>>
>>
>> Regards,
>>
>> Peter
>>
>>
>>
>> From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Ian Mayo
>> Sent: 06 November 2015 13:50
>> To: Science Industry Working Group
>> Subject: Re: [science-iwg] Data Structures Project Proposal
>>
>>
>> Tracy - I agree, most domains have a concept of "data".
>>
>>
>> I guess there is merit in using the "numerical" term - it seems to have been effective for NumPy.
>>
>>
>> Regards,
>>
>> Ian
>>
>>
>> On 6 November 2015 at 13:43, <Peter.Chang@xxxxxxxxxxxxx> wrote:
>>
>> Hi all,
>>
>>
>> I think we are open to any and all suggestions for the project name!
>>
>>
>> Peter
>>
>>
>>
>> From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Tracy Miranda
>> Sent: 06 November 2015 13:41
>> To: Science Industry Working Group
>>
>> Subject: Re: [science-iwg] Data Structures Project Proposal
>>
>>
>> Yes, I think that nicely illustrates the problem with the word 'data' - too overloaded.
>>
>>
>> On Fri, Nov 6, 2015 at 12:55 PM, <Matt.Gerring@xxxxxxxxxxxxx> wrote:
>>
>> Thanks Mike,
>>
>>
>> I did have a quick look. My understanding is that Peter’s work is more for nD numerical arrays than eclipse-collections appears to be. It seems to be more a ‘richer java collections’.
>>
>>
>> Best Regards,
>>
>>
>> Matt
>>
>>
>> From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Mike Milinkovich
>> Sent: 06 November 2015 12:33
>> To: Tracy Miranda; Science Industry Working Group
>> Subject: Re: [science-iwg] Data Structures Project Proposal
>>
>>
>>
>> I could be completely off base here, but ‎if you're interested in scalable data structures, you may be interested in the recently posted Eclipse Collections project.
>>
>>
>> ‎http://projects.eclipse.org/proposals/eclipse-collections
>>
>>
>> Mike Milinkovich
>> mike.milinkovich@xxxxxxxxxxx
>> +1.613.220.3223
>>
>> From: Tracy Miranda
>>
>> Sent: Friday, November 6, 2015 1:29 PM
>>
>> To: Science Industry Working Group
>>
>> Reply To: Science Industry Working Group
>>
>> Subject: [science-iwg] Data Structures Project Proposal
>>
>>
>> Hi all,
>>
>>
>> One of the many fruitful topics of discussion at EclipseCon was data structures for the SWG, and I'd like to summarize the progress to date and solicit some input from the group.
>>
>>
>> First a brief history:
>>
>> Back in August in Trondheim, the SWG was in general agreement having a set of standardised data structures would be the way forward for good tool interoperability. Matt Gerring ran through pros & cons of different options and proposed we adopt Diamond's IDataset.
>>
>>
>> Following on from that, Jonah Graham forked IDataset to make it more accesible to SWG members.
>>
>> https://github.com/jonahkichwacoders/org.eclipse.dataset.examples
>>
>> https://github.com/jonahkichwacoders/org.eclipse.dataset
>>
>>
>> Peter Chang then further cleaned up and restructured the classes getting them into a very good state to be used by the community.
>>
>>
>> The logical next step was to make this available as a standalone project as part of the Science Working Group. So this was brought up at the unconference this week. What I learnt is that a project can have 2 project co-leads. JayJay Billings recommended this (as is being done in Triquetrium) and it has a lot of merit. That way if we have Jonah Graham & Peter Chang as co-project leads we have a good split of responsibilities.
>>
>>
>> SWG members at the unconference were very enthusiastic and many offers of potential contributions as well.
>>
>>
>> Also, that old issue of naming cropped up, e.g. Tobias raised some good points about how the naming is very generic, Idatasets represent ndarrays but perhaps we should consider other data structures too (e.g. what about dataframes...)
>>
>>
>> In conclusion, we would like to go ahead and make this happen. Jay Billings has offered to get the ball rolling in terms of giving Eclipse Foundation the heads up. What we need now is your feedback, but mostly the list of 'interested parties' to strengthen the project proposal.
>>
>>
>> Regards,
>>
>> Tracy
>>
>>
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>>
>>
>> _______________________________________________
>> science-iwg mailing list
>> science-iwg@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/science-iwg
>>
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>>
>>
>> _______________________________________________
>> science-iwg mailing list
>> science-iwg@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/science-iwg
>>
>>
>>
>>
>> --
>>
>> Ian Mayo
>> Deep Blue C Technology Ltd
>> - New technologies, implemented well
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>>
>>
>> _______________________________________________
>> science-iwg mailing list
>> science-iwg@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/science-iwg
>> _______________________________________________
>> science-iwg mailing list
>> science-iwg@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/science-iwg
>
> _______________________________________________
> science-iwg mailing list
> science-iwg@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/science-iwg


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

--

Jay Jay Billings

Oak Ridge National Laboratory

Twitter Handle: @jayjaybillings

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg