Re: [chemclipse-dev] What is the suitable file format for spectrum data

[Philip Wenig <philip.wenig@xxxxxxxxxxxxx>]

Yep, HDF5 is a good choice.

Let's include the HDF5 library as a 3rdpl dependency. I'll be back in the office on Thursday, so let's discuss further steps then.

Am 05.09.2016 um 08:46 schrieb Trig Chen:

I've tested H2 MVStore and HDF5. 

MVStore can't open the same file in both writing and reading thread. 

But HDF5 can open the same file for write and read in two thread.

So, for my case, I think I can use HDF5 to write the data acquired from instrument to a file, and at the same time read the data from the same file and plot it in OpenChrom. 

Best regards,

Trig

2016-09-01 19:16 GMT+08:00 Philip Wenig <philip.wenig@xxxxxxxxxxxxx>:

I would recommend HDF5, but H2 is also fine. It depends on how much data you expect to record. Normally, TOF records are big (50 MB to ~15 GB), so HDF5 would be the better choice here.

Please ask the people from the Science mailing list whether they have experience with one or the other HDF5 library. They can definitively give an advise which one to take.


Best,
Philip

Am 01.09.2016 um 07:37 schrieb Trig Chen:

There are two Java libraries to manipulate HDF5 file. One is from HDF Group and the other one (JHDF5,  created by Rinn Bernd,ID SIS) is from https://wiki-bsse.ethz.ch/display/JHDF5. The former one is formal but awesome to use. The later one is more Java style and easy to use.

I've made a preliminary try on HDF5.  It would need more addition effort to make an encryption, checksum or compression.

By the way,both libraries are a little big (~10MB for Windows/Linux/OSX).

I found the MVStore from H2 Database attractive. Simple API, small footprint of library (~200KB).

Best regards,

Trig

2016-08-31 14:58 GMT+08:00 Philip Wenig <philip.wenig@xxxxxxxxxxxxx>:

I wouldn't recommend using a database. Rather try to use HDF5. We recently had some discussion in our Science Working Group how to best utilize HDF5. Some projects like DAWNSci use HDF5 already. Please ask at the science mailing list to get best practice infos. If you'd like to use HDF5, we could try to add the third party dependencies.

https://dev.eclipse.org/mailman/listinfo/science-iwg
science-iwg@xxxxxxxxxxx


Best,
Philip

Am 30.08.2016 um 18:18 schrieb Trig Chen:

Hi all,

When I implemented the function of acquiring data from spectrometer, I found that the zip file format  currently used in OpenChrom was not suitable for random and parallel writing data. The zip format is just suitable for one-time writing in end.

What file format do you use in the case of parallel writing and reading big data? 

Database such as H2, SQLite?  or HDF?

Best regards,

Trig

_______________________________________________
chemclipse-dev mailing list
chemclipse-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/chemclipse-dev

-- 
~~~~~~~~~~~~~~~~~~~~~~~~
OpenChrom - the open source alternative for chromatography / mass spectrometry
Dr. Philip Wenig » Founder » philip.wenig@xxxxxxxxxxxxx » http://www.openchrom.net
~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________ chemclipse-dev mailing list chemclipse-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/chemclipse-dev

_______________________________________________
chemclipse-dev mailing list
chemclipse-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/chemclipse-dev

-- 
~~~~~~~~~~~~~~~~~~~~~~~~
OpenChrom - the open source alternative for chromatography / mass spectrometry
Dr. Philip Wenig » Founder » philip.wenig@xxxxxxxxxxxxx » http://www.openchrom.net
~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________ chemclipse-dev mailing list chemclipse-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/chemclipse-dev

_______________________________________________
chemclipse-dev mailing list
chemclipse-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/chemclipse-dev

-- 
~~~~~~~~~~~~~~~~~~~~~~~~
OpenChrom - the open source alternative for chromatography / mass spectrometry
Dr. Philip Wenig » Founder » philip.wenig@xxxxxxxxxxxxx » http://www.openchrom.net
~~~~~~~~~~~~~~~~~~~~~~~~