Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc) » [CDO/EMF] Modeling BIG amounts of tiny data
[CDO/EMF] Modeling BIG amounts of tiny data [message #1066971] |
Fri, 05 July 2013 09:50 |
Christophe Bouhier Messages: 937 Registered: July 2009 |
Senior Member |
|
|
Hi,
For our application, we are currently modeling a data object [1] , in
CDO this translates to an object which in an MySQL Store for CDO looks
like [2].
As we are talking about 100's of millions of these volatile objects, we
are considering modeling this differently (Size of the DB is the reason,
see the SQL table in [2]).
BTW the operations we perform is
- Writing (single values).
- Reading, with a select on the value Timestamp. (Currently with CDO
SQL query handler).
I am considering two options.
1) Using a custom data type. I am not sure what this means for CDO. I
pressume, the Value object will still be an object with lifecycle etc..
2) using a specific file based store, whereby the 'Value' will be
referred to by a CDO External reference, letting a specific EMF Resource
implementation retrieving the objects from a file.
From the community, I would like to ask if someone has experience with
1) and 2) and what should be considered. For 2) for example I can
imagine, that the query facility would be something to build from
scratch and/or would perhaps require some indexing.
Any advise on this?
Thank You,
Christophe
[1] In .xsd it looks like this:
<!--_____________________________ -->
<!--| VALUE | -->
<!--|___________________________| -->
<xs:complexType name="Value">
<xs:attribute name="Value" type="xs:double"/>
<xs:attribute name="TimeStamp" type="xs:dateTime"/>
</xs:complexType>
[2] In CDO--> MySQL
mysql> describe generics_Value;
+---------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------+------+-----+---------+-------+
| cdo_id | bigint(20) | NO | PRI | NULL | |
| cdo_version | int(11) | NO | PRI | NULL | |
| cdo_created | bigint(20) | NO | | NULL | |
| cdo_revised | bigint(20) | NO | | NULL | |
| cdo_resource | bigint(20) | NO | | NULL | |
| cdo_container | bigint(20) | NO | | NULL | |
| cdo_feature | int(11) | NO | | NULL | |
| timeStamp0 | longtext | YES | | NULL | |
| value | longtext | YES | | NULL | |
| cdo_set_value | tinyint(1) | YES | | NULL | |
+---------------+------------+------+-----+---------+-------+
|
|
|
Re: [CDO/EMF] Modeling BIG amounts of tiny data [message #1067146 is a reply to message #1066971] |
Sat, 06 July 2013 12:48 |
Erdal Karaca Messages: 854 Registered: July 2009 |
Senior Member |
|
|
I remember someone asking for a NetCDF integration (maybe, store for CDO)... That would be the preferred format for high volume time series data, I guess.
Christophe Bouhier wrote on Fri, 05 July 2013 11:50Hi,
For our application, we are currently modeling a data object [1] , in
CDO this translates to an object which in an MySQL Store for CDO looks
like [2].
As we are talking about 100's of millions of these volatile objects, we
are considering modeling this differently (Size of the DB is the reason,
see the SQL table in [2]).
BTW the operations we perform is
- Writing (single values).
- Reading, with a select on the value Timestamp. (Currently with CDO
SQL query handler).
I am considering two options.
1) Using a custom data type. I am not sure what this means for CDO. I
pressume, the Value object will still be an object with lifecycle etc..
2) using a specific file based store, whereby the 'Value' will be
referred to by a CDO External reference, letting a specific EMF Resource
implementation retrieving the objects from a file.
From the community, I would like to ask if someone has experience with
1) and 2) and what should be considered. For 2) for example I can
imagine, that the query facility would be something to build from
scratch and/or would perhaps require some indexing.
Any advise on this?
Thank You,
Christophe
[1] In .xsd it looks like this:
<!--_____________________________ -->
<!--| VALUE | -->
<!--|___________________________| -->
<xs:complexType name="Value">
<xs:attribute name="Value" type="xs:double"/>
<xs:attribute name="TimeStamp" type="xs:dateTime"/>
</xs:complexType>
[2] In CDO--> MySQL
mysql> describe generics_Value;
+---------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------+------+-----+---------+-------+
| cdo_id | bigint(20) | NO | PRI | NULL | |
| cdo_version | int(11) | NO | PRI | NULL | |
| cdo_created | bigint(20) | NO | | NULL | |
| cdo_revised | bigint(20) | NO | | NULL | |
| cdo_resource | bigint(20) | NO | | NULL | |
| cdo_container | bigint(20) | NO | | NULL | |
| cdo_feature | int(11) | NO | | NULL | |
| timeStamp0 | longtext | YES | | NULL | |
| value | longtext | YES | | NULL | |
| cdo_set_value | tinyint(1) | YES | | NULL | |
+---------------+------------+------+-----+---------+-------+
|
|
|
Re: [CDO/EMF] Modeling BIG amounts of tiny data [message #1067385 is a reply to message #1067146] |
Mon, 08 July 2013 13:23 |
Christophe Bouhier Messages: 937 Registered: July 2009 |
Senior Member |
|
|
On 06-07-13 14:48, Erdal Karaca wrote:
> I remember someone asking for a NetCDF integration (maybe, store for
Thanks Erdal! Will check NetCDF.
> CDO)... That would be the preferred format for high volume time series
> data, I guess.
>
> Christophe Bouhier wrote on Fri, 05 July 2013 11:50
>> Hi,
>>
>> For our application, we are currently modeling a data object [1] , in
>> CDO this translates to an object which in an MySQL Store for CDO looks
>> like [2].
>>
>> As we are talking about 100's of millions of these volatile objects,
>> we are considering modeling this differently (Size of the DB is the
>> reason, see the SQL table in [2]).
>>
>> BTW the operations we perform is
>> - Writing (single values).
>> - Reading, with a select on the value Timestamp. (Currently with CDO
>> SQL query handler).
>>
>> I am considering two options.
>>
>> 1) Using a custom data type. I am not sure what this means for CDO. I
>> pressume, the Value object will still be an object with lifecycle etc..
>>
>> 2) using a specific file based store, whereby the 'Value' will be
>> referred to by a CDO External reference, letting a specific EMF
>> Resource implementation retrieving the objects from a file.
>>
>>
>> From the community, I would like to ask if someone has experience
>> with 1) and 2) and what should be considered. For 2) for example I can
>> imagine, that the query facility would be something to build from
>> scratch and/or would perhaps require some indexing.
>>
>> Any advise on this?
>> Thank You,
>> Christophe
>>
>> [1] In .xsd it looks like this:
>>
>> <!--_____________________________ -->
>> <!--| VALUE | -->
>> <!--|___________________________| -->
>> <xs:complexType name="Value">
>> <xs:attribute name="Value" type="xs:double"/>
>> <xs:attribute name="TimeStamp" type="xs:dateTime"/>
>> </xs:complexType>
>>
>> [2] In CDO--> MySQL
>>
>> mysql> describe generics_Value;
>> +---------------+------------+------+-----+---------+-------+
>> | Field | Type | Null | Key | Default | Extra |
>> +---------------+------------+------+-----+---------+-------+
>> | cdo_id | bigint(20) | NO | PRI | NULL | |
>> | cdo_version | int(11) | NO | PRI | NULL | |
>> | cdo_created | bigint(20) | NO | | NULL | |
>> | cdo_revised | bigint(20) | NO | | NULL | |
>> | cdo_resource | bigint(20) | NO | | NULL | |
>> | cdo_container | bigint(20) | NO | | NULL | |
>> | cdo_feature | int(11) | NO | | NULL | |
>> | timeStamp0 | longtext | YES | | NULL | |
>> | value | longtext | YES | | NULL | |
>> | cdo_set_value | tinyint(1) | YES | | NULL | |
>> +---------------+------------+------+-----+---------+-------+
>
>
|
|
|
Goto Forum:
Current Time: Fri Apr 19 13:25:31 GMT 2024
Powered by FUDForum. Page generated in 0.02400 seconds
|