Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [CDO/EMF] Modeling BIG amounts of tiny data
[CDO/EMF] Modeling BIG amounts of tiny data [message #1066971] Fri, 05 July 2013 09:50 Go to next message
Christophe Bouhier is currently offline Christophe BouhierFriend
Messages: 937
Registered: July 2009
Senior Member
Hi,

For our application, we are currently modeling a data object [1] , in
CDO this translates to an object which in an MySQL Store for CDO looks
like [2].

As we are talking about 100's of millions of these volatile objects, we
are considering modeling this differently (Size of the DB is the reason,
see the SQL table in [2]).

BTW the operations we perform is
- Writing (single values).
- Reading, with a select on the value Timestamp. (Currently with CDO
SQL query handler).

I am considering two options.

1) Using a custom data type. I am not sure what this means for CDO. I
pressume, the Value object will still be an object with lifecycle etc..

2) using a specific file based store, whereby the 'Value' will be
referred to by a CDO External reference, letting a specific EMF Resource
implementation retrieving the objects from a file.


From the community, I would like to ask if someone has experience with
1) and 2) and what should be considered. For 2) for example I can
imagine, that the query facility would be something to build from
scratch and/or would perhaps require some indexing.

Any advise on this?
Thank You,
Christophe

[1] In .xsd it looks like this:

<!--_____________________________ -->
<!--| VALUE | -->
<!--|___________________________| -->
<xs:complexType name="Value">
<xs:attribute name="Value" type="xs:double"/>
<xs:attribute name="TimeStamp" type="xs:dateTime"/>
</xs:complexType>

[2] In CDO--> MySQL

mysql> describe generics_Value;
+---------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------+------+-----+---------+-------+
| cdo_id | bigint(20) | NO | PRI | NULL | |
| cdo_version | int(11) | NO | PRI | NULL | |
| cdo_created | bigint(20) | NO | | NULL | |
| cdo_revised | bigint(20) | NO | | NULL | |
| cdo_resource | bigint(20) | NO | | NULL | |
| cdo_container | bigint(20) | NO | | NULL | |
| cdo_feature | int(11) | NO | | NULL | |
| timeStamp0 | longtext | YES | | NULL | |
| value | longtext | YES | | NULL | |
| cdo_set_value | tinyint(1) | YES | | NULL | |
+---------------+------------+------+-----+---------+-------+
Re: [CDO/EMF] Modeling BIG amounts of tiny data [message #1067146 is a reply to message #1066971] Sat, 06 July 2013 12:48 Go to previous messageGo to next message
Erdal Karaca is currently offline Erdal KaracaFriend
Messages: 854
Registered: July 2009
Senior Member
I remember someone asking for a NetCDF integration (maybe, store for CDO)... That would be the preferred format for high volume time series data, I guess.

Christophe Bouhier wrote on Fri, 05 July 2013 11:50
Hi,

For our application, we are currently modeling a data object [1] , in
CDO this translates to an object which in an MySQL Store for CDO looks
like [2].

As we are talking about 100's of millions of these volatile objects, we
are considering modeling this differently (Size of the DB is the reason,
see the SQL table in [2]).

BTW the operations we perform is
- Writing (single values).
- Reading, with a select on the value Timestamp. (Currently with CDO
SQL query handler).

I am considering two options.

1) Using a custom data type. I am not sure what this means for CDO. I
pressume, the Value object will still be an object with lifecycle etc..

2) using a specific file based store, whereby the 'Value' will be
referred to by a CDO External reference, letting a specific EMF Resource
implementation retrieving the objects from a file.


From the community, I would like to ask if someone has experience with
1) and 2) and what should be considered. For 2) for example I can
imagine, that the query facility would be something to build from
scratch and/or would perhaps require some indexing.

Any advise on this?
Thank You,
Christophe

[1] In .xsd it looks like this:

<!--_____________________________ -->
<!--| VALUE | -->
<!--|___________________________| -->
<xs:complexType name="Value">
<xs:attribute name="Value" type="xs:double"/>
<xs:attribute name="TimeStamp" type="xs:dateTime"/>
</xs:complexType>

[2] In CDO--> MySQL

mysql> describe generics_Value;
+---------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------+------+-----+---------+-------+
| cdo_id | bigint(20) | NO | PRI | NULL | |
| cdo_version | int(11) | NO | PRI | NULL | |
| cdo_created | bigint(20) | NO | | NULL | |
| cdo_revised | bigint(20) | NO | | NULL | |
| cdo_resource | bigint(20) | NO | | NULL | |
| cdo_container | bigint(20) | NO | | NULL | |
| cdo_feature | int(11) | NO | | NULL | |
| timeStamp0 | longtext | YES | | NULL | |
| value | longtext | YES | | NULL | |
| cdo_set_value | tinyint(1) | YES | | NULL | |
+---------------+------------+------+-----+---------+-------+

Re: [CDO/EMF] Modeling BIG amounts of tiny data [message #1067385 is a reply to message #1067146] Mon, 08 July 2013 13:23 Go to previous message
Christophe Bouhier is currently offline Christophe BouhierFriend
Messages: 937
Registered: July 2009
Senior Member
On 06-07-13 14:48, Erdal Karaca wrote:
> I remember someone asking for a NetCDF integration (maybe, store for

Thanks Erdal! Will check NetCDF.

> CDO)... That would be the preferred format for high volume time series
> data, I guess.
>
> Christophe Bouhier wrote on Fri, 05 July 2013 11:50
>> Hi,
>>
>> For our application, we are currently modeling a data object [1] , in
>> CDO this translates to an object which in an MySQL Store for CDO looks
>> like [2].
>>
>> As we are talking about 100's of millions of these volatile objects,
>> we are considering modeling this differently (Size of the DB is the
>> reason, see the SQL table in [2]).
>>
>> BTW the operations we perform is
>> - Writing (single values).
>> - Reading, with a select on the value Timestamp. (Currently with CDO
>> SQL query handler).
>>
>> I am considering two options.
>>
>> 1) Using a custom data type. I am not sure what this means for CDO. I
>> pressume, the Value object will still be an object with lifecycle etc..
>>
>> 2) using a specific file based store, whereby the 'Value' will be
>> referred to by a CDO External reference, letting a specific EMF
>> Resource implementation retrieving the objects from a file.
>>
>>
>> From the community, I would like to ask if someone has experience
>> with 1) and 2) and what should be considered. For 2) for example I can
>> imagine, that the query facility would be something to build from
>> scratch and/or would perhaps require some indexing.
>>
>> Any advise on this?
>> Thank You,
>> Christophe
>>
>> [1] In .xsd it looks like this:
>>
>> <!--_____________________________ -->
>> <!--| VALUE | -->
>> <!--|___________________________| -->
>> <xs:complexType name="Value">
>> <xs:attribute name="Value" type="xs:double"/>
>> <xs:attribute name="TimeStamp" type="xs:dateTime"/>
>> </xs:complexType>
>>
>> [2] In CDO--> MySQL
>>
>> mysql> describe generics_Value;
>> +---------------+------------+------+-----+---------+-------+
>> | Field | Type | Null | Key | Default | Extra |
>> +---------------+------------+------+-----+---------+-------+
>> | cdo_id | bigint(20) | NO | PRI | NULL | |
>> | cdo_version | int(11) | NO | PRI | NULL | |
>> | cdo_created | bigint(20) | NO | | NULL | |
>> | cdo_revised | bigint(20) | NO | | NULL | |
>> | cdo_resource | bigint(20) | NO | | NULL | |
>> | cdo_container | bigint(20) | NO | | NULL | |
>> | cdo_feature | int(11) | NO | | NULL | |
>> | timeStamp0 | longtext | YES | | NULL | |
>> | value | longtext | YES | | NULL | |
>> | cdo_set_value | tinyint(1) | YES | | NULL | |
>> +---------------+------------+------+-----+---------+-------+
>
>
Previous Topic:Howto create a reference to an attribute?
Next Topic:[EMF Store] External Editor
Goto Forum:
  


Current Time: Fri Apr 19 13:25:31 GMT 2024

Powered by FUDForum. Page generated in 0.02400 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top