Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] experience with huge data amount([CDO] experience with huge data amount)
[CDO] experience with huge data amount [message #915984] Tue, 18 September 2012 10:55 Go to next message
peter meyer is currently offline peter meyerFriend
Messages: 13
Registered: September 2012
Junior Member
Hi,

currently i am evaluating different technologies for a new product.

The application domain is "energy producing plants".

On key feature of the product is, that different customers have different businessmodels. So the idea is to model the businessmodel with EMF and persist it via a CDO Server in a relational database.
Despite the pure businessmodel for the plants, there are huge amount of timeseries data which are measured during the runtime of the product.
For example:
there is a businessobject "transformer" which received voltage is measured every minute (-> timeseries measurement-timestamp/value). Both, the transformer and the timeseries data should persisted via a CDO-Server.

So my question is:
Is CDO-Server the right persistenz technologie for appliactions with real big data amounts?
As far as i see for each datarecord persisted via CDO-Server one record in the table CDOObject is inserted. Will this scale for applications with hundrets of million datarecords?


with kind regards
Peter
Re: [CDO] experience with huge data amount [message #916085 is a reply to message #915984] Tue, 18 September 2012 13:54 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 5574
Registered: July 2009
Senior Member
Am 18.09.2012 12:55, schrieb peter meyer:
> Hi,
>
> currently i am evaluating different technologies for a new product.
>
> The application domain is "energy producing plants".
>
> On key feature of the product is, that different customers have different businessmodels. So the idea is to model the
> businessmodel with EMF and persist it via a CDO Server in a relational database.
> Despite the pure businessmodel for the plants, there are huge amount of timeseries data which are measured during the
> runtime of the product. For example:
> there is a businessobject "transformer" which received voltage is measured every minute (-> timeseries
> measurement-timestamp/value). Both, the transformer and the timeseries data should persisted via a CDO-Server.
>
> So my question is:
> Is CDO-Server the right persistenz technologie for appliactions with real big data amounts?
> As far as i see for each datarecord persisted via CDO-Server one record in the table CDOObject is inserted. Will this
> scale for applications with hundrets of million datarecords?
The records themselves map to many different tables?

If the central cdo_objects table is a blocker for you you could try if the HibernateStore is smarter than the DBStore in
this regard. I vaguely remember it embeds the class/table names into the object IDs so that it may not need this central
table to map object IDs to tables.

In general I know of no other model repository for huge models than CDO ;-)


Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] experience with huge data amount [message #916240 is a reply to message #915984] Tue, 18 September 2012 19:42 Go to previous message
Erdal Karaca is currently offline Erdal KaracaFriend
Messages: 756
Registered: July 2009
Senior Member
I have used CDO with "big data sets" (300k+ rows in a single table with 10+ numeric columns)...
I used Oracle DB as backend which had good performance for SELECTs (using sql query handler) out of the box without any tweaking...
Deleting an object took really long when the ensure referential integrity was set to true (querying for cross references).

So, it depends on the backend you will be using.

As Eike has written, the only problem could be the central cdo_objects table which has an entry of each object that is managed by CDO.
Maybe, it is possible to split your time series data by plants, e.g. for each plant provide a repository which manages the objects in a different db/schema?

BTW if you continuously produce time series data at an interval of 1 minute by just one "transformer", then you will have reached 100M+ objects just after 190 years Smile

peter meyer wrote on Tue, 18 September 2012 06:55
Hi,

currently i am evaluating different technologies for a new product.

The application domain is "energy producing plants".

On key feature of the product is, that different customers have different businessmodels. So the idea is to model the businessmodel with EMF and persist it via a CDO Server in a relational database.
Despite the pure businessmodel for the plants, there are huge amount of timeseries data which are measured during the runtime of the product.
For example:
there is a businessobject "transformer" which received voltage is measured every minute (-> timeseries measurement-timestamp/value). Both, the transformer and the timeseries data should persisted via a CDO-Server.

So my question is:
Is CDO-Server the right persistenz technologie for appliactions with real big data amounts?
As far as i see for each datarecord persisted via CDO-Server one record in the table CDOObject is inserted. Will this scale for applications with hundrets of million datarecords?


with kind regards
Peter

Previous Topic:[CDO] Bug in error handling causes misleading LegacyModeNotEnabledException
Next Topic:[ECORE/Teneo] Persistence clarification
Goto Forum:
  


Current Time: Fri Nov 28 03:18:55 GMT 2014

Powered by FUDForum. Page generated in 0.11768 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software