ETL Performance? [message #483582] |
Wed, 02 September 2009 08:31 |
Eclipse User |
|
|
|
Originally posted by: d.clowes.lboro.ac.uk
Hi All,
Does anyone have any suggestions on improving the performance of ETL
scripts.
I am trying to transform the raw XML data into our models. The data is
split across several files totaling approx 40mb. So far the transformation
has being running for over 26hours and is only approx 8mb through. (The
current file is 9mb and has taken 24hours on its own so far.)
It also looks as if Eclipse will not maximise the processor usage. I have
a Dual Core processor but both processors only seem to be running at 50%
whilst it is processing the Transform.
I'm reluctant to stop the process even though it looks like taking a week
as I don't want to havve to restart it if there are no known performance
tweeks.
Thanks for any suggestions,
Darren
|
|
|
|
Re: ETL Performance? [message #483604 is a reply to message #483588] |
Wed, 02 September 2009 09:56 |
Eclipse User |
|
|
|
Originally posted by: d.clowes.lboro.ac.uk
Thanks Dimitris I shall probably give those a try as I know I have a
native object in a deep loop at some point.
With regards to equivalent() I also make heavy use of this. Are you
suggesting rather than call equivalent, I should create the new object
within the parent rule?
i.e. rather than:
for (c in t1.cell){
t2.cells.add(c.equivalent());
}
DO:
for (c in t1.cell){
var x : Cell;
x.text := c.text;
t2.cells.add(x);
}
Thanks Darren
|
|
|
|
Re: ETL Performance? [message #483622 is a reply to message #483605] |
Wed, 02 September 2009 11:31 |
Eclipse User |
|
|
|
Originally posted by: d.clowes.lboro.ac.uk
Thanks Dimitris,
Your suggestion have improved performance significantly. I'm working on a
slower machine at the moment but what took 6hours now takes approx 30mins
on this slower machine. It still looks like it will take a few hours to
complete but that is much better than a few days :D
Darren
|
|
|
|
|
|
|
|
|
|
|
Powered by
FUDForum. Page generated in 0.04611 seconds