Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » BIRT » .rptdocument creation seems to have problems working with large datasets
.rptdocument creation seems to have problems working with large datasets [message #367721] Fri, 03 April 2009 15:55 Go to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
I have a fairly complex report layout that seems to run well (runs and
rendered in a few seconds) when using small datasets of 50 - 150 rows,
although the final rptdocument is about 4MB (renders to 45kb pdf). When
using a dataset of ~6000 rows, however it takes 4-5 minutes to complete,
and the rptdocument is over 100MB (yields 4.5MB pdf). And when
supplying ~18000 rows the session expires after about 40 minutes, and by
that time the rptdocument has reached over 300MB. Any ideas what steps
I can take to find the area(s) that are causing this performance issue?

The problem I am speaking of is strictly run related - once the
rptdocument is created, the rendering performance is quite fast (at
least where the rptdocument has successfully been created).

Upping the heap size hasn't helped for this - max memory allocated
(512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367722 is a reply to message #367721] Fri, 03 April 2009 15:57 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2

Pinny Markowitz wrote:
> I have a fairly complex report layout that seems to run well (runs and
> rendered in a few seconds) when using small datasets of 50 - 150 rows,
> although the final rptdocument is about 4MB (renders to 45kb pdf). When
> using a dataset of ~6000 rows, however it takes 4-5 minutes to complete,
> and the rptdocument is over 100MB (yields 4.5MB pdf). And when
> supplying ~18000 rows the session expires after about 40 minutes, and by
> that time the rptdocument has reached over 300MB. Any ideas what steps
> I can take to find the area(s) that are causing this performance issue?
>
> The problem I am speaking of is strictly run related - once the
> rptdocument is created, the rendering performance is quite fast (at
> least where the rptdocument has successfully been created).
>
> Upping the heap size hasn't helped for this - max memory allocated
> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367730 is a reply to message #367722] Mon, 06 April 2009 13:43 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
I have seen Jason's comment (2/16) about setting memory cache size to
perhaps improve performance for larger datasets - is there a way to do
this as a parameter passed as part of the URL? Also, is there a default
configuration setting for this?

Pinny Markowitz wrote:
> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>
> Pinny Markowitz wrote:
>> I have a fairly complex report layout that seems to run well (runs and
>> rendered in a few seconds) when using small datasets of 50 - 150 rows,
>> although the final rptdocument is about 4MB (renders to 45kb pdf).
>> When using a dataset of ~6000 rows, however it takes 4-5 minutes to
>> complete, and the rptdocument is over 100MB (yields 4.5MB pdf). And
>> when supplying ~18000 rows the session expires after about 40 minutes,
>> and by that time the rptdocument has reached over 300MB. Any ideas
>> what steps I can take to find the area(s) that are causing this
>> performance issue?
>>
>> The problem I am speaking of is strictly run related - once the
>> rptdocument is created, the rendering performance is quite fast (at
>> least where the rptdocument has successfully been created).
>>
>> Upping the heap size hasn't helped for this - max memory allocated
>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367735 is a reply to message #367730] Mon, 06 April 2009 15:35 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: jasonweathersby.alltel.net

Pinny,

Any chance you open a bug for this? Any performance issues we like to
get recorded.

Jason

Pinny Markowitz wrote:
> I have seen Jason's comment (2/16) about setting memory cache size to
> perhaps improve performance for larger datasets - is there a way to do
> this as a parameter passed as part of the URL? Also, is there a default
> configuration setting for this?
>
> Pinny Markowitz wrote:
>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>
>> Pinny Markowitz wrote:
>>> I have a fairly complex report layout that seems to run well (runs
>>> and rendered in a few seconds) when using small datasets of 50 - 150
>>> rows, although the final rptdocument is about 4MB (renders to 45kb
>>> pdf). When using a dataset of ~6000 rows, however it takes 4-5
>>> minutes to complete, and the rptdocument is over 100MB (yields 4.5MB
>>> pdf). And when supplying ~18000 rows the session expires after about
>>> 40 minutes, and by that time the rptdocument has reached over 300MB.
>>> Any ideas what steps I can take to find the area(s) that are causing
>>> this performance issue?
>>>
>>> The problem I am speaking of is strictly run related - once the
>>> rptdocument is created, the rendering performance is quite fast (at
>>> least where the rptdocument has successfully been created).
>>>
>>> Upping the heap size hasn't helped for this - max memory allocated
>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367754 is a reply to message #367735] Mon, 06 April 2009 19:49 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
To do so would require changing the datasource to use an xml (or some
other portable) format, so that this can be reproduced outside of my
environment. I will need to research some method for converting the
data - will keep you posted.


Jason Weathersby wrote:
> Pinny,
>
> Any chance you open a bug for this? Any performance issues we like to
> get recorded.
>
> Jason
>
> Pinny Markowitz wrote:
>> I have seen Jason's comment (2/16) about setting memory cache size to
>> perhaps improve performance for larger datasets - is there a way to do
>> this as a parameter passed as part of the URL? Also, is there a
>> default configuration setting for this?
>>
>> Pinny Markowitz wrote:
>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>
>>> Pinny Markowitz wrote:
>>>> I have a fairly complex report layout that seems to run well (runs
>>>> and rendered in a few seconds) when using small datasets of 50 - 150
>>>> rows, although the final rptdocument is about 4MB (renders to 45kb
>>>> pdf). When using a dataset of ~6000 rows, however it takes 4-5
>>>> minutes to complete, and the rptdocument is over 100MB (yields 4.5MB
>>>> pdf). And when supplying ~18000 rows the session expires after
>>>> about 40 minutes, and by that time the rptdocument has reached over
>>>> 300MB. Any ideas what steps I can take to find the area(s) that are
>>>> causing this performance issue?
>>>>
>>>> The problem I am speaking of is strictly run related - once the
>>>> rptdocument is created, the rendering performance is quite fast (at
>>>> least where the rptdocument has successfully been created).
>>>>
>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367757 is a reply to message #367735] Mon, 06 April 2009 22:04 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
While stepping through the process with the debugger, it seems that the
problem is stemming from the PassManager.doPopulation() method (as part
of ResultSet caching). During this process, the engine seems to be
trying to sort the entire result set and with great difficulty - is
there any way to perhaps turn this off?

Jason Weathersby wrote:
> Pinny,
>
> Any chance you open a bug for this? Any performance issues we like to
> get recorded.
>
> Jason
>
> Pinny Markowitz wrote:
>> I have seen Jason's comment (2/16) about setting memory cache size to
>> perhaps improve performance for larger datasets - is there a way to do
>> this as a parameter passed as part of the URL? Also, is there a
>> default configuration setting for this?
>>
>> Pinny Markowitz wrote:
>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>
>>> Pinny Markowitz wrote:
>>>> I have a fairly complex report layout that seems to run well (runs
>>>> and rendered in a few seconds) when using small datasets of 50 - 150
>>>> rows, although the final rptdocument is about 4MB (renders to 45kb
>>>> pdf). When using a dataset of ~6000 rows, however it takes 4-5
>>>> minutes to complete, and the rptdocument is over 100MB (yields 4.5MB
>>>> pdf). And when supplying ~18000 rows the session expires after
>>>> about 40 minutes, and by that time the rptdocument has reached over
>>>> 300MB. Any ideas what steps I can take to find the area(s) that are
>>>> causing this performance issue?
>>>>
>>>> The problem I am speaking of is strictly run related - once the
>>>> rptdocument is created, the rendering performance is quite fast (at
>>>> least where the rptdocument has successfully been created).
>>>>
>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367769 is a reply to message #367757] Tue, 07 April 2009 15:57 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: jasonweathersby.alltel.net

Pinny,

It does this if the data needs multiple passes do to aggregation or
sorting. Even if you cant post the data it would be good to get a
bugzilla entry started for this to verify performance implications.

Jason

Pinny Markowitz wrote:
> While stepping through the process with the debugger, it seems that the
> problem is stemming from the PassManager.doPopulation() method (as part
> of ResultSet caching). During this process, the engine seems to be
> trying to sort the entire result set and with great difficulty - is
> there any way to perhaps turn this off?
>
> Jason Weathersby wrote:
>> Pinny,
>>
>> Any chance you open a bug for this? Any performance issues we like to
>> get recorded.
>>
>> Jason
>>
>> Pinny Markowitz wrote:
>>> I have seen Jason's comment (2/16) about setting memory cache size to
>>> perhaps improve performance for larger datasets - is there a way to
>>> do this as a parameter passed as part of the URL? Also, is there a
>>> default configuration setting for this?
>>>
>>> Pinny Markowitz wrote:
>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>
>>>> Pinny Markowitz wrote:
>>>>> I have a fairly complex report layout that seems to run well (runs
>>>>> and rendered in a few seconds) when using small datasets of 50 -
>>>>> 150 rows, although the final rptdocument is about 4MB (renders to
>>>>> 45kb pdf). When using a dataset of ~6000 rows, however it takes
>>>>> 4-5 minutes to complete, and the rptdocument is over 100MB (yields
>>>>> 4.5MB pdf). And when supplying ~18000 rows the session expires
>>>>> after about 40 minutes, and by that time the rptdocument has
>>>>> reached over 300MB. Any ideas what steps I can take to find the
>>>>> area(s) that are causing this performance issue?
>>>>>
>>>>> The problem I am speaking of is strictly run related - once the
>>>>> rptdocument is created, the rendering performance is quite fast (at
>>>>> least where the rptdocument has successfully been created).
>>>>>
>>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367777 is a reply to message #367769] Tue, 07 April 2009 18:52 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
It seems correct that the primary problem was indeed the sorting -
removed all sorting from the report design and that was a tremendous
boost for performance.

However, I am finding that using a nested table within a group header
(bound to the containing table's dataset) is a big drain on performance
as well. The rptdocument was created much faster when I moved the
nested contents back to the parent - and it was much smaller in size as
well (under 2 minutes for ~18000 rows, compared to 10+ minutes when
using nested table). Rendering time (as pdf) was affected by the nested
table as well, although not as significantly. When using a simpler
layout, I was able to recreate similar performance differences although
on a smaller scale.

Jason Weathersby wrote:
> Pinny,
>
> It does this if the data needs multiple passes do to aggregation or
> sorting. Even if you cant post the data it would be good to get a
> bugzilla entry started for this to verify performance implications.
>
> Jason
>
> Pinny Markowitz wrote:
>> While stepping through the process with the debugger, it seems that
>> the problem is stemming from the PassManager.doPopulation() method (as
>> part of ResultSet caching). During this process, the engine seems to
>> be trying to sort the entire result set and with great difficulty - is
>> there any way to perhaps turn this off?
>>
>> Jason Weathersby wrote:
>>> Pinny,
>>>
>>> Any chance you open a bug for this? Any performance issues we like
>>> to get recorded.
>>>
>>> Jason
>>>
>>> Pinny Markowitz wrote:
>>>> I have seen Jason's comment (2/16) about setting memory cache size
>>>> to perhaps improve performance for larger datasets - is there a way
>>>> to do this as a parameter passed as part of the URL? Also, is there
>>>> a default configuration setting for this?
>>>>
>>>> Pinny Markowitz wrote:
>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> I have a fairly complex report layout that seems to run well (runs
>>>>>> and rendered in a few seconds) when using small datasets of 50 -
>>>>>> 150 rows, although the final rptdocument is about 4MB (renders to
>>>>>> 45kb pdf). When using a dataset of ~6000 rows, however it takes
>>>>>> 4-5 minutes to complete, and the rptdocument is over 100MB (yields
>>>>>> 4.5MB pdf). And when supplying ~18000 rows the session expires
>>>>>> after about 40 minutes, and by that time the rptdocument has
>>>>>> reached over 300MB. Any ideas what steps I can take to find the
>>>>>> area(s) that are causing this performance issue?
>>>>>>
>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>
>>>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367778 is a reply to message #367777] Tue, 07 April 2009 19:03 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
Submitted as 271512.

Pinny Markowitz wrote:
> It seems correct that the primary problem was indeed the sorting -
> removed all sorting from the report design and that was a tremendous
> boost for performance.
>
> However, I am finding that using a nested table within a group header
> (bound to the containing table's dataset) is a big drain on performance
> as well. The rptdocument was created much faster when I moved the
> nested contents back to the parent - and it was much smaller in size as
> well (under 2 minutes for ~18000 rows, compared to 10+ minutes when
> using nested table). Rendering time (as pdf) was affected by the nested
> table as well, although not as significantly. When using a simpler
> layout, I was able to recreate similar performance differences although
> on a smaller scale.
>
> Jason Weathersby wrote:
>> Pinny,
>>
>> It does this if the data needs multiple passes do to aggregation or
>> sorting. Even if you cant post the data it would be good to get a
>> bugzilla entry started for this to verify performance implications.
>>
>> Jason
>>
>> Pinny Markowitz wrote:
>>> While stepping through the process with the debugger, it seems that
>>> the problem is stemming from the PassManager.doPopulation() method
>>> (as part of ResultSet caching). During this process, the engine
>>> seems to be trying to sort the entire result set and with great
>>> difficulty - is there any way to perhaps turn this off?
>>>
>>> Jason Weathersby wrote:
>>>> Pinny,
>>>>
>>>> Any chance you open a bug for this? Any performance issues we like
>>>> to get recorded.
>>>>
>>>> Jason
>>>>
>>>> Pinny Markowitz wrote:
>>>>> I have seen Jason's comment (2/16) about setting memory cache size
>>>>> to perhaps improve performance for larger datasets - is there a way
>>>>> to do this as a parameter passed as part of the URL? Also, is
>>>>> there a default configuration setting for this?
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>> (runs and rendered in a few seconds) when using small datasets of
>>>>>>> 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument is
>>>>>>> over 100MB (yields 4.5MB pdf). And when supplying ~18000 rows
>>>>>>> the session expires after about 40 minutes, and by that time the
>>>>>>> rptdocument has reached over 300MB. Any ideas what steps I can
>>>>>>> take to find the area(s) that are causing this performance issue?
>>>>>>>
>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>>
>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>> allocated (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367808 is a reply to message #367778] Thu, 09 April 2009 02:37 Go to previous messageGo to next message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Hi Pinny,

How many groups do you have after the report is rendered? For each group
BIRT will make a query execution against database to fetch nested table
data.

It is always a good practice that avoid nested table in report design.

It will help the performance tuning if you can post your report design.

Thanks.
Lin
"Pinny Markowitz" <pinny@medwiztech.com> wrote in message
news:grg82b$6be$1@build.eclipse.org...
> Submitted as 271512.
>
> Pinny Markowitz wrote:
>> It seems correct that the primary problem was indeed the sorting -
>> removed all sorting from the report design and that was a tremendous
>> boost for performance.
>>
>> However, I am finding that using a nested table within a group header
>> (bound to the containing table's dataset) is a big drain on performance
>> as well. The rptdocument was created much faster when I moved the nested
>> contents back to the parent - and it was much smaller in size as well
>> (under 2 minutes for ~18000 rows, compared to 10+ minutes when using
>> nested table). Rendering time (as pdf) was affected by the nested table
>> as well, although not as significantly. When using a simpler layout, I
>> was able to recreate similar performance differences although on a
>> smaller scale.
>>
>> Jason Weathersby wrote:
>>> Pinny,
>>>
>>> It does this if the data needs multiple passes do to aggregation or
>>> sorting. Even if you cant post the data it would be good to get a
>>> bugzilla entry started for this to verify performance implications.
>>>
>>> Jason
>>>
>>> Pinny Markowitz wrote:
>>>> While stepping through the process with the debugger, it seems that the
>>>> problem is stemming from the PassManager.doPopulation() method (as part
>>>> of ResultSet caching). During this process, the engine seems to be
>>>> trying to sort the entire result set and with great difficulty - is
>>>> there any way to perhaps turn this off?
>>>>
>>>> Jason Weathersby wrote:
>>>>> Pinny,
>>>>>
>>>>> Any chance you open a bug for this? Any performance issues we like to
>>>>> get recorded.
>>>>>
>>>>> Jason
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> I have seen Jason's comment (2/16) about setting memory cache size to
>>>>>> perhaps improve performance for larger datasets - is there a way to
>>>>>> do this as a parameter passed as part of the URL? Also, is there a
>>>>>> default configuration setting for this?
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> I have a fairly complex report layout that seems to run well (runs
>>>>>>>> and rendered in a few seconds) when using small datasets of 50 -
>>>>>>>> 150 rows, although the final rptdocument is about 4MB (renders to
>>>>>>>> 45kb pdf). When using a dataset of ~6000 rows, however it takes
>>>>>>>> 4-5 minutes to complete, and the rptdocument is over 100MB (yields
>>>>>>>> 4.5MB pdf). And when supplying ~18000 rows the session expires
>>>>>>>> after about 40 minutes, and by that time the rptdocument has
>>>>>>>> reached over 300MB. Any ideas what steps I can take to find the
>>>>>>>> area(s) that are causing this performance issue?
>>>>>>>>
>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>> rptdocument is created, the rendering performance is quite fast (at
>>>>>>>> least where the rptdocument has successfully been created).
>>>>>>>>
>>>>>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>>>>>> (512MB) is far more than what is being utilized (~350MB).
Re: .rptdocument creation seems to have problems working with large datasets [message #367809 is a reply to message #367808] Thu, 09 April 2009 03:07 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: jasonweathersby.alltel.net

Lin,

You mean many nested tables as each additional one affects performance,
correct?

Jason

lzhu wrote:
> Hi Pinny,
>
> How many groups do you have after the report is rendered? For each group
> BIRT will make a query execution against database to fetch nested table
> data.
>
> It is always a good practice that avoid nested table in report design.
>
> It will help the performance tuning if you can post your report design.
>
> Thanks.
> Lin
> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
> news:grg82b$6be$1@build.eclipse.org...
>> Submitted as 271512.
>>
>> Pinny Markowitz wrote:
>>> It seems correct that the primary problem was indeed the sorting -
>>> removed all sorting from the report design and that was a tremendous
>>> boost for performance.
>>>
>>> However, I am finding that using a nested table within a group header
>>> (bound to the containing table's dataset) is a big drain on performance
>>> as well. The rptdocument was created much faster when I moved the nested
>>> contents back to the parent - and it was much smaller in size as well
>>> (under 2 minutes for ~18000 rows, compared to 10+ minutes when using
>>> nested table). Rendering time (as pdf) was affected by the nested table
>>> as well, although not as significantly. When using a simpler layout, I
>>> was able to recreate similar performance differences although on a
>>> smaller scale.
>>>
>>> Jason Weathersby wrote:
>>>> Pinny,
>>>>
>>>> It does this if the data needs multiple passes do to aggregation or
>>>> sorting. Even if you cant post the data it would be good to get a
>>>> bugzilla entry started for this to verify performance implications.
>>>>
>>>> Jason
>>>>
>>>> Pinny Markowitz wrote:
>>>>> While stepping through the process with the debugger, it seems that the
>>>>> problem is stemming from the PassManager.doPopulation() method (as part
>>>>> of ResultSet caching). During this process, the engine seems to be
>>>>> trying to sort the entire result set and with great difficulty - is
>>>>> there any way to perhaps turn this off?
>>>>>
>>>>> Jason Weathersby wrote:
>>>>>> Pinny,
>>>>>>
>>>>>> Any chance you open a bug for this? Any performance issues we like to
>>>>>> get recorded.
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> I have seen Jason's comment (2/16) about setting memory cache size to
>>>>>>> perhaps improve performance for larger datasets - is there a way to
>>>>>>> do this as a parameter passed as part of the URL? Also, is there a
>>>>>>> default configuration setting for this?
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> I have a fairly complex report layout that seems to run well (runs
>>>>>>>>> and rendered in a few seconds) when using small datasets of 50 -
>>>>>>>>> 150 rows, although the final rptdocument is about 4MB (renders to
>>>>>>>>> 45kb pdf). When using a dataset of ~6000 rows, however it takes
>>>>>>>>> 4-5 minutes to complete, and the rptdocument is over 100MB (yields
>>>>>>>>> 4.5MB pdf). And when supplying ~18000 rows the session expires
>>>>>>>>> after about 40 minutes, and by that time the rptdocument has
>>>>>>>>> reached over 300MB. Any ideas what steps I can take to find the
>>>>>>>>> area(s) that are causing this performance issue?
>>>>>>>>>
>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>> rptdocument is created, the rendering performance is quite fast (at
>>>>>>>>> least where the rptdocument has successfully been created).
>>>>>>>>>
>>>>>>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>>>>>>> (512MB) is far more than what is being utilized (~350MB).
>
>
Re: .rptdocument creation seems to have problems working with large datasets [message #367844 is a reply to message #367809] Mon, 13 April 2009 07:01 Go to previous messageGo to next message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Jason,

Correct. For each nest table execution we will visit DB once. If a nested
table is put into group head, and the group have 100 instances, then for
acqire nested table data we've to query DB for 100 times. These are
expensive operations.

Thanks.
Lin
"Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
news:grjonm$mps$1@build.eclipse.org...
> Lin,
>
> You mean many nested tables as each additional one affects performance,
> correct?
>
> Jason
>
> lzhu wrote:
>> Hi Pinny,
>>
>> How many groups do you have after the report is rendered? For each group
>> BIRT will make a query execution against database to fetch nested table
>> data.
>>
>> It is always a good practice that avoid nested table in report design.
>>
>> It will help the performance tuning if you can post your report design.
>>
>> Thanks.
>> Lin
>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>> news:grg82b$6be$1@build.eclipse.org...
>>> Submitted as 271512.
>>>
>>> Pinny Markowitz wrote:
>>>> It seems correct that the primary problem was indeed the sorting -
>>>> removed all sorting from the report design and that was a tremendous
>>>> boost for performance.
>>>>
>>>> However, I am finding that using a nested table within a group header
>>>> (bound to the containing table's dataset) is a big drain on performance
>>>> as well. The rptdocument was created much faster when I moved the
>>>> nested contents back to the parent - and it was much smaller in size as
>>>> well (under 2 minutes for ~18000 rows, compared to 10+ minutes when
>>>> using nested table). Rendering time (as pdf) was affected by the
>>>> nested table as well, although not as significantly. When using a
>>>> simpler layout, I was able to recreate similar performance differences
>>>> although on a smaller scale.
>>>>
>>>> Jason Weathersby wrote:
>>>>> Pinny,
>>>>>
>>>>> It does this if the data needs multiple passes do to aggregation or
>>>>> sorting. Even if you cant post the data it would be good to get a
>>>>> bugzilla entry started for this to verify performance implications.
>>>>>
>>>>> Jason
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> While stepping through the process with the debugger, it seems that
>>>>>> the problem is stemming from the PassManager.doPopulation() method
>>>>>> (as part of ResultSet caching). During this process, the engine
>>>>>> seems to be trying to sort the entire result set and with great
>>>>>> difficulty - is there any way to perhaps turn this off?
>>>>>>
>>>>>> Jason Weathersby wrote:
>>>>>>> Pinny,
>>>>>>>
>>>>>>> Any chance you open a bug for this? Any performance issues we like
>>>>>>> to get recorded.
>>>>>>>
>>>>>>> Jason
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache size
>>>>>>>> to perhaps improve performance for larger datasets - is there a way
>>>>>>>> to do this as a parameter passed as part of the URL? Also, is
>>>>>>>> there a default configuration setting for this?
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>> (runs and rendered in a few seconds) when using small datasets of
>>>>>>>>>> 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument is
>>>>>>>>>> over 100MB (yields 4.5MB pdf). And when supplying ~18000 rows
>>>>>>>>>> the session expires after about 40 minutes, and by that time the
>>>>>>>>>> rptdocument has reached over 300MB. Any ideas what steps I can
>>>>>>>>>> take to find the area(s) that are causing this performance issue?
>>>>>>>>>>
>>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>>>>>
>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>> (~350MB).
>>
Re: .rptdocument creation seems to have problems working with large datasets [message #367845 is a reply to message #367844] Mon, 13 April 2009 14:14 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
Is this true even when the nested table is bound directly to the
containing parent table instead of a standard dataset? I needed to use
a nested table to achieve the vertical spanning effect as described here:

http://www.eclipse.org/newsportal/article.php?id=33244&g roup=eclipse.birt#33244

Is there any intention to fix this in the near future?

lzhu wrote:
> Jason,
>
> Correct. For each nest table execution we will visit DB once. If a nested
> table is put into group head, and the group have 100 instances, then for
> acqire nested table data we've to query DB for 100 times. These are
> expensive operations.
>
> Thanks.
> Lin
> "Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
> news:grjonm$mps$1@build.eclipse.org...
>> Lin,
>>
>> You mean many nested tables as each additional one affects performance,
>> correct?
>>
>> Jason
>>
>> lzhu wrote:
>>> Hi Pinny,
>>>
>>> How many groups do you have after the report is rendered? For each group
>>> BIRT will make a query execution against database to fetch nested table
>>> data.
>>>
>>> It is always a good practice that avoid nested table in report design.
>>>
>>> It will help the performance tuning if you can post your report design.
>>>
>>> Thanks.
>>> Lin
>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>> news:grg82b$6be$1@build.eclipse.org...
>>>> Submitted as 271512.
>>>>
>>>> Pinny Markowitz wrote:
>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>> removed all sorting from the report design and that was a tremendous
>>>>> boost for performance.
>>>>>
>>>>> However, I am finding that using a nested table within a group header
>>>>> (bound to the containing table's dataset) is a big drain on performance
>>>>> as well. The rptdocument was created much faster when I moved the
>>>>> nested contents back to the parent - and it was much smaller in size as
>>>>> well (under 2 minutes for ~18000 rows, compared to 10+ minutes when
>>>>> using nested table). Rendering time (as pdf) was affected by the
>>>>> nested table as well, although not as significantly. When using a
>>>>> simpler layout, I was able to recreate similar performance differences
>>>>> although on a smaller scale.
>>>>>
>>>>> Jason Weathersby wrote:
>>>>>> Pinny,
>>>>>>
>>>>>> It does this if the data needs multiple passes do to aggregation or
>>>>>> sorting. Even if you cant post the data it would be good to get a
>>>>>> bugzilla entry started for this to verify performance implications.
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> While stepping through the process with the debugger, it seems that
>>>>>>> the problem is stemming from the PassManager.doPopulation() method
>>>>>>> (as part of ResultSet caching). During this process, the engine
>>>>>>> seems to be trying to sort the entire result set and with great
>>>>>>> difficulty - is there any way to perhaps turn this off?
>>>>>>>
>>>>>>> Jason Weathersby wrote:
>>>>>>>> Pinny,
>>>>>>>>
>>>>>>>> Any chance you open a bug for this? Any performance issues we like
>>>>>>>> to get recorded.
>>>>>>>>
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache size
>>>>>>>>> to perhaps improve performance for larger datasets - is there a way
>>>>>>>>> to do this as a parameter passed as part of the URL? Also, is
>>>>>>>>> there a default configuration setting for this?
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>
>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>> (runs and rendered in a few seconds) when using small datasets of
>>>>>>>>>>> 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument is
>>>>>>>>>>> over 100MB (yields 4.5MB pdf). And when supplying ~18000 rows
>>>>>>>>>>> the session expires after about 40 minutes, and by that time the
>>>>>>>>>>> rptdocument has reached over 300MB. Any ideas what steps I can
>>>>>>>>>>> take to find the area(s) that are causing this performance issue?
>>>>>>>>>>>
>>>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>>>>>>
>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>> (~350MB).
>
Re: .rptdocument creation seems to have problems working with large datasets [message #367846 is a reply to message #367808] Mon, 13 April 2009 14:22 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
I have trimmed down the report so that I now have 3 groups in the parent
table, and no groups in the nested table. Performance has improved
somewhat, however the difference between using the nested table and not
is still quite extreme.

I was under the impression that the current release version (v2.3.2) was
caching the dataset so that query execution would only occur once. In
addition, the nested table correctly only renders detail rows belonging
to the parent group - without any explicit filtering. This seems to
indicate that no dataset lookup is being done - rather the data is
fetched from the resultset of the parent group itself. Is this not how
it is working?

lzhu wrote:
> Hi Pinny,
>
> How many groups do you have after the report is rendered? For each group
> BIRT will make a query execution against database to fetch nested table
> data.
>
> It is always a good practice that avoid nested table in report design.
>
> It will help the performance tuning if you can post your report design.
>
> Thanks.
> Lin
> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
> news:grg82b$6be$1@build.eclipse.org...
>> Submitted as 271512.
>>
>> Pinny Markowitz wrote:
>>> It seems correct that the primary problem was indeed the sorting -
>>> removed all sorting from the report design and that was a tremendous
>>> boost for performance.
>>>
>>> However, I am finding that using a nested table within a group header
>>> (bound to the containing table's dataset) is a big drain on performance
>>> as well. The rptdocument was created much faster when I moved the nested
>>> contents back to the parent - and it was much smaller in size as well
>>> (under 2 minutes for ~18000 rows, compared to 10+ minutes when using
>>> nested table). Rendering time (as pdf) was affected by the nested table
>>> as well, although not as significantly. When using a simpler layout, I
>>> was able to recreate similar performance differences although on a
>>> smaller scale.
>>>
>>> Jason Weathersby wrote:
>>>> Pinny,
>>>>
>>>> It does this if the data needs multiple passes do to aggregation or
>>>> sorting. Even if you cant post the data it would be good to get a
>>>> bugzilla entry started for this to verify performance implications.
>>>>
>>>> Jason
>>>>
>>>> Pinny Markowitz wrote:
>>>>> While stepping through the process with the debugger, it seems that the
>>>>> problem is stemming from the PassManager.doPopulation() method (as part
>>>>> of ResultSet caching). During this process, the engine seems to be
>>>>> trying to sort the entire result set and with great difficulty - is
>>>>> there any way to perhaps turn this off?
>>>>>
>>>>> Jason Weathersby wrote:
>>>>>> Pinny,
>>>>>>
>>>>>> Any chance you open a bug for this? Any performance issues we like to
>>>>>> get recorded.
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> I have seen Jason's comment (2/16) about setting memory cache size to
>>>>>>> perhaps improve performance for larger datasets - is there a way to
>>>>>>> do this as a parameter passed as part of the URL? Also, is there a
>>>>>>> default configuration setting for this?
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> I have a fairly complex report layout that seems to run well (runs
>>>>>>>>> and rendered in a few seconds) when using small datasets of 50 -
>>>>>>>>> 150 rows, although the final rptdocument is about 4MB (renders to
>>>>>>>>> 45kb pdf). When using a dataset of ~6000 rows, however it takes
>>>>>>>>> 4-5 minutes to complete, and the rptdocument is over 100MB (yields
>>>>>>>>> 4.5MB pdf). And when supplying ~18000 rows the session expires
>>>>>>>>> after about 40 minutes, and by that time the rptdocument has
>>>>>>>>> reached over 300MB. Any ideas what steps I can take to find the
>>>>>>>>> area(s) that are causing this performance issue?
>>>>>>>>>
>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>> rptdocument is created, the rendering performance is quite fast (at
>>>>>>>>> least where the rptdocument has successfully been created).
>>>>>>>>>
>>>>>>>>> Upping the heap size hasn't helped for this - max memory allocated
>>>>>>>>> (512MB) is far more than what is being utilized (~350MB).
>
>
Re: .rptdocument creation seems to have problems working with large datasets [message #367848 is a reply to message #367846] Mon, 13 April 2009 16:15 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: jasonweathersby.alltel.net

Pinny,

If it is the same dataset it should be cached, but if you have
parameters in the query it will not be the same dataset.

Jason

Pinny Markowitz wrote:
> I have trimmed down the report so that I now have 3 groups in the parent
> table, and no groups in the nested table. Performance has improved
> somewhat, however the difference between using the nested table and not
> is still quite extreme.
>
> I was under the impression that the current release version (v2.3.2) was
> caching the dataset so that query execution would only occur once. In
> addition, the nested table correctly only renders detail rows belonging
> to the parent group - without any explicit filtering. This seems to
> indicate that no dataset lookup is being done - rather the data is
> fetched from the resultset of the parent group itself. Is this not how
> it is working?
>
> lzhu wrote:
>> Hi Pinny,
>>
>> How many groups do you have after the report is rendered? For each
>> group BIRT will make a query execution against database to fetch
>> nested table data.
>>
>> It is always a good practice that avoid nested table in report design.
>>
>> It will help the performance tuning if you can post your report design.
>>
>> Thanks.
>> Lin
>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>> news:grg82b$6be$1@build.eclipse.org...
>>> Submitted as 271512.
>>>
>>> Pinny Markowitz wrote:
>>>> It seems correct that the primary problem was indeed the sorting -
>>>> removed all sorting from the report design and that was a tremendous
>>>> boost for performance.
>>>>
>>>> However, I am finding that using a nested table within a group
>>>> header (bound to the containing table's dataset) is a big drain on
>>>> performance as well. The rptdocument was created much faster when I
>>>> moved the nested contents back to the parent - and it was much
>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared
>>>> to 10+ minutes when using nested table). Rendering time (as pdf)
>>>> was affected by the nested table as well, although not as
>>>> significantly. When using a simpler layout, I was able to recreate
>>>> similar performance differences although on a smaller scale.
>>>>
>>>> Jason Weathersby wrote:
>>>>> Pinny,
>>>>>
>>>>> It does this if the data needs multiple passes do to aggregation or
>>>>> sorting. Even if you cant post the data it would be good to get a
>>>>> bugzilla entry started for this to verify performance implications.
>>>>>
>>>>> Jason
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> While stepping through the process with the debugger, it seems
>>>>>> that the problem is stemming from the PassManager.doPopulation()
>>>>>> method (as part of ResultSet caching). During this process, the
>>>>>> engine seems to be trying to sort the entire result set and with
>>>>>> great difficulty - is there any way to perhaps turn this off?
>>>>>>
>>>>>> Jason Weathersby wrote:
>>>>>>> Pinny,
>>>>>>>
>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>> like to get recorded.
>>>>>>>
>>>>>>> Jason
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>> there a way to do this as a parameter passed as part of the
>>>>>>>> URL? Also, is there a default configuration setting for this?
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>> (runs and rendered in a few seconds) when using small datasets
>>>>>>>>>> of 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument
>>>>>>>>>> is over 100MB (yields 4.5MB pdf). And when supplying ~18000
>>>>>>>>>> rows the session expires after about 40 minutes, and by that
>>>>>>>>>> time the rptdocument has reached over 300MB. Any ideas what
>>>>>>>>>> steps I can take to find the area(s) that are causing this
>>>>>>>>>> performance issue?
>>>>>>>>>>
>>>>>>>>>> The problem I am speaking of is strictly run related - once
>>>>>>>>>> the rptdocument is created, the rendering performance is quite
>>>>>>>>>> fast (at least where the rptdocument has successfully been
>>>>>>>>>> created).
>>>>>>>>>>
>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>> (~350MB).
>>
>>
Re: .rptdocument creation seems to have problems working with large datasets [message #367851 is a reply to message #367848] Mon, 13 April 2009 19:02 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
The parameters being passed to the query in my case are defined as the
parameters that are set through the URL - once the report initializes
these parameters should never change, and neither should the query or
the dataset.

In any case, I have monitored the processes and I am certain that the
the database is being hit only once - CPU utilization begins with the
database process for a short time, and then is taken over by the J2EE
container for the remainder of the generation time.

I hope to have a flat data file ready soon so this can be
replicated/examined in other environments.


Jason Weathersby wrote:
> Pinny,
>
> If it is the same dataset it should be cached, but if you have
> parameters in the query it will not be the same dataset.
>
> Jason
>
> Pinny Markowitz wrote:
>> I have trimmed down the report so that I now have 3 groups in the
>> parent table, and no groups in the nested table. Performance has
>> improved somewhat, however the difference between using the nested
>> table and not is still quite extreme.
>>
>> I was under the impression that the current release version (v2.3.2)
>> was caching the dataset so that query execution would only occur
>> once. In addition, the nested table correctly only renders detail
>> rows belonging to the parent group - without any explicit filtering.
>> This seems to indicate that no dataset lookup is being done - rather
>> the data is fetched from the resultset of the parent group itself. Is
>> this not how it is working?
>>
>> lzhu wrote:
>>> Hi Pinny,
>>>
>>> How many groups do you have after the report is rendered? For each
>>> group BIRT will make a query execution against database to fetch
>>> nested table data.
>>>
>>> It is always a good practice that avoid nested table in report design.
>>>
>>> It will help the performance tuning if you can post your report design.
>>>
>>> Thanks.
>>> Lin
>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>> news:grg82b$6be$1@build.eclipse.org...
>>>> Submitted as 271512.
>>>>
>>>> Pinny Markowitz wrote:
>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>> removed all sorting from the report design and that was a
>>>>> tremendous boost for performance.
>>>>>
>>>>> However, I am finding that using a nested table within a group
>>>>> header (bound to the containing table's dataset) is a big drain on
>>>>> performance as well. The rptdocument was created much faster when
>>>>> I moved the nested contents back to the parent - and it was much
>>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared
>>>>> to 10+ minutes when using nested table). Rendering time (as pdf)
>>>>> was affected by the nested table as well, although not as
>>>>> significantly. When using a simpler layout, I was able to recreate
>>>>> similar performance differences although on a smaller scale.
>>>>>
>>>>> Jason Weathersby wrote:
>>>>>> Pinny,
>>>>>>
>>>>>> It does this if the data needs multiple passes do to aggregation
>>>>>> or sorting. Even if you cant post the data it would be good to
>>>>>> get a bugzilla entry started for this to verify performance
>>>>>> implications.
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> While stepping through the process with the debugger, it seems
>>>>>>> that the problem is stemming from the PassManager.doPopulation()
>>>>>>> method (as part of ResultSet caching). During this process, the
>>>>>>> engine seems to be trying to sort the entire result set and with
>>>>>>> great difficulty - is there any way to perhaps turn this off?
>>>>>>>
>>>>>>> Jason Weathersby wrote:
>>>>>>>> Pinny,
>>>>>>>>
>>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>>> like to get recorded.
>>>>>>>>
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>>> there a way to do this as a parameter passed as part of the
>>>>>>>>> URL? Also, is there a default configuration setting for this?
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>
>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>> (runs and rendered in a few seconds) when using small
>>>>>>>>>>> datasets of 50 - 150 rows, although the final rptdocument is
>>>>>>>>>>> about 4MB (renders to 45kb pdf). When using a dataset of
>>>>>>>>>>> ~6000 rows, however it takes 4-5 minutes to complete, and the
>>>>>>>>>>> rptdocument is over 100MB (yields 4.5MB pdf). And when
>>>>>>>>>>> supplying ~18000 rows the session expires after about 40
>>>>>>>>>>> minutes, and by that time the rptdocument has reached over
>>>>>>>>>>> 300MB. Any ideas what steps I can take to find the area(s)
>>>>>>>>>>> that are causing this performance issue?
>>>>>>>>>>>
>>>>>>>>>>> The problem I am speaking of is strictly run related - once
>>>>>>>>>>> the rptdocument is created, the rendering performance is
>>>>>>>>>>> quite fast (at least where the rptdocument has successfully
>>>>>>>>>>> been created).
>>>>>>>>>>>
>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>> (~350MB).
>>>
>>>
Re: .rptdocument creation seems to have problems working with large datasets [message #367855 is a reply to message #367845] Tue, 14 April 2009 02:04 Go to previous messageGo to next message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Please note that in BIRT 2.3.2 the binding of a nested table to its parent
report item is not supported any more. So for nested table it can only be
bound to a data set.

Thanks.
Lin
"Pinny Markowitz" <pinny@medwiztech.com> wrote in message
news:grvhbu$lbi$1@build.eclipse.org...
> Is this true even when the nested table is bound directly to the
> containing parent table instead of a standard dataset? I needed to use a
> nested table to achieve the vertical spanning effect as described here:
>
> http://www.eclipse.org/newsportal/article.php?id=33244&g roup=eclipse.birt#33244
>
> Is there any intention to fix this in the near future?
>
> lzhu wrote:
>> Jason,
>>
>> Correct. For each nest table execution we will visit DB once. If a nested
>> table is put into group head, and the group have 100 instances, then for
>> acqire nested table data we've to query DB for 100 times. These are
>> expensive operations.
>>
>> Thanks.
>> Lin
>> "Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
>> news:grjonm$mps$1@build.eclipse.org...
>>> Lin,
>>>
>>> You mean many nested tables as each additional one affects performance,
>>> correct?
>>>
>>> Jason
>>>
>>> lzhu wrote:
>>>> Hi Pinny,
>>>>
>>>> How many groups do you have after the report is rendered? For each
>>>> group BIRT will make a query execution against database to fetch nested
>>>> table data.
>>>>
>>>> It is always a good practice that avoid nested table in report design.
>>>>
>>>> It will help the performance tuning if you can post your report design.
>>>>
>>>> Thanks.
>>>> Lin
>>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>>> news:grg82b$6be$1@build.eclipse.org...
>>>>> Submitted as 271512.
>>>>>
>>>>> Pinny Markowitz wrote:
>>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>>> removed all sorting from the report design and that was a tremendous
>>>>>> boost for performance.
>>>>>>
>>>>>> However, I am finding that using a nested table within a group header
>>>>>> (bound to the containing table's dataset) is a big drain on
>>>>>> performance as well. The rptdocument was created much faster when I
>>>>>> moved the nested contents back to the parent - and it was much
>>>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared to
>>>>>> 10+ minutes when using nested table). Rendering time (as pdf) was
>>>>>> affected by the nested table as well, although not as significantly.
>>>>>> When using a simpler layout, I was able to recreate similar
>>>>>> performance differences although on a smaller scale.
>>>>>>
>>>>>> Jason Weathersby wrote:
>>>>>>> Pinny,
>>>>>>>
>>>>>>> It does this if the data needs multiple passes do to aggregation or
>>>>>>> sorting. Even if you cant post the data it would be good to get a
>>>>>>> bugzilla entry started for this to verify performance implications.
>>>>>>>
>>>>>>> Jason
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> While stepping through the process with the debugger, it seems that
>>>>>>>> the problem is stemming from the PassManager.doPopulation() method
>>>>>>>> (as part of ResultSet caching). During this process, the engine
>>>>>>>> seems to be trying to sort the entire result set and with great
>>>>>>>> difficulty - is there any way to perhaps turn this off?
>>>>>>>>
>>>>>>>> Jason Weathersby wrote:
>>>>>>>>> Pinny,
>>>>>>>>>
>>>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>>>> like to get recorded.
>>>>>>>>>
>>>>>>>>> Jason
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>>>> there a way to do this as a parameter passed as part of the URL?
>>>>>>>>>> Also, is there a default configuration setting for this?
>>>>>>>>>>
>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>>
>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>>> (runs and rendered in a few seconds) when using small datasets
>>>>>>>>>>>> of 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument
>>>>>>>>>>>> is over 100MB (yields 4.5MB pdf). And when supplying ~18000
>>>>>>>>>>>> rows the session expires after about 40 minutes, and by that
>>>>>>>>>>>> time the rptdocument has reached over 300MB. Any ideas what
>>>>>>>>>>>> steps I can take to find the area(s) that are causing this
>>>>>>>>>>>> performance issue?
>>>>>>>>>>>>
>>>>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>>>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>>>>>>>
>>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>>> (~350MB).
>>
Re: .rptdocument creation seems to have problems working with large datasets [message #367866 is a reply to message #367855] Tue, 14 April 2009 15:42 Go to previous messageGo to next message
Pinny Markowitz is currently offline Pinny MarkowitzFriend
Messages: 28
Registered: July 2009
Junior Member
In the 'Edit Data Binding...' dialog of the nested table, I am seeing
'Data Set' radio button checked with 'Container's Data Set' selected.
Does this actually fetch the same dataset by running the same query
multiple times? If so, I would question why that is necessary. If not,
I am wondering why performance is being hurt.

I am using 2.3.2 as mentioned above.

lzhu wrote:
> Please note that in BIRT 2.3.2 the binding of a nested table to its parent
> report item is not supported any more. So for nested table it can only be
> bound to a data set.
>
> Thanks.
> Lin
> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
> news:grvhbu$lbi$1@build.eclipse.org...
>> Is this true even when the nested table is bound directly to the
>> containing parent table instead of a standard dataset? I needed to use a
>> nested table to achieve the vertical spanning effect as described here:
>>
>> http://www.eclipse.org/newsportal/article.php?id=33244&g roup=eclipse.birt#33244
>>
>> Is there any intention to fix this in the near future?
>>
>> lzhu wrote:
>>> Jason,
>>>
>>> Correct. For each nest table execution we will visit DB once. If a nested
>>> table is put into group head, and the group have 100 instances, then for
>>> acqire nested table data we've to query DB for 100 times. These are
>>> expensive operations.
>>>
>>> Thanks.
>>> Lin
>>> "Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
>>> news:grjonm$mps$1@build.eclipse.org...
>>>> Lin,
>>>>
>>>> You mean many nested tables as each additional one affects performance,
>>>> correct?
>>>>
>>>> Jason
>>>>
>>>> lzhu wrote:
>>>>> Hi Pinny,
>>>>>
>>>>> How many groups do you have after the report is rendered? For each
>>>>> group BIRT will make a query execution against database to fetch nested
>>>>> table data.
>>>>>
>>>>> It is always a good practice that avoid nested table in report design.
>>>>>
>>>>> It will help the performance tuning if you can post your report design.
>>>>>
>>>>> Thanks.
>>>>> Lin
>>>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>>>> news:grg82b$6be$1@build.eclipse.org...
>>>>>> Submitted as 271512.
>>>>>>
>>>>>> Pinny Markowitz wrote:
>>>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>>>> removed all sorting from the report design and that was a tremendous
>>>>>>> boost for performance.
>>>>>>>
>>>>>>> However, I am finding that using a nested table within a group header
>>>>>>> (bound to the containing table's dataset) is a big drain on
>>>>>>> performance as well. The rptdocument was created much faster when I
>>>>>>> moved the nested contents back to the parent - and it was much
>>>>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared to
>>>>>>> 10+ minutes when using nested table). Rendering time (as pdf) was
>>>>>>> affected by the nested table as well, although not as significantly.
>>>>>>> When using a simpler layout, I was able to recreate similar
>>>>>>> performance differences although on a smaller scale.
>>>>>>>
>>>>>>> Jason Weathersby wrote:
>>>>>>>> Pinny,
>>>>>>>>
>>>>>>>> It does this if the data needs multiple passes do to aggregation or
>>>>>>>> sorting. Even if you cant post the data it would be good to get a
>>>>>>>> bugzilla entry started for this to verify performance implications.
>>>>>>>>
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> While stepping through the process with the debugger, it seems that
>>>>>>>>> the problem is stemming from the PassManager.doPopulation() method
>>>>>>>>> (as part of ResultSet caching). During this process, the engine
>>>>>>>>> seems to be trying to sort the entire result set and with great
>>>>>>>>> difficulty - is there any way to perhaps turn this off?
>>>>>>>>>
>>>>>>>>> Jason Weathersby wrote:
>>>>>>>>>> Pinny,
>>>>>>>>>>
>>>>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>>>>> like to get recorded.
>>>>>>>>>>
>>>>>>>>>> Jason
>>>>>>>>>>
>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>>>>> there a way to do this as a parameter passed as part of the URL?
>>>>>>>>>>> Also, is there a default configuration setting for this?
>>>>>>>>>>>
>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>>>
>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>>>> (runs and rendered in a few seconds) when using small datasets
>>>>>>>>>>>>> of 50 - 150 rows, although the final rptdocument is about 4MB
>>>>>>>>>>>>> (renders to 45kb pdf). When using a dataset of ~6000 rows,
>>>>>>>>>>>>> however it takes 4-5 minutes to complete, and the rptdocument
>>>>>>>>>>>>> is over 100MB (yields 4.5MB pdf). And when supplying ~18000
>>>>>>>>>>>>> rows the session expires after about 40 minutes, and by that
>>>>>>>>>>>>> time the rptdocument has reached over 300MB. Any ideas what
>>>>>>>>>>>>> steps I can take to find the area(s) that are causing this
>>>>>>>>>>>>> performance issue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem I am speaking of is strictly run related - once the
>>>>>>>>>>>>> rptdocument is created, the rendering performance is quite fast
>>>>>>>>>>>>> (at least where the rptdocument has successfully been created).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>>>> (~350MB).
>
>
Re: .rptdocument creation seems to have problems working with large datasets [message #367878 is a reply to message #367866] Wed, 15 April 2009 05:55 Go to previous messageGo to next message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Hi,

When we are talking about nested table, usually we refer to a table that has
different data set with its container.

If you table has used "container's Data Set", then we call it a sub-table.
For the sub-table we will only execute the query against the database once.

Similar to that of nested table, the sub-table usually will slow down the
BIRT query execution.

So, do you mean the "nested table" you mentioned in previous thread is
actually a "sub-table"?

If will be appreciated if you can upload a report design that can demostrate
your problem :) so that keep us in same page.

Thanks.
Lin
"Pinny Markowitz" <pinny@medwiztech.com> wrote in message
news:gs2au0$pca$1@build.eclipse.org...
> In the 'Edit Data Binding...' dialog of the nested table, I am seeing
> 'Data Set' radio button checked with 'Container's Data Set' selected. Does
> this actually fetch the same dataset by running the same query multiple
> times? If so, I would question why that is necessary. If not, I am
> wondering why performance is being hurt.
>
> I am using 2.3.2 as mentioned above.
>
> lzhu wrote:
>> Please note that in BIRT 2.3.2 the binding of a nested table to its
>> parent report item is not supported any more. So for nested table it can
>> only be bound to a data set.
>>
>> Thanks.
>> Lin
>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>> news:grvhbu$lbi$1@build.eclipse.org...
>>> Is this true even when the nested table is bound directly to the
>>> containing parent table instead of a standard dataset? I needed to use
>>> a nested table to achieve the vertical spanning effect as described
>>> here:
>>>
>>> http://www.eclipse.org/newsportal/article.php?id=33244&g roup=eclipse.birt#33244
>>>
>>> Is there any intention to fix this in the near future?
>>>
>>> lzhu wrote:
>>>> Jason,
>>>>
>>>> Correct. For each nest table execution we will visit DB once. If a
>>>> nested table is put into group head, and the group have 100 instances,
>>>> then for acqire nested table data we've to query DB for 100 times.
>>>> These are expensive operations.
>>>>
>>>> Thanks.
>>>> Lin
>>>> "Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
>>>> news:grjonm$mps$1@build.eclipse.org...
>>>>> Lin,
>>>>>
>>>>> You mean many nested tables as each additional one affects
>>>>> performance, correct?
>>>>>
>>>>> Jason
>>>>>
>>>>> lzhu wrote:
>>>>>> Hi Pinny,
>>>>>>
>>>>>> How many groups do you have after the report is rendered? For each
>>>>>> group BIRT will make a query execution against database to fetch
>>>>>> nested table data.
>>>>>>
>>>>>> It is always a good practice that avoid nested table in report
>>>>>> design.
>>>>>>
>>>>>> It will help the performance tuning if you can post your report
>>>>>> design.
>>>>>>
>>>>>> Thanks.
>>>>>> Lin
>>>>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>>>>> news:grg82b$6be$1@build.eclipse.org...
>>>>>>> Submitted as 271512.
>>>>>>>
>>>>>>> Pinny Markowitz wrote:
>>>>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>>>>> removed all sorting from the report design and that was a
>>>>>>>> tremendous boost for performance.
>>>>>>>>
>>>>>>>> However, I am finding that using a nested table within a group
>>>>>>>> header (bound to the containing table's dataset) is a big drain on
>>>>>>>> performance as well. The rptdocument was created much faster when
>>>>>>>> I moved the nested contents back to the parent - and it was much
>>>>>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared
>>>>>>>> to 10+ minutes when using nested table). Rendering time (as pdf)
>>>>>>>> was affected by the nested table as well, although not as
>>>>>>>> significantly. When using a simpler layout, I was able to recreate
>>>>>>>> similar performance differences although on a smaller scale.
>>>>>>>>
>>>>>>>> Jason Weathersby wrote:
>>>>>>>>> Pinny,
>>>>>>>>>
>>>>>>>>> It does this if the data needs multiple passes do to aggregation
>>>>>>>>> or sorting. Even if you cant post the data it would be good to
>>>>>>>>> get a bugzilla entry started for this to verify performance
>>>>>>>>> implications.
>>>>>>>>>
>>>>>>>>> Jason
>>>>>>>>>
>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>> While stepping through the process with the debugger, it seems
>>>>>>>>>> that the problem is stemming from the PassManager.doPopulation()
>>>>>>>>>> method (as part of ResultSet caching). During this process, the
>>>>>>>>>> engine seems to be trying to sort the entire result set and with
>>>>>>>>>> great difficulty - is there any way to perhaps turn this off?
>>>>>>>>>>
>>>>>>>>>> Jason Weathersby wrote:
>>>>>>>>>>> Pinny,
>>>>>>>>>>>
>>>>>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>>>>>> like to get recorded.
>>>>>>>>>>>
>>>>>>>>>>> Jason
>>>>>>>>>>>
>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>>>>>> there a way to do this as a parameter passed as part of the
>>>>>>>>>>>> URL? Also, is there a default configuration setting for this?
>>>>>>>>>>>>
>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>>>>> (runs and rendered in a few seconds) when using small
>>>>>>>>>>>>>> datasets of 50 - 150 rows, although the final rptdocument is
>>>>>>>>>>>>>> about 4MB (renders to 45kb pdf). When using a dataset of
>>>>>>>>>>>>>> ~6000 rows, however it takes 4-5 minutes to complete, and the
>>>>>>>>>>>>>> rptdocument is over 100MB (yields 4.5MB pdf). And when
>>>>>>>>>>>>>> supplying ~18000 rows the session expires after about 40
>>>>>>>>>>>>>> minutes, and by that time the rptdocument has reached over
>>>>>>>>>>>>>> 300MB. Any ideas what steps I can take to find the area(s)
>>>>>>>>>>>>>> that are causing this performance issue?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The problem I am speaking of is strictly run related - once
>>>>>>>>>>>>>> the rptdocument is created, the rendering performance is
>>>>>>>>>>>>>> quite fast (at least where the rptdocument has successfully
>>>>>>>>>>>>>> been created).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>>>>> (~350MB).
>>
Re: .rptdocument creation seems to have problems working with large datasets [message #367938 is a reply to message #367878] Tue, 21 April 2009 03:13 Go to previous messageGo to next message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Hi Pinny,

Any update?

Thanks.
Lin
"lzhu" <lzhu@actuate.com> wrote in message
news:gs3stb$6rq$1@build.eclipse.org...
> Hi,
>
> When we are talking about nested table, usually we refer to a table that
> has different data set with its container.
>
> If you table has used "container's Data Set", then we call it a sub-table.
> For the sub-table we will only execute the query against the database
> once.
>
> Similar to that of nested table, the sub-table usually will slow down the
> BIRT query execution.
>
> So, do you mean the "nested table" you mentioned in previous thread is
> actually a "sub-table"?
>
> If will be appreciated if you can upload a report design that can
> demostrate your problem :) so that keep us in same page.
>
> Thanks.
> Lin
> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
> news:gs2au0$pca$1@build.eclipse.org...
>> In the 'Edit Data Binding...' dialog of the nested table, I am seeing
>> 'Data Set' radio button checked with 'Container's Data Set' selected.
>> Does this actually fetch the same dataset by running the same query
>> multiple times? If so, I would question why that is necessary. If not,
>> I am wondering why performance is being hurt.
>>
>> I am using 2.3.2 as mentioned above.
>>
>> lzhu wrote:
>>> Please note that in BIRT 2.3.2 the binding of a nested table to its
>>> parent report item is not supported any more. So for nested table it can
>>> only be bound to a data set.
>>>
>>> Thanks.
>>> Lin
>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>> news:grvhbu$lbi$1@build.eclipse.org...
>>>> Is this true even when the nested table is bound directly to the
>>>> containing parent table instead of a standard dataset? I needed to use
>>>> a nested table to achieve the vertical spanning effect as described
>>>> here:
>>>>
>>>> http://www.eclipse.org/newsportal/article.php?id=33244&g roup=eclipse.birt#33244
>>>>
>>>> Is there any intention to fix this in the near future?
>>>>
>>>> lzhu wrote:
>>>>> Jason,
>>>>>
>>>>> Correct. For each nest table execution we will visit DB once. If a
>>>>> nested table is put into group head, and the group have 100 instances,
>>>>> then for acqire nested table data we've to query DB for 100 times.
>>>>> These are expensive operations.
>>>>>
>>>>> Thanks.
>>>>> Lin
>>>>> "Jason Weathersby" <jasonweathersby@alltel.net> wrote in message
>>>>> news:grjonm$mps$1@build.eclipse.org...
>>>>>> Lin,
>>>>>>
>>>>>> You mean many nested tables as each additional one affects
>>>>>> performance, correct?
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> lzhu wrote:
>>>>>>> Hi Pinny,
>>>>>>>
>>>>>>> How many groups do you have after the report is rendered? For each
>>>>>>> group BIRT will make a query execution against database to fetch
>>>>>>> nested table data.
>>>>>>>
>>>>>>> It is always a good practice that avoid nested table in report
>>>>>>> design.
>>>>>>>
>>>>>>> It will help the performance tuning if you can post your report
>>>>>>> design.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> Lin
>>>>>>> "Pinny Markowitz" <pinny@medwiztech.com> wrote in message
>>>>>>> news:grg82b$6be$1@build.eclipse.org...
>>>>>>>> Submitted as 271512.
>>>>>>>>
>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>> It seems correct that the primary problem was indeed the sorting -
>>>>>>>>> removed all sorting from the report design and that was a
>>>>>>>>> tremendous boost for performance.
>>>>>>>>>
>>>>>>>>> However, I am finding that using a nested table within a group
>>>>>>>>> header (bound to the containing table's dataset) is a big drain on
>>>>>>>>> performance as well. The rptdocument was created much faster when
>>>>>>>>> I moved the nested contents back to the parent - and it was much
>>>>>>>>> smaller in size as well (under 2 minutes for ~18000 rows, compared
>>>>>>>>> to 10+ minutes when using nested table). Rendering time (as pdf)
>>>>>>>>> was affected by the nested table as well, although not as
>>>>>>>>> significantly. When using a simpler layout, I was able to recreate
>>>>>>>>> similar performance differences although on a smaller scale.
>>>>>>>>>
>>>>>>>>> Jason Weathersby wrote:
>>>>>>>>>> Pinny,
>>>>>>>>>>
>>>>>>>>>> It does this if the data needs multiple passes do to aggregation
>>>>>>>>>> or sorting. Even if you cant post the data it would be good to
>>>>>>>>>> get a bugzilla entry started for this to verify performance
>>>>>>>>>> implications.
>>>>>>>>>>
>>>>>>>>>> Jason
>>>>>>>>>>
>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>> While stepping through the process with the debugger, it seems
>>>>>>>>>>> that the problem is stemming from the PassManager.doPopulation()
>>>>>>>>>>> method (as part of ResultSet caching). During this process, the
>>>>>>>>>>> engine seems to be trying to sort the entire result set and with
>>>>>>>>>>> great difficulty - is there any way to perhaps turn this off?
>>>>>>>>>>>
>>>>>>>>>>> Jason Weathersby wrote:
>>>>>>>>>>>> Pinny,
>>>>>>>>>>>>
>>>>>>>>>>>> Any chance you open a bug for this? Any performance issues we
>>>>>>>>>>>> like to get recorded.
>>>>>>>>>>>>
>>>>>>>>>>>> Jason
>>>>>>>>>>>>
>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>> I have seen Jason's comment (2/16) about setting memory cache
>>>>>>>>>>>>> size to perhaps improve performance for larger datasets - is
>>>>>>>>>>>>> there a way to do this as a parameter passed as part of the
>>>>>>>>>>>>> URL? Also, is there a default configuration setting for this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>>> Forgot to mention, I'm running Tomcat 5.5, using BIRT v2.3.2
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pinny Markowitz wrote:
>>>>>>>>>>>>>>> I have a fairly complex report layout that seems to run well
>>>>>>>>>>>>>>> (runs and rendered in a few seconds) when using small
>>>>>>>>>>>>>>> datasets of 50 - 150 rows, although the final rptdocument is
>>>>>>>>>>>>>>> about 4MB (renders to 45kb pdf). When using a dataset of
>>>>>>>>>>>>>>> ~6000 rows, however it takes 4-5 minutes to complete, and
>>>>>>>>>>>>>>> the rptdocument is over 100MB (yields 4.5MB pdf). And when
>>>>>>>>>>>>>>> supplying ~18000 rows the session expires after about 40
>>>>>>>>>>>>>>> minutes, and by that time the rptdocument has reached over
>>>>>>>>>>>>>>> 300MB. Any ideas what steps I can take to find the area(s)
>>>>>>>>>>>>>>> that are causing this performance issue?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem I am speaking of is strictly run related - once
>>>>>>>>>>>>>>> the rptdocument is created, the rendering performance is
>>>>>>>>>>>>>>> quite fast (at least where the rptdocument has successfully
>>>>>>>>>>>>>>> been created).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Upping the heap size hasn't helped for this - max memory
>>>>>>>>>>>>>>> allocated (512MB) is far more than what is being utilized
>>>>>>>>>>>>>>> (~350MB).
>>>
>
Re: .rptdocument creation seems to have problems working with large datasets [message #367981 is a reply to message #367855] Fri, 24 April 2009 13:32 Go to previous messageGo to next message
SuperTime is currently offline SuperTimeFriend
Messages: 45
Registered: July 2009
Member
We are currently running BIRT 2.2.2 and planning to upgrade to 2.3.2. We
have reports that are using nested tables, where the main table is bound
to one data set the nested table that lies in the group header of the
main table is bound to another dataset. But going forward the main table
and nested tables are linked (like a subreport). In your post are you
saying that in 2.3.2 the linking of the nested table with the item on the
main table is not supported.
Please clarify. Thank you!
Re: .rptdocument creation seems to have problems working with large datasets [message #368095 is a reply to message #367981] Mon, 04 May 2009 03:40 Go to previous message
Lin Zhu is currently offline Lin ZhuFriend
Messages: 72
Registered: July 2009
Member
Hi,

No, this will be still supported. In my previous post I only want to clarify
that this kind of nest will lead to multi queries into DB so that may have
some performance issue.

Thanks.
Lin
"SuperTime " <jayagolani@hotmail.com> wrote in message
news:a30559bdca668a5b4ae712c932bf64f4$1@www.eclipse.org...
> We are currently running BIRT 2.2.2 and planning to upgrade to 2.3.2. We
> have reports that are using nested tables, where the main table is bound
> to one data set the nested table that lies in the group header of the
> main table is bound to another dataset. But going forward the main table
> and nested tables are linked (like a subreport). In your post are you
> saying that in 2.3.2 the linking of the nested table with the item on the
> main table is not supported. Please clarify. Thank you!
>
Previous Topic:Integrating BIRT with Jackrabbit
Next Topic:Why the preview tab doesn't show up in the exported RCP product.
Goto Forum:
  


Current Time: Thu Apr 25 20:23:48 GMT 2024

Powered by FUDForum. Page generated in 0.06201 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top