Home » Archived » Test and Performance Tools Platform (TPTP) » Some personal thoughts about profiling with TPTP
| Some personal thoughts about profiling with TPTP [message #112236] |
Wed, 12 September 2007 09:23  |
Eclipse User |
|
|
|
Originally posted by: mauro.molinari.cardinis.com
Hello all.
After spending about an entire week trying to profile a medium-size
application (just its starting phase, with data loading from a database)
with TPTP, I would like to share with you my personal experience in
order to understand your point of view.
Why?
Because I'm sorry, but I must admit that NetBeans profiler is many steps
ahead TPTP... I don't like NetBeans and I think it's some years behind
Eclipse in many areas, but there's no story about the profiler.
A co-worker of mine and I spent almost an entire week trying to make
TPTP work as we wanted, without succeeding, while we were able to do the
same thing with NetBeans profiler within one hour (!!!), even if we had
to install it and use for the very first time!
I'm explaining why.
PRELIMINARY REMARK: in this post, when I talk about "profiling" I'm
referring to Execution Time Analysis with JRE 1.5 or newer (actually, we
used 1.5), with the following options:
- collect method CPU time information UNCHECKED
- Show execution flow graphical details SELECTED
Apart from the fact I couldn't make TPTP work on my own PC (but I thank
Igor Alelekov very much for his support on this issue, see bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=201474), my collegue and I
could make it work on a Linux system (equipped with an Athlon 64 2800+,
1.5GB RAM) and on another Windows XP system (equipped with a Sempron
2600+, 1.5 GB RAM).
First of all, under Linux there are some annoying issues:
- we couldn't use the IAC: we must use the stand-alone agent controller;
it's not a big problem, although it makes TPTP not so "plug 'n' play"
for the beginner
- even if we were using a powerful system, we had to calibrate the
filters very strictly in order to get data in a reasonable amount of
time and without using too much system resources; this required quite a
big amount of time, because we had to exclude almost ALL classes and
then include just those we wanted to analyse... the problem was to
identify those classes and understand the difference between an
inclusion filter like "com.something.*" and "com.something*"... However,
the performance was quite poor even if we fine tuned the filters so
carefully...
- on this system (running an Debian 4.1 with Gnome window manager),
navigating through the Call Tree view is a pain... Just clicking on some
of the entries (in order to highlight them) takes AGES... don't ask me why!
- we couldn't export the data: when we tried to do that, Eclipse froze,
no progress bar appeared, the CPU went to 100% and, although we left the
PC for several minutes, when we came back it was still there... so we
had to kill Eclipse...
- we had alternate luck on profiling: the overall impression was that
TPTP is not so stable and reliable, so we concluded that it was safer to
close Eclipse and start it again from one profiling session to another
On the other system, we had better luck. Under Windows XP, in fact, the
user experience seems a bit better:
- the IAC works and we used it
- there are no navigation problems in the Call Tree (the "pauses" shown
when clicking on some entries were acceptable)
However, there are still many problems. First of all, the performance:
we had to wait several minutes to complete the profiling of the load
process of our application (even if the filters were fine-tuned) and it
required more than 600 MB of RAM just to collect data (!!!).
Anyway, we wanted to do these simple tasks:
1) profile the application as it were and collect data
2) change the application startup algorithm
3) profile the application again and collect the new data
4) with both profiling data in the workbench, compare the results
We couldn't manage to perform these tasks, because:
- even if we could save data under Windows XP (honestly, I didn't try to
"export", as on Linux, but simply to "Save" data in the workbench),
after restarting Eclipse (it took AGES to start up again, I think it was
trying to load the profiling data just saved...) I tried to navigate
through that data but the views were... EMPTY!
- we coulnd't perform both profile sessions without restarting Eclipse,
because TPTP requires too much memory: on that system, with 1.5 GB RAM,
I couldn't start Eclipse with more the 1GB of maximum Java heap space,
and it was not enough, because each single profiling session ate
hundreds of MB of RAM (after manually starting the garbage collector
from the Eclipse heap status bar, 600 MB were still allocated by
Eclipse, even if the first profiling session was terminated)
Moreover:
- sometimes filters are not honoured: data collection seemed to stop at
a certain point and execution time information about some classes and
methods I would expect to find were not shown in the statistics; I had
to remove and add again some filtering rules from one profiling session
to another to make it more or less work as intended
- filters are not saved immediately: if Eclipse crashes and you restart
it, the filters you added in the previous session are not saved; you
must close Eclipse in a safe way in order to make them persistent from
one restart to another
- if I have this object hierarchy:
class A{ public void a(); }
class B extends A{ public void b(); }
if I'm interested in profiling just a() method, when executed on a B
instance, I have to include also B (not only A) in the filtering
criteria, otherwise data is not collected... Maybe this is the wanted
behaviour, but I found it hard (or, better: time consuming!!) to get to
this conclusion
- in the call tree, I just see method names, without the corresponding
classes: suppose you see a next() entry, it would be "nice" to know that
you are talking about SomeClass.next() intead of, say,
AnotherClass.next()... You have to right click and choose "Open Source"
(wouldn't "Go To Source" be better? ;-)) to go to the corresponding
source code
- we also would find it very useful if it were possible to define a
profiling criteria like this: "start to profile the mathing classes but
only after A.a() is invoked for the first time, and stop when B.b() is
invoked".
I must admit that the application we tried to profile was not so small
and loads a lot of data... but it is a real world application, were we
need a profiling tool to understand how to make it better, if needed! We
had to find out the bottle neck in the starting process and eliminate it...
At the end, we tried NetBeans profiler: using the Linux system, with
less than 300 MB of RAM used by NetBeans, we could do 4 subsequent
profiling sessions (!!!), filtering out JUST Java and Sun classes (so:
no fine-tuning of filters were needed here to get a decent performance
result!), navigate through data as if we were navigating through source
code (I mean: the software was extremely responsive, it didn't give the
impression of moving through some gigabytes of data or such...), save
data in a couple of seconds and show graphs and statistics updated even
in REAL-TIME!!! And without the need of any external agent controller
process...
Maybe I wouldn't have even written here if I didn't experience such an
impressive difference in performance and scalability of NetBeans
profiler against TPTP, because I'm not the typical user that thinks it's
funny to feed stupid wars like NetBeans vs Eclipse: however, I just
wonder if there are some other reasons behind these results...
Does TPTP do also other things that NetBeans doesn't, when profiling the
execution time analysis of Java code?
Is NetBeans using some "hidden" APIs or particular features of Sun VMs
that Eclipse can't or doesn't use to profile applications?
Thanks for reading up to here and sorry for such a long post...
Mauro.
|
|
|
| Re: Some personal thoughts about profiling with TPTP [message #112393 is a reply to message #112236] |
Thu, 13 September 2007 14:35  |
| Eclipse User |
|
|
|
Originally posted by: alexander.n.alexeev.intel.com
Mauro, your remarks and critics of Eclipse TPTP are very valuable.
And I appreciate that your spent quite a few your time for looking into it.
Could you put your points to Bugzilla as enhancements or bugs just not to
let them be forgotten. Also it allows to track issues separately and
define them more precisely.
Thanks.
Alex.
|
|
|
| Re: Some personal thoughts about profiling with TPTP [message #112405 is a reply to message #112236] |
Thu, 13 September 2007 16:08 |
 Oliver Schoett
Messages: 9
Registered: July 2009 |
Junior Member |
|
|
Mauro Molinari wrote:
> - on this system (running an Debian 4.1 with Gnome window manager),
> navigating through the Call Tree view is a pain... Just clicking on
> some of the entries (in order to highlight them) takes AGES... don't
> ask me why!
This could be Bug 200839
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=200839).
> However, there are still many problems. First of all, the performance:
> we had to wait several minutes to complete the profiling of the load
> process of our application (even if the filters were fine-tuned) and
> it required more than 600 MB of RAM just to collect data (!!!).
I think that the TPTP Profiler still records every single method
invocation and is prepared to display it in the call tree view (the CPU
time needed to handle clicking on a node seems to be proportional to the
number of calls of that node). This creates scaling problems for more
that a few thousand method calls and is almost always more than is
really needed.
The old Eclipse Profiler Project
(http://eclipsecolorer.sourceforge.net/index_profiler.html) got it
right: aggregate the data over the call tree and forget about the
individual calls. This way, you can see that Method A in the call tree
has been called 100 times and required 100 seconds, because the 100
calls have called method B 1 million times in total. There is no need
to look at the 100 calls of A or the 1 million calls of B individually,
and thus no need to record them.
With the Eclipse Profiler, I could obtain meaningful profiles at just
10-20% slowdown. Any slowdown > 50% (Factor 1,5) begins to become
unusable, because the profiling overhead begins to dominate the time
used by the application itself.
The Netbeans Profiler seems to collect less data than the old Eclipse
Profiler, namely just the method timings without the call graph. I have
worked with such profiles for a year, and it is much more cumbersome,
because you are forever trying to figure out the call relationships
between the methods that appear in the list.
Regards,
Oliver Schoett
|
|
|
| Re: Some personal thoughts about profiling with TPTP [message #112439 is a reply to message #112405] |
Fri, 14 September 2007 07:22 |
| Eclipse User |
|
|
|
Originally posted by: mauro.molinari.cardinis.com
Hi Oliver!
Oliver Schoett ha scritto:
> Mauro Molinari wrote:
>> - on this system (running an Debian 4.1 with Gnome window manager),
>> navigating through the Call Tree view is a pain... Just clicking on
>> some of the entries (in order to highlight them) takes AGES... don't
>> ask me why!
> This could be Bug 200839
> (https://bugs.eclipse.org/bugs/show_bug.cgi?id=200839).
Thank you for pointing it out. It may be the same bug, although my bad
experience was was on Linux but not on Windows XP (even if the latter
system was slower than the former).
> The Netbeans Profiler seems to collect less data than the old Eclipse
> Profiler, namely just the method timings without the call graph. I have
> worked with such profiles for a year, and it is much more cumbersome,
> because you are forever trying to figure out the call relationships
> between the methods that appear in the list.
In NetBeans 5.5 profiler there's a view that is just like TPTP Call Tree
view and I think it's quite comfortable to navigate and understand. I
don't know if it has also an actual "call graph", like TPTP, but I think
that it might be "easily" built from the call tree using the collected
summary data, am I wrong?
Mauro.
|
|
|
| Re: Some personal thoughts about profiling with TPTP [message #112451 is a reply to message #112393] |
Fri, 14 September 2007 07:23 |
| Eclipse User |
|
|
|
Originally posted by: mauro.molinari.cardinis.com
Alexander N. Alexeev ha scritto:
> Mauro, your remarks and critics of Eclipse TPTP are very valuable. And I
> appreciate that your spent quite a few your time for looking into it.
>
> Could you put your points to Bugzilla as enhancements or bugs just not
> to let them be forgotten. Also it allows to track issues separately and
> define them more precisely.
>
> Thanks.
> Alex.
Thank you Alex for your kind reply.
I'm going to cut my original post down to extract some single request
enhancements and bug reports.
Cheers,
Mauro.
|
|
| | |
Goto Forum:
Current Time: Thu Apr 18 12:58:29 GMT 2024
Powered by FUDForum. Page generated in 0.02061 seconds |
] 
Search
Help
Register
Login
Home
