Home » Archived » Test and Performance Tools Platform (TPTP) » Some personal thoughts about profiling with TPTP
| Some personal thoughts about profiling with TPTP [message #112236] |
Wed, 12 September 2007 05:23  |
Eclipse User |
|
|
|
Originally posted by: mauro.molinari.cardinis.com
Hello all.
After spending about an entire week trying to profile a medium-size
application (just its starting phase, with data loading from a database)
with TPTP, I would like to share with you my personal experience in
order to understand your point of view.
Why?
Because I'm sorry, but I must admit that NetBeans profiler is many steps
ahead TPTP... I don't like NetBeans and I think it's some years behind
Eclipse in many areas, but there's no story about the profiler.
A co-worker of mine and I spent almost an entire week trying to make
TPTP work as we wanted, without succeeding, while we were able to do the
same thing with NetBeans profiler within one hour (!!!), even if we had
to install it and use for the very first time!
I'm explaining why.
PRELIMINARY REMARK: in this post, when I talk about "profiling" I'm
referring to Execution Time Analysis with JRE 1.5 or newer (actually, we
used 1.5), with the following options:
- collect method CPU time information UNCHECKED
- Show execution flow graphical details SELECTED
Apart from the fact I couldn't make TPTP work on my own PC (but I thank
Igor Alelekov very much for his support on this issue, see bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=201474), my collegue and I
could make it work on a Linux system (equipped with an Athlon 64 2800+,
1.5GB RAM) and on another Windows XP system (equipped with a Sempron
2600+, 1.5 GB RAM).
First of all, under Linux there are some annoying issues:
- we couldn't use the IAC: we must use the stand-alone agent controller;
it's not a big problem, although it makes TPTP not so "plug 'n' play"
for the beginner
- even if we were using a powerful system, we had to calibrate the
filters very strictly in order to get data in a reasonable amount of
time and without using too much system resources; this required quite a
big amount of time, because we had to exclude almost ALL classes and
then include just those we wanted to analyse... the problem was to
identify those classes and understand the difference between an
inclusion filter like "com.something.*" and "com.something*"... However,
the performance was quite poor even if we fine tuned the filters so
carefully...
- on this system (running an Debian 4.1 with Gnome window manager),
navigating through the Call Tree view is a pain... Just clicking on some
of the entries (in order to highlight them) takes AGES... don't ask me why!
- we couldn't export the data: when we tried to do that, Eclipse froze,
no progress bar appeared, the CPU went to 100% and, although we left the
PC for several minutes, when we came back it was still there... so we
had to kill Eclipse...
- we had alternate luck on profiling: the overall impression was that
TPTP is not so stable and reliable, so we concluded that it was safer to
close Eclipse and start it again from one profiling session to another
On the other system, we had better luck. Under Windows XP, in fact, the
user experience seems a bit better:
- the IAC works and we used it
- there are no navigation problems in the Call Tree (the "pauses" shown
when clicking on some entries were acceptable)
However, there are still many problems. First of all, the performance:
we had to wait several minutes to complete the profiling of the load
process of our application (even if the filters were fine-tuned) and it
required more than 600 MB of RAM just to collect data (!!!).
Anyway, we wanted to do these simple tasks:
1) profile the application as it were and collect data
2) change the application startup algorithm
3) profile the application again and collect the new data
4) with both profiling data in the workbench, compare the results
We couldn't manage to perform these tasks, because:
- even if we could save data under Windows XP (honestly, I didn't try to
"export", as on Linux, but simply to "Save" data in the workbench),
after restarting Eclipse (it took AGES to start up again, I think it was
trying to load the profiling data just saved...) I tried to navigate
through that data but the views were... EMPTY!
- we coulnd't perform both profile sessions without restarting Eclipse,
because TPTP requires too much memory: on that system, with 1.5 GB RAM,
I couldn't start Eclipse with more the 1GB of maximum Java heap space,
and it was not enough, because each single profiling session ate
hundreds of MB of RAM (after manually starting the garbage collector
from the Eclipse heap status bar, 600 MB were still allocated by
Eclipse, even if the first profiling session was terminated)
Moreover:
- sometimes filters are not honoured: data collection seemed to stop at
a certain point and execution time information about some classes and
methods I would expect to find were not shown in the statistics; I had
to remove and add again some filtering rules from one profiling session
to another to make it more or less work as intended
- filters are not saved immediately: if Eclipse crashes and you restart
it, the filters you added in the previous session are not saved; you
must close Eclipse in a safe way in order to make them persistent from
one restart to another
- if I have this object hierarchy:
class A{ public void a(); }
class B extends A{ public void b(); }
if I'm interested in profiling just a() method, when executed on a B
instance, I have to include also B (not only A) in the filtering
criteria, otherwise data is not collected... Maybe this is the wanted
behaviour, but I found it hard (or, better: time consuming!!) to get to
this conclusion
- in the call tree, I just see method names, without the corresponding
classes: suppose you see a next() entry, it would be "nice" to know that
you are talking about SomeClass.next() intead of, say,
AnotherClass.next()... You have to right click and choose "Open Source"
(wouldn't "Go To Source" be better? ;-)) to go to the corresponding
source code
- we also would find it very useful if it were possible to define a
profiling criteria like this: "start to profile the mathing classes but
only after A.a() is invoked for the first time, and stop when B.b() is
invoked".
I must admit that the application we tried to profile was not so small
and loads a lot of data... but it is a real world application, were we
need a profiling tool to understand how to make it better, if needed! We
had to find out the bottle neck in the starting process and eliminate it...
At the end, we tried NetBeans profiler: using the Linux system, with
less than 300 MB of RAM used by NetBeans, we could do 4 subsequent
profiling sessions (!!!), filtering out JUST Java and Sun classes (so:
no fine-tuning of filters were needed here to get a decent performance
result!), navigate through data as if we were navigating through source
code (I mean: the software was extremely responsive, it didn't give the
impression of moving through some gigabytes of data or such...), save
data in a couple of seconds and show graphs and statistics updated even
in REAL-TIME!!! And without the need of any external agent controller
process...
Maybe I wouldn't have even written here if I didn't experience such an
impressive difference in performance and scalability of NetBeans
profiler against TPTP, because I'm not the typical user that thinks it's
funny to feed stupid wars like NetBeans vs Eclipse: however, I just
wonder if there are some other reasons behind these results...
Does TPTP do also other things that NetBeans doesn't, when profiling the
execution time analysis of Java code?
Is NetBeans using some "hidden" APIs or particular features of Sun VMs
that Eclipse can't or doesn't use to profile applications?
Thanks for reading up to here and sorry for such a long post...
Mauro.
|
|
| |
| Re: Some personal thoughts about profiling with TPTP [message #112405 is a reply to message #112236] |
Thu, 13 September 2007 12:08  |
| Eclipse User |
|
|
|
Mauro Molinari wrote:
> - on this system (running an Debian 4.1 with Gnome window manager),
> navigating through the Call Tree view is a pain... Just clicking on
> some of the entries (in order to highlight them) takes AGES... don't
> ask me why!
This could be Bug 200839
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=200839).
> However, there are still many problems. First of all, the performance:
> we had to wait several minutes to complete the profiling of the load
> process of our application (even if the filters were fine-tuned) and
> it required more than 600 MB of RAM just to collect data (!!!).
I think that the TPTP Profiler still records every single method
invocation and is prepared to display it in the call tree view (the CPU
time needed to handle clicking on a node seems to be proportional to the
number of calls of that node). This creates scaling problems for more
that a few thousand method calls and is almost always more than is
really needed.
The old Eclipse Profiler Project
(http://eclipsecolorer.sourceforge.net/index_profiler.html) got it
right: aggregate the data over the call tree and forget about the
individual calls. This way, you can see that Method A in the call tree
has been called 100 times and required 100 seconds, because the 100
calls have called method B 1 million times in total. There is no need
to look at the 100 calls of A or the 1 million calls of B individually,
and thus no need to record them.
With the Eclipse Profiler, I could obtain meaningful profiles at just
10-20% slowdown. Any slowdown > 50% (Factor 1,5) begins to become
unusable, because the profiling overhead begins to dominate the time
used by the application itself.
The Netbeans Profiler seems to collect less data than the old Eclipse
Profiler, namely just the method timings without the call graph. I have
worked with such profiles for a year, and it is much more cumbersome,
because you are forever trying to figure out the call relationships
between the methods that appear in the list.
Regards,
Oliver Schoett
|
|
| | | |
| Re: Some personal thoughts about profiling with TPTP [message #112500 is a reply to message #112439] |
Fri, 14 September 2007 12:59 |
| Eclipse User |
|
|
|
Mauro Molinari wrote:
> In NetBeans 5.5 profiler there's a view that is just like TPTP Call
> Tree view and I think it's quite comfortable to navigate and
> understand. I don't know if it has also an actual "call graph", like
> TPTP, but I think that it might be "easily" built from the call tree
> using the collected summary data, am I wrong?
Yes, I found it. It is available only when you press "Take Snapshot",
not when you press "Live Results". That seems to be exactly what I have
been looking for.
Screenshot: http://profiler.netbeans.org/images/screenshots50/14_sol_big .png
Regards,
Oliver Schoett
|
|
|
Goto Forum:
Current Time: Wed Jul 23 13:24:39 EDT 2025
Powered by FUDForum. Page generated in 0.12761 seconds |
Home