Re: [stellation-res] UNITTESTS DROP: support for nested directories

["Mark C. Chu-Carroll" <mcc@xxxxxxxxxxxxxx>]

On Saturday 03 August 2002 06:13 pm, Florin Iucha wrote:
> On Sat, Aug 03, 2002 at 04:25:09PM -0400, Mark C. Chu-Carroll wrote:
> > On Saturday 03 August 2002 03:54 pm, Florin Iucha wrote:
> > > New features: nested directories.
> > >
> > > Question: why is stellation using unix commands for file manipulation
> > > instead of java native io API? Is it faster this way?
> >
> > All of the Unix commands are supposed to be in isolated code,
> > where there's a non-Unix alternative. (If it's not, then it's a bug,
> > and should be fixed.)
>
> Well, there is a problem. I have run the SvcCheckinTest under the java
> profiler that comes with the JDK and the results are not pretty:
>
> CPU SAMPLES BEGIN (total = 13266) Sat Aug  3 16:58:01 2002
> rank   self  accum   count trace method
>    1 73.96% 73.96%    9812   233 java.lang.UNIXProcessReaper.run
>    2 25.31% 99.27%    3357   156 java.net.SocketInputStream.socketRead0
>    3  0.08% 99.34%      10   234 java.io.PipedInputStream.receivedLast
>    4  0.06% 99.40%       8    50 java.lang.ClassLoader.defineClass0
>    5  0.04% 99.44%       5     8 java.util.zip.ZipFile.getEntry
>    6  0.03% 99.47%       4   272 java.lang.ProcessInputStream.run
>    7  0.03% 99.50%       4   271 java.lang.UNIXProcess.deadChild
>    8  0.02% 99.53%       3    36 java.lang.ClassLoader.defineClass0
>    9  0.02% 99.55%       3   274 java.io.FileInputStream.readBytes
>   10  0.02% 99.56%       2    52 java.lang.ClassLoader.findBootstrapClass
>   11  0.02% 99.58%       2    49 java.util.zip.Inflater.inflateBytes
>   12  0.02% 99.59%       2   264
> org.eclipse.stellation.comm.messaging.Connection.run
>
> I have used -Xrunhprof:cpu=samples,file=profile.txt,depth=4 with SUN's
> 1.4.0_01-b03 JDK on Linux 2.4.19-pre5-ac1.
>
> With IBM's J2RE 1.3.1 IBM build cxia32131-20020410 JDK, the stats are:
>
> CPU SAMPLES BEGIN (total = 7206) Sat Aug  3 17:08:01 2002
> rank   self  accum   count trace method
>    1 53.26% 53.26%    3838   226 java.lang.UNIXProcessReaper.run
>    2 45.52% 98.78%    3280   148 java.net.SocketInputStream.socketRead0
>    3  0.12% 98.90%       9     9 java.lang.ClassLoader.defineClass0
>    4  0.11% 99.01%       8   231 java.io.PipedInputStream.receivedLast
>    5  0.07% 99.08%       5     1 java.io.FileInputStream.readBytes
>    6  0.04% 99.13%       3   280 java.io.FileInputStream.readBytes
>    7  0.03% 99.15%       2   232 java.lang.ProcessInputStream.run
>    8  0.03% 99.18%       2   185 java.net.SocketOutputStream.socketWrite0
>    9  0.03% 99.21%       2   286 java.io.FileInputStream.readBytes
>   10  0.03% 99.24%       2    41 java.lang.Throwable.fillInStackTrace
>   11  0.01% 99.25%       1   288
> org.apache.xerces.impl.XMLEntityManager.expandSystemId 12  0.01% 99.26%    
>   1   247 java.lang.String.intern
>
> This is pretty much in line with what one sees when using top: during a
> unittest run, the CPU is used in bursts, and there are some inactivity
> periods.

I need to check the profiling data in more detail (but it'll have to wait 
until after my current debugging pass at the earlest.).

However: CPU bursts are an expected property of the performance.

Remember that this is a client/server system. Watching my system on a
monitor, what I see matches the communication performance pretty
precisely. 

For example, when I do a checkin, I see a burst of CPU activity (the
local workspace code being scanned for changes), followed by a
lull in CPU activity but a major burst of network traffic (changes going
onto the wire), followed by a lull in both CPU and network (changes being
processed by the server), followed by a blip on the network and a burst
on the local CPU (updating the workspace metadata). 

> Something needs to be done: probably use native calls via JNI.

I dislike the exec calls to outside commands already. I *really* don't
want to see JNI getting into Stellation. It kills portability, makes 
compilation dramatically harder, makes installation harder... And 
I'm not convinced that the real performance problems are best
addressed by going to JNI. 

Like I said, the workspace/command-line code isn't exactly my area
of expertise. But in general, a lot of our code has plenty of room
to clean up the performance. A lot of the workspace code has
sort of evolved into its current state in a fairly haphazard way,
and there's a good chance that the performance could be cleaned up
by doing some re-architecting to clean it up (which I really want to do
anyway; now that I need to look at it more, I find it hopelessly confusing). 
I'd rather spend some serious effort on  trying to clean up the stuff we have 
now, and be absolutely certain that there was no other answer to the 
performance issue before going to JNI.

	-Mark

-- 
Mark Craig Chu-Carroll,  IBM T.J. Watson Research Center  
*** The Stellation project: Advanced SCM for Collaboration
***		http://www.eclipse.org/stellation
*** Work Email: mcc@xxxxxxxxxxxxxx  ------- Personal Email: markcc@xxxxxxxxxxx