Re: [stellation-res] Stellation on Mckoi

["Mark C. Chu-Carroll" <mcc@xxxxxxxxxxxxxx>]

On Tue, 2003-07-22 at 13:59, Snow wrote:
> Hi,
> 
> I started writing this a month or two ago and intended to come back to it,
> but I haven't had the time. However I thought I'd send it anyway in case
> it's useful.

I hate to be the bearer of bad news, but:

We can't take this contribution. McKoi is GPL; Stellation is CPL.
The two are not compatible. Therefore we cannot include McKoi
support in our distribution of Stellation. 

We will, shortly, be adding plug-in support so that it will be easy
for you to package the McKoi support as a separate component,
which you can distribute under the GPL as an add-on for
Stellation. But we can't. (Sorry - this is the agreement that
we struck with the lawyers in order to get permission to 
do this project as open-source - we have to scrupulously stick
to their license interpretations, and they say we can't do this.)

Anyway, I'll respond to the rest point by point.

> 
> ----------------------------------------------
> 
> I've been trying to get Stellation running on the Mckoi DB. Mckoi can run
> either standalone (DB in its own JVM) and embedded (where the DB engine is
> in the Stellation JVM).
> 
> As far as SQL is concerned there were two small problems:
> 1) Mckoi doesn't like having 'REFERENCE' after column definitions. Since
> these seem to duplicate the FOREIGN KEY constaint, removing them fixed the
> problem.
> 2) I had to turn off case-sensitivity in mckoi.

That shouldn't be too hard using the database layer. I assume this
wasn't too painful. Are there any changes that you would suggest to
the docs based on doing this?


> Now I encountered the biggest problem: how to get the command line options
> for
> the database into the program. At first I was irritated, then I thought it
> was funny. Finally I realized that the command line processing was the
> product of someone very, very evil. Someone who needs to be locked up in a
> padded cell before they try to destroy mankind.

Well, I wouldn't go *quite* that far. But yes, our current command-line
code is rather a disaster area. The member of the team who wrote is
has since left the project for greener pastures, and those of us
who are still working on Stellation have largely given up on the
current command-line code as a lost cause: it's so tangled together
in ways that we simply don't understand that fixing it is likely
more work than just replacing it. 

> Everytime I changed the order in one place some other place would randomly
> rearrange the order of things in the StringList or Location or other String
> collection object thing chosen apparently at random. After spending 4 hours
> trying to get the orders straight I gave up. For the moment Mckoi gets its
> details from the system-properties 'dburl', 'dbschema', 'dbusername' and
> 'dbpassword'. I may try again some other time.

Don't put a lot of effort into it. Before too long, we'll replace it
with a new command-line tool based on headless Eclipse, and I promise,
the command-line processing will be *much* more straightforward.

Basically, we just plain got the Location stuff wrong. The original
idea was to have something URL-like for specifying repository
locations. But it just never worked well. Even if the command-line code
were the cleanest, most comprehensible code in the world, the
whole location notation would still cause endless confusion and
grief. In the next version of the command line, we're going to
discard location strings in favor of multiple command-line
parameters.

So, for example, instead of:
	svc --location=remote:stellation.eclipse.org:6060:stellrepo

We would use:

	svc --mode=remote --hostname=stellation.eclipse.org --port=6060
--dbname=stellrepo

(And the default for mode will be "remote" if there is a hostname
parameter, and the default for port will be 6060, so actually,
you'd really use:
	svc ---hostname=stellation.eclipse.org --dbname=stellrepo

Much better, neh?)

And it may get better than that: John Gossage was working on
a configuration support module, so that you register named
repositories using a config tool, and then just access by name,
so you'd just have to "svc --repos=stellremote", and it would
look up the mode, hostname, dbname, and any other needed parameters.

> At this point all the tests in test/build.xml passed.
> 
> However if I forget to add a comment to command line the tsvc program would
> ask for one and ctrl-Z doesn't work as the terminator like the docs say so I
> pressed ctrl-C. However whenever I press ctrl-C the server goes into an
> inifinite loop and the CPU goes to 99%. The problem was at
> 
> org.eclipse.stellation.remote.engine.Connection.run(Connection.java:114)

That sounds like a bug. Could you drop a bugzilla entry for it?

^Z on a blank line should work; there's a fix that I haven't checked
in yet which also allows "." on a blank line (and generates a message
to tell you that). 

> When the stream is terminated it throws an IOException, but this is just
> caught and goes and tries again. I added a failureCount++ to the start of
> the loop which seemed to fix it, but I'm not sure if it's right.
> 
> Also when I do moderately large operations (eg checkin 200 files) the memory
> usage goes up rapidly (200MB+). Is this normal? I thought it might be the
> JDK1.4.1/JDOM bug, but I tried JDK1.4.0 and it still happened.

I think that's McKOI. The Stellation scripts limit the memory size that
the JVM is allowed to allocate to 160M. McKOI's docs say that its
default config trades off performance for memory.

That said, Stellation is currently very memory inefficient. The
communication layer wastes a *lot* of memory on checkins and
checkouts. The fix isn't too difficult, but it's never made it
to the front of our priority queue. If it's really important to you,
put it into bugzilla as a problem. If you're *really* ambitious,
I can explain where the problem comes from and how to fix it, and
you can take a stab at the code.

> Other irks I came across:
> 
> 1) Calling exit from bat files is annoying.

Yes. It is. We should fix that. 

> 2) bat files need quotes around paths and usernames with spaces in.

Usernames shouldn't have spaces. But yes, we do definitely need
to make sure that we properly handle filenames with spaces. Our
main problem here is that we're primarily a linux group, and
we've been relying on outside contributors to help get things working
on windows. If you'd like to help, we'd appreciate it.


> 3) The docs all say the database separator is : it's actually +

That's doc lag. We had to change from colon for windows support: the
colon interfered with specifying the database name for Firebird.
The guy who's been helping us with windows support make that
change, and we didn't update the docs. We'll get that fixed ASAP.


> 4) I tried using the build.xml files for a while before reading the
> instructions and using build-core.xml.

That's unavoidable. Eclipse generates a build.xml file for
plugin projects. We need a separate build file for command-line
compilation, and there's no way to tell eclipse to use a different
name. 

> 5) "tsvc add ." doesn't add the current directory.

It doesn't? That's bad.... Could you put that into bugzilla?

> 6) On Windows getCanonicalizedName() is v. v. slow.

Due to lack of windows knowledge, neither me, nor the person who wrote
the command-line interface knew that. Do you know of an efficient
alternative that will do the same job?

> All the above tests were done on Sun JDK1.4.1_02 on WinXP. With repository
> and client
> on same machine.
> 
> Once I had it working on Windows I tried Linux. For some reason mckoi
> couldn't find its database files in embedded mode so I had to run it
> standalone. When I tried to run tsvcadmin on Linux it kept saying it doesn't
> understand 'initialize' parameter. I couldn't track down what was causing
> this
> so I ran tsvcadmin on my Windows machine pointing at the DB on the Linux
> machine.
> 
> Next I ran up the tsvcserver; this started without problems. However
> whenever I tried to connect to it (from the Win machine) it would just
> freeze. Looking at the thread dumps both ends seem to be reading (though the
> IBM JDK thread-dump is quite hard to read so I may be wrong).


I would appreciate it if you could send me command-line transripts of
running tsvcadmin an linux with --debug=2, as well as a copy of the
JDK thread dump. I'd like to try to see what's going wrong.

	-Mark

--