RE: [stellation-res] Stellation on Mckoi

["Jonathan Gossage" <jgossage@xxxxxxxx>]

Replies are embedded:

> -----Original Message-----
> From: stellation-res-admin@xxxxxxxxxxxxxxx
> [mailto:stellation-res-admin@xxxxxxxxxxxxxxx]On Behalf Of Mark C.
> Chu-Carroll
> Sent: Tuesday, July 22, 2003 2:45 PM
> To: Stellation-res
> Subject: Re: [stellation-res] Stellation on Mckoi
>
>
> On Tue, 2003-07-22 at 13:59, Snow wrote:
> > Hi,
> >
> > I started writing this a month or two ago and intended to come
> back to it,
> > but I haven't had the time. However I thought I'd send it anyway in case
> > it's useful.
>
> I hate to be the bearer of bad news, but:
>
> We can't take this contribution. McKoi is GPL; Stellation is CPL.
> The two are not compatible. Therefore we cannot include McKoi
> support in our distribution of Stellation.
>
> We will, shortly, be adding plug-in support so that it will be easy
> for you to package the McKoi support as a separate component,
> which you can distribute under the GPL as an add-on for
> Stellation. But we can't. (Sorry - this is the agreement that
> we struck with the lawyers in order to get permission to
> do this project as open-source - we have to scrupulously stick
> to their license interpretations, and they say we can't do this.)
>
> Anyway, I'll respond to the rest point by point.
>
> >
> > ----------------------------------------------
> >
> > I've been trying to get Stellation running on the Mckoi DB.
> Mckoi can run
> > either standalone (DB in its own JVM) and embedded (where the
> DB engine is
> > in the Stellation JVM).
> >
> > As far as SQL is concerned there were two small problems:
> > 1) Mckoi doesn't like having 'REFERENCE' after column definitions. Since
> > these seem to duplicate the FOREIGN KEY constaint, removing
> them fixed the
> > problem.
> > 2) I had to turn off case-sensitivity in mckoi.
>
> That shouldn't be too hard using the database layer. I assume this
> wasn't too painful. Are there any changes that you would suggest to
> the docs based on doing this?
>
>
> > Now I encountered the biggest problem: how to get the command
> line options
> > for
> > the database into the program. At first I was irritated, then I
> thought it
> > was funny. Finally I realized that the command line processing was the
> > product of someone very, very evil. Someone who needs to be
> locked up in a
> > padded cell before they try to destroy mankind.
>
> Well, I wouldn't go *quite* that far. But yes, our current command-line
> code is rather a disaster area. The member of the team who wrote is
> has since left the project for greener pastures, and those of us
> who are still working on Stellation have largely given up on the
> current command-line code as a lost cause: it's so tangled together
> in ways that we simply don't understand that fixing it is likely
> more work than just replacing it.
>
> > Everytime I changed the order in one place some other place
> would randomly
> > rearrange the order of things in the StringList or Location or
> other String
> > collection object thing chosen apparently at random. After
> spending 4 hours
> > trying to get the orders straight I gave up. For the moment
> Mckoi gets its
> > details from the system-properties 'dburl', 'dbschema', 'dbusername' and
> > 'dbpassword'. I may try again some other time.
>
> Don't put a lot of effort into it. Before too long, we'll replace it
> with a new command-line tool based on headless Eclipse, and I promise,
> the command-line processing will be *much* more straightforward.
>
> Basically, we just plain got the Location stuff wrong. The original
> idea was to have something URL-like for specifying repository
> locations. But it just never worked well. Even if the command-line code
> were the cleanest, most comprehensible code in the world, the
> whole location notation would still cause endless confusion and
> grief. In the next version of the command line, we're going to
> discard location strings in favor of multiple command-line
> parameters.

In fact, the location problems are the least of the problems with the
command line.There are also some absolutely horrendous performance problems
with workspace management. Last year, after I had got the MySql stuff going,
I started to do a high volume benchmark involving about 200K directories and
about 1.5M files. It took over 30 hours simply to add the files into the
workspace with the "add" command. I know why this happened but it would be
hard to fix in a robust manner with the current implementation. I have some
suggestions regarding the command line but I will present them in a separate
post.

>
> So, for example, instead of:
> 	svc --location=remote:stellation.eclipse.org:6060:stellrepo
>
> We would use:
>
> 	svc --mode=remote --hostname=stellation.eclipse.org --port=6060
> --dbname=stellrepo
>
> (And the default for mode will be "remote" if there is a hostname
> parameter, and the default for port will be 6060, so actually,
> you'd really use:
> 	svc ---hostname=stellation.eclipse.org --dbname=stellrepo
>
> Much better, neh?)
>
> And it may get better than that: John Gossage was working on
> a configuration support module, so that you register named
> repositories using a config tool, and then just access by name,
> so you'd just have to "svc --repos=stellremote", and it would
> look up the mode, hostname, dbname, and any other needed parameters.

The configuration stuff I was working on encapsulated the connection
parameters into an object that made things much easier to work with.

>
> > At this point all the tests in test/build.xml passed.
> >
> > However if I forget to add a comment to command line the tsvc
> program would
> > ask for one and ctrl-Z doesn't work as the terminator like the
> docs say so I
> > pressed ctrl-C. However whenever I press ctrl-C the server goes into an
> > inifinite loop and the CPU goes to 99%. The problem was at
> >
> > org.eclipse.stellation.remote.engine.Connection.run(Connection.java:114)
>
> That sounds like a bug. Could you drop a bugzilla entry for it?
>
> ^Z on a blank line should work; there's a fix that I haven't checked
> in yet which also allows "." on a blank line (and generates a message
> to tell you that).
>
> > When the stream is terminated it throws an IOException, but this is just
> > caught and goes and tries again. I added a failureCount++ to
> the start of
> > the loop which seemed to fix it, but I'm not sure if it's right.
> >
> > Also when I do moderately large operations (eg checkin 200
> files) the memory
> > usage goes up rapidly (200MB+). Is this normal? I thought it
> might be the
> > JDK1.4.1/JDOM bug, but I tried JDK1.4.0 and it still happened.
>
> I think that's McKOI. The Stellation scripts limit the memory size that
> the JVM is allowed to allocate to 160M. McKOI's docs say that its
> default config trades off performance for memory.
>
> That said, Stellation is currently very memory inefficient. The
> communication layer wastes a *lot* of memory on checkins and
> checkouts. The fix isn't too difficult, but it's never made it
> to the front of our priority queue. If it's really important to you,
> put it into bugzilla as a problem. If you're *really* ambitious,
> I can explain where the problem comes from and how to fix it, and
> you can take a stab at the code.
>
> > Other irks I came across:
> >
> > 1) Calling exit from bat files is annoying.
>
> Yes. It is. We should fix that.
>
> > 2) bat files need quotes around paths and usernames with spaces in.
>
> Usernames shouldn't have spaces. But yes, we do definitely need
> to make sure that we properly handle filenames with spaces. Our
> main problem here is that we're primarily a linux group, and
> we've been relying on outside contributors to help get things working
> on windows. If you'd like to help, we'd appreciate it.
>
>
> > 3) The docs all say the database separator is : it's actually +
>
> That's doc lag. We had to change from colon for windows support: the
> colon interfered with specifying the database name for Firebird.
> The guy who's been helping us with windows support make that
> change, and we didn't update the docs. We'll get that fixed ASAP.

This limitation would disappear in a re-write.

>
>
> > 4) I tried using the build.xml files for a while before reading the
> > instructions and using build-core.xml.
>
> That's unavoidable. Eclipse generates a build.xml file for
> plugin projects. We need a separate build file for command-line
> compilation, and there's no way to tell eclipse to use a different
> name.
>
> > 5) "tsvc add ." doesn't add the current directory.
>
> It doesn't? That's bad.... Could you put that into bugzilla?
>
> > 6) On Windows getCanonicalizedName() is v. v. slow.
>
> Due to lack of windows knowledge, neither me, nor the person who wrote
> the command-line interface knew that. Do you know of an efficient
> alternative that will do the same job?
>
> > All the above tests were done on Sun JDK1.4.1_02 on WinXP. With
> repository
> > and client
> > on same machine.
> >
> > Once I had it working on Windows I tried Linux. For some reason mckoi
> > couldn't find its database files in embedded mode so I had to run it
> > standalone. When I tried to run tsvcadmin on Linux it kept
> saying it doesn't
> > understand 'initialize' parameter. I couldn't track down what
> was causing
> > this
> > so I ran tsvcadmin on my Windows machine pointing at the DB on the Linux
> > machine.
> >
> > Next I ran up the tsvcserver; this started without problems. However
> > whenever I tried to connect to it (from the Win machine) it would just
> > freeze. Looking at the thread dumps both ends seem to be
> reading (though the
> > IBM JDK thread-dump is quite hard to read so I may be wrong).
>
>
> I would appreciate it if you could send me command-line transripts of
> running tsvcadmin an linux with --debug=2, as well as a copy of the
> JDK thread dump. I'd like to try to see what's going wrong.
>

Regards

Jonathan Gossage

Personal Email jgossage@xxxxxxxx
Business Email jonathan@xxxxxxxxxxxxxx