Re: [stellation-res] A database proposal, for fixing 22135.

["Mark C. Chu-Carroll" <mcc@xxxxxxxxxxxxxx>]

Prefix to the discussion... This is a *really* interesting discussion. If 
there are lurkers on the list with anything to contribute, *please* jump
in. This is a discussion with the potential to dramatically change the
future of the project, so speak up now if you have anything to say!

On Friday 09 August 2002 01:35 pm, Florin Iucha wrote:
> On Fri, Aug 09, 2002 at 08:11:12AM +0000, Mark C. Chu-Carroll wrote:
> > On Friday 09 August 2002 02:45 am, Florin Iucha wrote:
> > > On Sun, Aug 04, 2002 at 08:10:13PM -0400, Mark C. Chu-Carroll wrote:
> > > [snip]
> > >
> > > > OK. Enough background. Here's what I'd like to do.
> > > >
> > > > I'd like to separate the tags from the lines in the database. And I'd
> > > > also like to burst the tags out, so that instead of bunding the
> > > > information about what versions a given line is part of into a tag
> > > > string, it gets broken out into a bunch of lines in a different
> > > > table.
> > > >
> > > > So... The Texts table becomes two tables: TextLines, and
> > > > TextVersions. Each row in TextLines is an artifactID, and a line ID,
> > > > and the line string.
> > > >
> > > > TextVersions is a list of (linenumber, LineID, VersionIDs). There is
> > > > an entry in TextVersions for a LineID,VersionID if the line is a
> > > > member of the version.
> > >
> > > There is something that bugs me for a couple of days...
> > >
> > > Stellation is plugin based: each artifact is [supposed to be]
> > > manipulated by a plugin that knows how to parse/merge/diff it.
> > >
> > > If that't the case, why is the database schema assuming that text files
> > > can be _meaningfully_ sliced into lines of text?
> >
> > That's the database schema for *text* artifacts.
>
> Then I think it should be marked as such. It is not easy to see which
> tables are for the core and which for the plugins. And it is not
> documented either.

The documentation should be clearer about that. In typical hacker
fashion, the documentation has lagged behind the implementation. We
started with an architecture without agents, and set up the table structure.
Then I realized that the artifact types could be abstracted out, and
changed the code to use that. But the basic table structure documentation
never got updated to reflect the fact that there are two sorts of tables
in the database: the primary tables that reflect the intrinsic structure
of the repository, and the extension tables that implement an artifact
type. When I get a chance, I'll put that into bugzilla as a todo.

For the database itself, it's hard enough to get descriptive names
for things without running into either reserved words or name length
limits. I don't think it's a good idea to start requiring extensions to
do something like use a unique prefix for thier tables.

What we could do is add a responsibility table, where each extension
lists the tables that it's responsible for... That would let someone
browsing the database see what tables belong to what components.
Something like:
	TABLE Reponsibilities (
		ExtName VARCHAR(100), -- the name of the extension
		TableName	VARCHAR(100) -- the name of a table managed
					-- by the extension
       )
		
> > The idea is that as you add artifact types to the system, you provide
> > a database scheme for that artifact type. The ArtifactAgent (the
> > plugin type that provides a new artifact type implementation)
> > has methods to create the database tables needed to support
> > the artifact type, to retreive and store artifacts of the type from and
> > to the database, etc.
> >
> > So what we're talking about is the implementation of TextArtifactAgent.
> >
> > If you didn't want to use table layout that we're discussing, you'd
> > implement a new agent that created tables to store things however
> > you choose.
> >
> > For instance, we've been discussing supporting XML schema's
> > with another IBM group. They don't want the kind of storage that
> > we do for simple text; they want sliced-and-diced storage of meaningful
> > chunks of the schema. So they'd use something more like the list of
> > LOBs that you suggest.
> >
> > > Shouldn't the stellation core store the artifact data in whatever
> > > format it pleases (simple linked list of blobs containing data/delta or
> > > files stored at filesystem lever or ...) and serve it to the
> > > appropriate plugin as a big chunk'o'bits?
> >
> > But then we're dictating to the artifact type implementors that they
> > have to use that BLOB implementation, even if there's a reasonable,
> > efficient, easy to read and manipulate representation of their type
> > in a set of tables. Our approach is to not dictate storage
> > at all, except that it needs to be in the database.
>
> Nope. The core will have an interface to the raw data:
>    appendData(artifact, data) and
>    getDataSize(artifact)
>    readData(artifact, offset, size)
> or
>    getChunkIterator(artifact)
>    readChunk(iterator)
>    appendChunk(artifact, data)
> and all the plugins would use this interface to store their data.
>
> I am not really thrilled about storing the artifact data in the database
> either. Having a 10K line create a 10K * versions rows in the Text
> table is quite wastefull. Having a 10K lines XML document create a 25K
> (parsed, each <tag> takes one row) * versions is even worse.

This is exactly the concern that I started with when I proposed the
new texts structure. (On the other hand, I doubt that the right
structure for handling XML will be strict tag separation; I suspect
that they'd want something much more compact than current
XML syntax. I don't know what the right approach would be; I haven't
yet spent any time thinking about it.) The reason for the original
tagged text structure was trying to avoid this kind of storage
explosion. I'm pretty much convinced that the compact tag storage
is probably wrong. For now, I'm continuing the fully tabular
text representation. If we come up with a better model, I'll
happily replace it.

One minor, but important point to mention here is that we're aiming
towards fine-grained artifacts, where things like files are decomposed
into smaller units. You only generate new lines entries for new artifact 
versions; and the artifacts are small things. When you check in a new
version of that 10,000 line file, you're not changing every
artifact in that file. You'll only be storing new line entries for the
specific members of that file that changed. 

I really hate the idea of using a universal BLOB mechanism
like you're suggesting, for several reasons.

(1) A lot of data types are highly structured, and we can capture that
structure in database tables. It would be a shame to lose that by
pushing everything into BLOBs, and then reparsing the structured
data out of the BLOBs. 

(2) Keeping things in tables means that the whole database is
readable both by humans, and by other programs. For debugging,
it's been unbelievably valuable to go into the database and look at
it. For other stuff, we've got people interested in doing things like
bug management, change impact prediction, and various kinds of
change history analysis. Things like that don't work particularly well
through the repository interface. But if the data is in tables, they
can go ahead and mine the database for all sorts of interesting data.

(3) Table storage gives us exploitable structure. This sounds like
a repeat of point one, but I think it's getting at something different. Even
when the underlying data doesn't have a strong structure (like, for
instance, text; text structure is just a list of lines, not a particularly big
deal structure wise). But by capturing things into database tables, it means
that we can structure things so that the database query engine itself
can do things like efficient version retrieval. Instead of trying to optimize
that ourselves for each new data type in terms of blob operations, we
can exploit all of the work that goes on in the database community on
optimization of database operations.

(4) BLOB operations are often not truly transactional. (For example,
LOB operations are not fully transactional in DB2.) The importance
of transactionality cannot be overstated. If the system isn't fully
transactional, your data is not safe.

(5) BLOB operations are not very portable. We've already suffered
greatly using BLOBs for data artifacts. Postgres BLOBs don't implement
the JDBC blob interface. (They used to; it was dropped in 7.2. Now,
there's a dreadful "bytea" type that implements JDBC blob operations
extremely poorly, and true BLOBs can only be handled through a 
postgres specific LOB API.) Different databases handle BLOBs in
subtly different ways. Building out entire storage mechanism around
BLOBs is begging for incredible pain and suffering. 

> If you don't decompose XML at the tag/text level when you store it in
> the database, you will have to parse it anyway when you want to
> diff/merge it.

It's true on a naive level that you're either pushing things into a
very verbose tabular structure, or you need to do some parsing. That
doesn't mean that there is no happy medium point, where the XML
is somehow chunked and interlinked using the database structure. 

There are a lot of representations of ASTs (and XML is just an
AST syntax) that are really clever, and that make diffs and merges
extremely efficient. I'd like implementors to have the option of selecting
one of those representations and using the database to exploit it.

> Why not store the artifact data in the file system and keep a cookie in
> the database?

Because file systems are non-transactional. Transactionality is one
of the most under-appreciated, under-estimated pieces of functionality
that you get from using a database. Transactionality is *the* key to keeping 
your code safe in the repository.

Databases give you transactionality for free. You *can* implement 
transactionality in a filesystem. But to do it *right* is very hard. To
be *sure* that you've got it completely right - that you haven't missed
any end-cases, that there's no scenario where an interrupted operation
will corrupt data in your filesystem - that's very, very hard. For every
hundred people who've tried to implement transactionality, more than
99 of them get it wrong. And you don't know that you got it wrong until
some data gets lost.

I don't trust anyone involved in this project to really implement
transactionality *right*. That's not a dig at anyone involved in
the project. It's a reflection of how hard the problem is - and none
of us are experts on the subject.

> Checkin would mean
>    - file upload into a staging area - should not fail
>    - moving files into the permanent structure and create database
>      records for the artifacts - should not fail

"mv" is not guaranteed to be an atomic operation in Unix. How do you 
make sure that when you move it into the permanent structure that you didn't
mess up?

	-Mark

>    - create the database metadata records - renames, deletions,
>      additions
>    - queue the new data aditions for archival/deltification...
>
> If the third step fails, a scrubber process can remove the orphaned
> files from the file system and their database identification records.
>
> florin

-- 
Mark Craig Chu-Carroll,  IBM T.J. Watson Research Center  
*** The Stellation project: Advanced SCM for Collaboration
***		http://www.eclipse.org/stellation
*** Work Email: mcc@xxxxxxxxxxxxxx  ------- Personal Email: markcc@xxxxxxxxxxx