JET Enhancement Proposal (JET2)

Introduction

This document proposes enhancements for JET (Java Emitter Templates), the templating tool that is part of the Eclipse Modeling Framework. The enhancement work is tracked by Bugzilla 105966.

Revision History

Date Version Description Author
Aug 26, 2005 0.1 Initial Version Paul Elder
Aug 30, 2005 0.2 Added Introduction and Overview sections, plus minor edits Paul Elder
Aug 31, 2005 0.3 Fix typos Paul Elder
Sep 6, 2005 0.4 Minor edits in response to review meeting Paul Elder

Background

JET is typically used in the implementation of a "code generator". A code-generator is an important component of Model Driven Development (MDD). The goal of MDD is to describe a software system using abstract models (such as EMF/ECORE models or UML models), and then refine and transform these models into code. Although is possible to create abstract models, and manually transform them into code, the real power of MDD comes from automating this process. Such transformations accelerate the MDD process, and result in better code quality. The transformations can capture the "best practices" of experts, and can ensure that a project consistently employes these practices.

However, transformations are not always perfect. Best practices are often dependent on context - what is optimal in one context may be suboptimal in another. Transformations can address this issue by including some mechanism for end-user modification of the code generator. This is frequently done by using "templates" to create artifacts, and allowing users to substitute their own implementations of these templates if necessary. This is the role of JET.

Even though JET solves a significant problem in transformations, it has a number of limitations:

Overview

This document proposes enhancing JET in the following ways:

The scope of the changes proposed are extensive. For this reason, it is suggested that, if implemented, the resulting tool be given a new name. JET2 is proposed, and used throughout the remainder of this document.

Architectural Specifications

This section describes the central architectural features of the JET2 proposal.

The proposed architecture is divided into five parts:

Note that this document does not prescribe interface, class or method names. The purpose of this document is to describe key classes in JET2, and how they interact. The implementer is free to change the names to more appropriate names. However, this document should be updated to reflect such changes.

Compiling the JET2 language to Java

Each JET2 template is compiled into a Java class. This section describes the structure of the resulting Java class, and how JET2 tags interact with template-embedded Java code and template text. Although JET2 is similar to JSP, the JSP compilation practices are not always followed. This is because:

JET2 templates can be parsed into the following Abstract sytnax tree (AST):

The following sections describe how individual nodes in such an AST are compiled into Java code.

Rules for emitting Templates as a whole

  1. Emit one Java class per template.
  2. The java class must have a public 'generate' that accepts the templates arguments and produces the result of template expansion.
  3. The method includes some initialization code, and then the body of the template is emitted to within this method.
  4. The content of the template is written to this method.

The current JET 'generate' method has the signature "String generate(Object argument)", and has the following structure:

public String generate(Object argument) {
    StringBuffer stringBuffer = new StringBuffer();
    emitted code for contained nodes...
    
return stringBuffer.toString();
}

It is proposed that JET2 templates using tags have the following structure:

public void generate(JET2Context context, JET2Writer out) {
    emitted code for contained nodes...
}

Each JET2Context represents an instance of a template execution (or a group of related templates). It contains data associated with theexecution, including the input arguments. The JET2Context is proposed for the following reasons:

The JET2Writer is a generalized stream of characters. It will likely be a specialization of java.io.Writer. The JET2Writer is proposed for the following reasons:

Rules for emitting text

  1. Template text can be written directly to the current writer.

The following is an example:

out.write("some text");

Note that JET currently emits text as final fields in the template class, and then references these fields in calls to write(). However, it is unclear whether this results in any significant benefit in terms of performance (whether in memory usage or speed).

Rules for emitting Java Expressions and Scriplets

  1. Java Expressions and Scriptlets must all me emitted within the 'generate' method. (Variables defined in a scriptlet must be visible to subsequent scriptlets and expressions.)

Standard prolog for emitting Tags

  1. Emit a variable of type tag type, and assign it a new instance of the tag class.
  2. Call the setParent method, passing the parent tag variable, or null if the tag has no parent.
  3. Call the setContext method, passing the tag the current template context.
  4. Initialize tag attributes by calling appropriate tag setXXX() methods.
  5. Call the tag doStartTag() method, and saving the result in local variable. doStartTag() can return one of the following enumerated values EVAL_BODY, SKIP_BODY or EVAL_BODY_BUFFERED.

The following code illustrates this rules:

TagType tag = new TagType();
tag.
setParent(parentTag);
tag.setContext(context);

Rules for emitting tags without a body

  1. Use the standard prolog for emitting Tags.
  2. Emit a call to the tag's doStartTag() method.
  3. Emit a call to the tag's doEndTag() method.

standard prolog
tag.doStartTag();
tag.doEndTag();

Rules for standard tag with a body

A standard tag with a body neither iterates, nor processes the contents from its body. That is, it can emit content at the start of the tag, and the end of the tag, and it can determine whether is body is written or not.

  1. Use the standard prolog for Tags. doStartTag() will return EVAL_BODY or SKIP_BODY.
  2. Emit an if statement testing that doStartTag() did not return SKIP_BODY.
  3. Within the if statement, emit the code to process the body content.
  4. After the if statement, emit a call to doEndTag().

standard prolog
StartResult tagStartResult = tag.doStartTag();
if( tagStartResult != StartResult.SKIP_BODY ) {
    emitted code for body content...
}
tag.doEndTag();

Rules for tags that process the body content

  1. Use the standard prolog for Tags. doStartTag() will return one of EVAL_BODY, SKIP_BODY or EVAL_BODY_BUFFERED.
  2. Emit an if statement testing that doStartTag() did not return SKIP_BODY.
  3. With the if statement, emit a second if statement to test if doStartTag() returned EVAL_BODY_BUFFERED.
  4. Within the second if, allocate a buffered writer, and inform the tag by calling setBufferedWriter() and doInitBody().
  5. Close the second if.
  6. Emit the code for the body
  7. Emit a third if (still with-in the first), which pops the buffered writer if one was pushed earlier.

standard prolog
StartResult tagStartResult = tag.doStartTag();
if( tagStartResult != StartResult.SKIP_BODY ) {
    if( tagStartResult == StartResult.EVAL_BODY_BUFFERED ) {
        out = context.pushBufferedWriter();
        tag.setBufferedWriter((
BufferedWriter)out);
        tag.doInitBody();
    }
    emitted code for body content...

    if( tagStartResult == StartResult.EVAL_BODY_BUFFERED ) {
        out = context.popBufferedWriter();
    }
}
tag.doEndTag();

Rules for iterating tags

  1. Same as above, except that the code emitted by for the body is enclosed in a do {...} while() statement. The while condition calls the tag's doAfterBody() message, and continues if the result is DO_BODY_AGAIN.

standard prolog
StartResult tagStartResult = tag.doStartTag();
if( tagStartResult != StartResult.SKIP_BODY ) {
    if( tagStartResult == StartResult.EVAL_BODY_BUFFERED ) {
        out = context.newBufferedWriter();
        tag.setBodyWriter((BodyWriter)out);
        tag.doInitBody();
    }
    do {
         emitted code for body content...
     } while(tag.doAfterBody() != AfterResult.DO_BODY_AGAIN);
    if( tagStartResult == StartResult.EVAL_BODY_BUFFERED ) {
        out = context.popWriter();
    }
}
tag.doEndTag();

Notes on emitting code for tags

The above rules can result in deeply nested code should a template contain deeply nested tags. Note that the code emitted for any given tag can be refactored into a function call, provided that tag contains not nested Java Scriptlet or Expression. The extracted method could have the following structure:

private void tag(JET2Context context, JET2Writer out, Tag parentTag) {
    emitted code for the tag
}

In code emitted for a tag with its enclosing body would be simply:

tag(context, out, parentTag);

Tag Library Interfaces and Classes

The above discussion suggests tags have the following associated interfaces.

JET2 Parser

The current JET parser classes appear to be flexible enough to handle the expansion of the JET2 language to contain additional directives and most importantly, tag. The proposal is to:

TODO: Some mechanism to select between the compilers by some sort of runtime parameter is desireable, much the way this is possible with the JDT. Details to be worked out.

JET2 Project Structure

It is proposed that as much as possible, JET2 rely on existing Eclipse infrastructure to compile generated Java classes into code. In the subsequent section, it is proposed that JET2 dynamic template loading be accomplished by using the OSGi dynamic loading/unloading of Bundles. This implies that the JET2 project builder must ultimately produce a valid OSGI bundle.

It is proposed that projects containing dynamicly loadable templates (such as those that customize the behaviour of the EMF code generator, and the proposed transformations) have a specific Eclipse Nature that builds the JET2 templates into a loadable bundle. Such a nature would require the following builders:

This has a number of advantages over the current practice of creating a .JETEmitters project:

The main disadvantage of this approach is that builder artifacts (including generated .java and .class files) will appear in the project containing the templates. However, this can be mitigated by the use of filters, much as the Package Explorer filters out 'bin' directories by default.

Loading and executing JET2 templates within the development workspace

As the previous section alluded, it is proposed that the dynamic loading of JET2 templates be accomplished via the OSGi framework. This section docments the details of doing this.