Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[Dltk-dev] Xam/Xom - CodeGear proposal to the language independent API for querying and manipulating DLTK supported languages.

Hi folks,


Below is proposed concept from CodeGear. Talks on this was postponed until 0.9 release, and now it’s a time to recall ideas about refactoring (and code manipulation in general) support for DLTK languages. Comments and ideas are very welcome.


Kind Regards,



From: Mark Howe [mailto:Mark.Howe@xxxxxxxxxxxx]
Sent: Tuesday, March 20, 2007 6:20 AM
To: Andrey Platov; Mikhail Kalugin
Cc: Joe McGlynn; Shelby Sanders
Subject: Xam/Xom concept


This email describes the concept for a two way code editing API, Xam/Xom for DLTK. To give some background I've also attached JamJom documentation from JBuilder, the equivalent API for Java. I've also attached a description of why we created Jam/Jom as background to support the reasons for considering Xam/Xom.


Xam/Xom would be the language independent API for querying and manipulating DLTK supported languages. There could be a common Xam/Xom API or it could just be the common infrastructure that the individual language API's used for most of the manipulation. The rest of this description will describe the language specific API Ram/Rom, the Ruby version of Xam/Xom.


The basic idea is to create a reasonably compact and simple API to allow querying and manipulating Ruby source files. Ram/Rom would not replace any API's or functionality already defined in DLTK, instead would be a layer built on top of the tools already available in DLTK.


Ram is the high level read only abstraction of Ruby. Ram would be a fairly thin layer since all the functionality is already in DLTK. It would just consolidate that functionality into a single API.


Ram would have a central access point per project or workspace to query for objects representing the logical model of Ruby. The housekeeping of setting up search requests, contexts etc would be hidden from the API user, other than requiring some artifact such as IDLTKProject. An example of how this would differ than what already exists, is how it could encapsulated all information about a class. For instance a class "ExampleClass", that is defined across multiple files would be represented by a single entity (say a RubyClass). "ExampleClass" would know internally where each method is actually defined (i.e. would know about it's ISourceModule). The API would be kept free of any artifacts that don't represent the languages logical model, i.e. fields, methods, super classes etc.


RamFactory factory = RamFactory.instance(project);
RamClass ramClass = factory.getRubyClass("ActiveRecord::Base");
//some examples
ramClass.getMethods(); //returns all methods regardless of which source file and includes ancestors
RamMethod ramMethod = ramClass.getDeclaredMethod(String methodName); //only looks for method name that is declared directly by ramClass etc.


Rom is the detailed AST writable model of Ruby. It would be based on DLTK's ASTNode. Rom would attempt to make the majority of two way editing tasks as easy as possible, but not hinder more complicated manipulation of code.


A Rom would be instantiated from a Ram.


RomClass romClass = ramClass.getRamClass();
RomMethod romMethod = ramMethod.getRomMethod();


A Rom represents a physical artifact, i.e. a class or method in a file, has position information etc. Creating and inserting methods and fields would be easy with Rom. Every Rom would have a RomPosition that would be used when inserting or appending new Roms. For instance adding a method would be something like


RomMethod newRomMethod = RomFactory.createMethod(...) -- or maybe romClass.createMethod(..) since it has context
romClass.addMethod(existingRomMethod, newRomMethod);


where the existing rom method would provide the position information required by JomClass to add the new method. The same would be true for blocks, expressions and statements.


Changes would be collected until committed. Changes would also be able to be rolled back. For instance refactorings would have a change set of Rom's and would be committed collectively. If anything failed the changes would be rolled back.


 That's the basic idea,  

 Cheers Mark 



Title: JBuilder concepts: Jam/JOM

JBuilder JAM/JOM concepts

The JAM/JOM subsystems are a set of interfaces and classes used to parse and generate Java code. They can be used separately, but the most power and flexibility are achieved when used together. The main use is the parsing of source or class files to extract information about their contents. Examples of this include the various wizards in the object gallery (File|New) or the UI Designer.

JAM/JOM supersedes JOT as the Two-Way Tools backbone. The JOT package is deprecated; JBuilder 2005 ships with it solely in order not to break existing JOT-dependent OpenTools.

Features and subsystems

The heart of JAM/JOM is a series of interfaces and classes that describe the Java language in detail.

JAM (Java Abstraction Model) is a read-only layer that provides information about Java classes. The information only includes classes, methods and fields.

JOM (Java Object Model) is a read/write layer that deals with files and the constructs within those files. While JAM only abstracts classes, methods and fields, JOM deals with the details of source files such as position and constructs (statements, blocks, expressions etc).

The two API's work together but are kept separate for a couple of reasons. The main distinction is that JAM deals with the class as its main unit of abstraction, whereas JOM deals with the file as its main level of abstraction. The other big distinction is that, since JAM is not concerned with the writing of source code, it can use either the class file, source file, or both. Jom is only concerned with source code. This frees the user from having to worry about what representation the file comes from; if it's JAM the user doesn't care, and if it's JOM the user knows they are working with the source code.

A typical usage of JAM and JOM is using JAM to find the classes, methods and fields of interest and then converting to JOM when reading and writing source code is required.

JAM is a read-only layer that is basically a reflection layer. It hides details of whether the information comes from class or source. JAM chooses the correct backing based on what is being asked for and what is the fastest correct way of providing that information.

Accessing JAM/JOM

When you are working with JAM/JOM, you are either working with objects (classes, methods, fields) from a file that exists already, or you are creating a new file. If the file exists, it's usually part of your project (say, a selected node). If you are creating a new file, it is usually done in the context of the current project.

Accessing JAM

To access the JAM subsystem, use the static factory method instance(Project) of the JamFactoryobject, as this example:

Project project; // assume a valid, non-null instance of Project
JamFactory factory = JamFactory.instance(project);
JamClass jamClass = factory.getClass("com.borland.jbuilder.jam.TestJam");

The above code will return a jamClass if the class is found on the project paths.

JamFactory will return JamClass from class information if the class is up to date; otherwise it will return a JamClass with the information from the source file. If the information was from class information and the file is edited but not recompiled, then JAM will switch to using the source information when requested for new information.

Example: JAM from JOM alternative

An alternative way to obtain a JAM object from JOM is to use the getJomXXX() method on the relevant jom objects.

JamMethod jamMethod = jomMethod.getJamMethod();

Accessing JOM

To access the JOM subsystem use the static factory method of instance(project) JomFileManager;,

Project project; // assume a valid, non-null instance of Project

JomFileManager jomManager = JomFileManager.instance(project);

A typical usage is to get a JomFile for an existing Url file. This could be done like this:


Url fileUrl = new Url("c:/test/");

if (fileUrl.exists()) {
  // The getJomFile method below will return a JomFile for an existing file.

  // Obtains a JomFile for a file. (If the fileUrl doesn't exist a new will be created.)
  JomFile jomFile = jomManager.getJomFile(fileUrl); 
// Use JOM to modify the file:



Another usage is to create the file if it doesn't exist:


fileUrl = new Url("c:/test/");
// Obtains a  JomFile for an existing file or creates a new JomFile object:
JomFile jomFile = jomManager.getJomFile(fileUrl); 

// Use JOM to modify the file:

jomFile.commit class=(true);


The content of the source code in the file will not change until one jomFile.commit(...); method is called. JomFile should not be used after a commit on it is performed. A JomFile is intended to be a short-life object. You can use it to generate source code structures (joms) and commit the file. After this point, references to its JomFile(or any of its children) should not be kept around.

If the source code is modified by other thread/process on disk (or in the VFS buffer), access to jomobjects in this JomFile will throw a ConcurentModificationError.

Example: JOM object from JAM alternative

An alternative way to obtain a JOM object from JAM is to use the getJomXXX() method on the relevant jamobjects.

JomMethod  jomMethod = jamMethod.getJomMethod();

How JAM sees a class

JAM views a class in a manner similar to how the java.lang.reflect package does. A representation of a Class exists (either from source or from a compiled unit). This Class is composed of member items: fields and methods. In many cases, you'll find working with these higher-level items familiar if you've previously worked with the Reflection API. Reflection provides access to the methods, constructors, and fields of a class. You can glean information on the class itself by using java.lang.Class directly.

Relevant JAM objects

  • JamClass
    JamClass has methods for discovering its superclasses, interfaces, methods and fields. The naming conventions are similar to the Reflection API in the JDK. For instance, to find out all the methods that are declared in the current class, use getDeclaredMethods(), but to find all methods including those of its ancestors, use getMethods(). This convention does not apply to constructors, since only the constructors that are declared in the current class are available. To get the declared constructors, use getConstructors().
  • JamMethod
    JamMethod provides information about the method you are interested in: return type, the parameter types and names, plus all the modifier information.
  • JamField
    JamField provides information about the field you are interested in: type, name and modifiers.
  • JamType
    Abstracts the notion of "type" and is used wherever type information is required in JAM and JOM. JamType represents both primitive types and Object types.

    The only public subclass of JamType is JamMethodType, which represents a method signature. All JamTypes are intended to represent fully qualified type information. JamType performs all conversions between qualified type names and signatures. Complex types are also represented by JamType, such as arrays, multi-dimension arrays, etc.

    Future changes to the language, such as the addition of generics and enumerated types, will be represented in JamType as well. Code written with JAM and JOM will be able to take advantage of these features, often without any changes.

How JOM sees a class

Using JOM, you can do all that and go much deeper than using reflection alone. Jom has constructs which wrap around the text in the source code, providing for an easy programmatic access to the source code. It also adds a JomFile object, which is the representation for the file that contains the source code. These detail items follow the language guidelines laid out on the Java Language Specification (JLS.)

  • File: contains class(es) and items that aren't directly a part of a class such as package and import statements. This is represented by JomFile.
  • Class: contains fields, methods, and inner classes. This is represented by JomClass.
  • Method: has a code block. This is represented by JomMethod.
  • Code blocks: contains statements, other methods, and so on. It is represented by JomBlock and its contents are JomStatements or descendants of that class.

JOM continues where reflection stops by allowing you to work with the individual statements or expressions that make up the heart of most Java code.

Using JAM to read Java

Once a JamClass is obtained as previously described, it can be queried for methods, fields, superclasses and subclasses.

JAM introduces JamType which represents a primitive type or an Objec type. It also recursively represents arrays of a type. There are predefined types for the primitives and String, i.e. JamType.INT or JamType.String. The method asArray() returns a JamType for the type, for instance to get a type for int[].

You can create a type from text (i.e. full type name) or from the type signature (how it's represented in the class files). Here are several ways of creating a JamType:

JamType type;

//Predefined types:
type = JamType.INT;
type = JamType.STRING;

//or specifying the type from the type name (long name for objects):
type = JamType.fromText("short");
type = JamType.fromText("java.lang.String");

//or from the class files signature:
type = JamType.fromSignature("Ljava/lang/String;");
type = JamType.fromSignature("B"); //Byte primitive

//creating arrays:
type = JamType.fromText("int[]");
type = JamType.INT.asArray();
type = type.asArray() //Whatever type is now type[]

A difficult problem that JAM addresses is finding methods. Methods have signatures as well as names. Class files deal with the signature of a method, and the compiler and sources deal with the parameter types. Both represent the same thing, but in a completely different way. JAM provides a special JamType, JamMethodType, which represents the method signature (not including the name).

JamMethodType methodType;

//Predefined types:
JamMethodType methodType = new JamMethodType();
//void x();

//void x(int i);

//void x(int i, String s);

//Bar x(int i, String s);

Once you have defined the method type, it's easy to find a method from a JamClass.

JamClass  jamClass  = factory.getClass("com.borland.jbuilder.jam.TestJam");
JamMethod jamMethod = jamClass.getMethod("x", methodType); 

Finding a field is similar, except only the name is required:

JamClass jamClass = factory.getClass("com.borland.jbuilder.jam.TestJam");
JamField jamField = jamClass.getField("foobar"); 

Using JOM to read Java

JOM can parse Java source files. The methods specified by JOM for parsing Java use the naming convention get<XYZ> to extract whatever XYZ represents. These getter methods are generally used to parse the nested nature of Java code and usually return another item that is represented by a JOM.

During parsing, everything works from the outside in. At the outermost level (a JomFile), the parsing procedure is simple. A JomFile contains one or more classes (represented by a JomClass), and each of those classes is composed of variables and methods. Usually, you have a loop that iterates through all the classes in the file. From each of those classes, you can get an array of methods defined in that class and loop through each of these. The implementation of these methods makes things a bit more complicated. For example, a class can have many methods and each method can be composed of several statements, and each of those statements can be a method that is composed of many statements, and so on. At this level, recursion is common.

The base object in the JOM hierarchy is the Jom object. Every Java Language (JL) construct derives from this object. In order to remove any Jom you just need to call the remove method on it.

Every Jom also has an associated beforeComment/afterComment. (In order to obtain them you need to call the appropriate getXXX() methods.) The underlined Jom object of a comment is JomComment. It can be JavaDoc, standard comment, or EOL comment. JomComment allows you to modify the comment text.

JOM has a very high level of granularity. Children of Jom objects (JomMethodsfor JomClass, JomStatement for JomBlock etc.) are created as lazily as possible. The children Joms are created on-demand (class=jomFile.getJom class=(<JomPosition>).) Another way to populate recursively all the children of a Jom(JomClass, JomMethod, JomBlock, JomStatement, etc.) is to call getDeepChildren() method of a JomContainer. Yet another way is to use the FillExistingChildrenVisitor object, as shown below:

Jom jomToFill = <some Jom>;

com.borland.jbuilder.jom.FillExistingChildrenVisitor fillVisitor 
        = new com.borland.jbuilder.jom.FillExistingChildrenVisitor();


Using JomContainer's getChildren class=(boolean includePendingRemovals) returns all the children. (Whether to include the one removed from the source is specified by the includePendingRemovals parameter.)

JOM allows you to parse down to the atomic level. Consider a simple example of a for loop:

public void forLoop(){

  // a bunch of statements in the method body...
  for(int i = 0; i < 5; i++){     
    System.out.println("Hello Jom!");
  // even more statements in the method body...

The for statement is defined in section 14.12 of the Java Language Specification, which states:

"The for statement executes some initialization code, then executes an _expression_, a Statement and some update code repeatedly until the value of the _expression_ is false."

In our example, the initialization code is: int i = 0. The _expression_ is: i < 5. The statement that is executed is the println() inside the code block of the for loop. The update code is i++.

As the Java Language Specification states, a for loop is a type of statement. This means, at a high level, the for loop in the code would be parsed as being of type JomStatement. This is very generic as many things are JomStatements, so if you are working with a bunch of statements with JOM, you would determine if the type of statement you had encountered was really an instance of JomForLoop using instanceof.

JomForLoop breaks the for statement down even further. The _expression_ portion of a for statement describes the condition that causes the loop to terminate. Any valid Java _expression_ can be used here, ranging from a simple value check to a complex _expression_. JomForLoop.getCondJom() returns a JomExpression object that you could further interrogate to determine what type of an _expression_ you had and what it was composed of, and so on.

Use the following code to see how a simple class is broken down and parsed with JOM:

package com.borland.samples.jom;

import java.awt.*;

public class JotExample extends Component{
   static final int PI = 4;
  public JomExample(){
    // xtor implementation goes here
  public boolean doesItRock(int x){
    int ctr = x;	

    for(int i = 0; i < ctr; i++){        
      System.out.println("JBuilder Rocks!");
      System.out.println("Oh yes, it does");
    return true;

For this example, we have an instance of JomFile named jsf that represents this file. To get the package statement to use this code:

JomPackage pkg = jsf.getPackage();

To get the import statements from the file, use this code and iterate through the array:

String [] imports = jsf.getImports();

Note: You can also use the JomContainer (such as JomFile) methods getChildren(boolean includePendingRemovals) and iterate through the children checking for instanceof JomImport to get the import statements.

To get the class(es) from the file, use this code:

JomClass [] classes = jsf.getClasses();

In this example, there's only one class file represented, so you know that your class is the first element in the array of classes returned by JomFile.getClasses(). Therefore, you can use this code to get the specific instance of JomClass:

JomClass jc = classes[0];

From the JomClass you can extract information about the class definition as well as its member fields and methods. The basic pattern remains the same; that is, a getter returns a value or object that you can interrogate further for more information.

// info about the class declaration
JomModifiers mod = class=jc.getJomModifiers();
JomExtends super = jc.findJomExtends();
JomTypeList imps = findJomImplements

These methods allow you to find out the modifiers for the class, the super class and the List of interfaces implemented by the class. You could get more information about the superclass by using the methods in JAM to get a JamClass for the JomExtends object.

Where modifiers are used in Java, as, for example a public class or a static final variable, JOM allows access through a getJomModifiers class=() method. getJomModifiers class=() returns an object, which provides methods for manipulating the modifiers on any JL construct.

If you continue with the parsing of the class, you can get a list of fields and a list of methods contained in the class. Again, you can further parse the results to get to the atomic portions of the fields and methods:

// details on the contents of the class
JomField  [] fields  = jc.getFields();
JomMethod [] methods = jc.getMethods();

For a more complete example of using JOM to parse source, see the sample This sample takes the selected file and gathers information about classes and methods using JOM. This information is sent to the message window inside the IDE.

An example of using JAM/JOM to parse source and/or classes is the sample  com.borland.samples.opentools.jom.packagetree.PackageTree. This sample takes the selected node in the IDE, determines its package, and then builds a hierarchy tree for all the classes in that package.

Using JOM to write Java source

Jommodels the source file and its contents. The starting component for Jomis JomFile. JomFileis obtained from the projects JomFileManageror from a JamClass, JamMethod or JamField. Jom is a high level representation of source code and provides many features to simplify the development of code.

Jomprovides a set of advanced features for writing two way code, auto formatting, auto importing, comment support, references, renaming within file and position encapsulation. Also, Jom provides a complete mapping of language constructs to logical Jom constructs.

Some of the important JOM constructs are JomFile, Jom, JomPosition, JomContainer and JomReference.

  • JomFile
    JomFile represents a single source file and provides a logical mapping of the source code. Source code can be safely inspected and modified. Retrieve a JomFile from the JomManagerwith an Url or fully qualified class name, or from a Jam that represents a class, method, or field in a source file.

    JomFile is a JomContainerwhich holds other Joms. The two high level components in a JomFile are a JomClass and a JomImport. JomClass is the container for most other Jom types. When the modifications to a JomFile are complete, a commit writes the changes including (optionally) formatting, importing and adding Javadoc.

    A simple example of using JomFile is getting a class and then committing it. This simple program will create the class if it does not already exist:
Url fileUrl = manager.getUrlFromClassName("root.subpackage.MyClass1");
JomFile file = manager.getJomFile(fileUrl);
JomClass jomClass = file.getPrimaryClass();
  • JomPosition
    Contains the information about positions of Joms within a source file. It translates between compiler positions, buffer positions and line/column positions. All Joms have positions associated with them. Joms can be tested if they are before or after other Joms and using a JomContainer. A JomContaineris any Jom that contains another Jom's classes, methods, blocks etc. Joms can be used as position markers within other Joms.
  • JomContainer
    Holds other Joms. Has methods for inserting before or after other Joms and a general method for appending to the container.
  • JomFactory
    This is a factory object used for creation of Joms.

When writing source with JOM, you deal with much higher level items like methods or statements, instead of the very fine-grained items available to you when reading source files with JOM. The methods specified by JOM for writing Java code use the naming convention append<XYZ>, insert<XYZ>, add<XYZ> and set<XYZ> where XYZ represents the item you are creating or modifying. For example, JomClass.getBody class=().append() adds a child to a given class. class=JomMethod.getJomModifiers class=().setFinal(boolean bValue) sets the modifier's final bit for that method.

A convenience method in JomFile is JomClass getPrimaryClass(). This method returns the primary class of a file (the public class, the first top level class.) If the JomFile is empty (for instance, if just being created), this method will return a new JomClass, ready to be edited (using JOM) and later committed.

Notes: JOM is very forgiving of being told to write bad code and will write anything you have told it to write.

JOM expects all the type strings (return types, parameter types etc.) to be fully qualified. The types and appropriate import statements will be automatically generated based on the user code generation properties.

Example: writing code using JOM

Let's create the same class that was used in the parsing section, but we'll build it up in smaller chunks. This simple "starting point" of the class looks like:

package com.borland.samples.jom;

import java.awt.Component;

public class JomExample extends Component{


Again, we begin with the outermost item, the source file itself. For JOM, the source file is represented by a JomFile that encapsulates the package statements, the imports, and the class declaration.

Assuming you have an instance of a JomFile named jsf, use the following methods to create the class:

jomClass = JomFactory.createJomClass("com.borland.samples.jom.JomExample", jomFile);

JomExtends jomExtends = jomClass.getJomExtends();

jomExtends.setName class=("java.awt.Component"); // NORES

JomModifiers mod = class=jomClass.getJomModifiers();

mod.setPublic class=();

To finish, you need to add the rest of the content (a member variable, a constructor and a method) to the class. This is what the code that is generated will look like:

package com.borland.samples.jom;

import java.awt.Component;

public class JomExample extends Component {
  static final int PI = 4;

  public JomExample() {
    // xtor implementation goes here...

  public boolean doesItRock(int x){
    int ctr = x;

    for(int i = 0; i < ctr; i++) {
      System.out.println("JBuilder Rocks!");
      System.out.println("Oh yes, it does");

  return true;

To add the field, use code like this:

JomField jf = jcs.getBody().append(JomFactory.createJomField("PI",JamType.INT);

jf.setInitialValue class=("4");

To make the field PI both staticand final, do this:

jf.getJomModifiers class=().setStatic(true);
jf.getJomModifiers class=().setFinal(true);

To create the constructor and make it public, do this:

JomConstructor xtor = jcs.append(JomFactory.createJomConstructor(jcs, JomParameterList.EMPTY));


To set up the "shell" of the method with a parameter x, use this code:

JomParameterList parList = JomFactory.createJomParameterList();

JomParameter parameter = JomFactory.createJomParameter("x", JamType.INT);
jomMethod = JomFactory.createJomMethod("doesItRock", parList, JamType.BOOLEAN);

mod = jomMethod.getJomModifiers();


Add the local variable ctrand assign it the value of the parameter passed to the method:

JomVariable jv = JomFactory.createJomVariable("ctr", JamType.INT);

jv.setInitialValue class=("x");

jomMethod.append class=(jv);

Add in the for statement. There are two ways to create the JomForLoop:

  • You could create the specific statement (JomForLoop) and fill its children Joms appropriately, or
  • You could simply generate all the code in new, arbitrary JomStatement objects and add it to the method. (Note that JomText, JomStatement and JomExpression can be used to hold any text that needs to be generated in the source code. If the text has more than 255 characters in a row, each row over 255 characters must be broken into smaller rows by JomCommitter.NEWLINE.)

In a JomStatement, you can put any arbitrary string and it will be generated in the source code. In the following JomStatement object, we are actually generating two statements: The ForLoopand the return statement:

StringBuffer statementText = new StringBuffer();

statementText.append("for(int i = 0; i < ctr; i++) {");
statementText.append("System.out.println(\"JBuilder Rocks!\");");
statementText.append("System.out.println(\"Oh yes, it does");");
statementText.append("return true;");


For a more complete example of using JOM to write source, see the sample com.borland.samples.opentools.jom.write.WritingSource. This sample generates a simple class file. Another sample of using JOM to read and write is com.borland.samples.opentools.jom.readwrite.Commenter. This sample takes the selected source node and adds Javadoc comments for top level classes and methods in classes. It only adds them if there isn't a comment there already.

Using JAM and JOM together

Use JAM to discover the main construct you are interested in. Once you have the appropriate JAM object, you can directly convert to a JOM for manipulating source code.

For instance, assume you have a method named bar that takes an int and returns a Stringin the class com.foobar.Foo. There are 4 simple steps:

  1. Get the JamFactory for the project.
  2. From the factory, get the JamClass.
  3. Ask the JamClass for the JamMethod, passing in a JamMethodType and the method name.
  4. Then get the JomMethod from the JamMethod.

Notice that JAM deals in logical constructs entirely -- you don't need to know where files reside on the source path, just the full class name. JamType and JamMethodTypealso simplify the process of defining the method you are searching for.

Project project; // assume a valid, non-null instance of Project

JamFactory factory = class=JamFactory.instance(project);

JamClass jamClass = factory.getClass("com.foobar.Foo");

JamMethodType type = new class=JamMethodType();

type.setReturnType class=(JamType.STRING);

type.addParameter class=(JamType.INT);

JamMethod jamMethod = jamClass.getMethod("bar", type);

JomMethod jomMethod = jamMethod.getJomMethod();


Attachment: JamJomBackground.pdf
Description: Adobe PDF document

Back to the top