| [Dltk-dev] Xam/Xom - CodeGear proposal to the language independent API for querying and manipulating DLTK supported languages. |
|
Hi folks, Below is proposed concept from CodeGear. Talks on this was postponed
until 0.9 release, and now it’s a time to recall ideas about refactoring
(and code manipulation in general) support for DLTK languages. Comments and
ideas are very welcome. Kind Regards, Andrey From: Mark Howe [mailto:Mark.Howe@xxxxxxxxxxxx] This email describes
the concept for a two way code editing API, Xam/Xom for DLTK. To give some
background I've also attached JamJom documentation from JBuilder, the
equivalent API for Java. I've also attached a description of why we
created Jam/Jom as background to support the reasons for considering Xam/Xom. Xam/Xom
would be the language independent API for querying and manipulating DLTK
supported languages. There could be a common Xam/Xom API or it could just be
the common infrastructure that the individual language API's used for most of
the manipulation. The rest of this description will describe the language
specific API Ram/Rom, the Ruby version of Xam/Xom. The
basic idea is to create a reasonably compact and simple API to allow querying
and manipulating Ruby source files. Ram/Rom would not replace any API's or
functionality already defined in DLTK, instead would be a layer built on top of
the tools already available in DLTK. Ram
is the high level read only abstraction of Ruby. Ram would be a fairly thin
layer since all the functionality is already in DLTK. It would just consolidate
that functionality into a single API. Ram
would have a central access point per project or workspace to query for objects
representing the logical model of Ruby. The housekeeping of setting up search
requests, contexts etc would be hidden from the API user, other than requiring
some artifact such as IDLTKProject. An example of how this would differ than
what already exists, is how it could encapsulated all information about a
class. For instance a class "ExampleClass", that is defined across
multiple files would be represented by a single entity (say a RubyClass).
"ExampleClass" would know internally where each method is actually
defined (i.e. would know about it's ISourceModule). The API would be kept free
of any artifacts that don't represent the languages logical model, i.e. fields,
methods, super classes etc. RamFactory
factory = RamFactory.instance(project); Rom
is the detailed AST writable model of Ruby. It would be based on DLTK's
ASTNode. Rom would attempt to make the majority of two way editing tasks as
easy as possible, but not hinder more complicated manipulation of code. A
Rom would be instantiated from a Ram. RomClass
romClass = ramClass.getRamClass(); A
Rom represents a physical artifact, i.e. a class or method in a file, has
position information etc. Creating and inserting methods and fields would be
easy with Rom. Every Rom would have a RomPosition that would be used when
inserting or appending new Roms. For instance adding a method would be
something like RomMethod
newRomMethod = RomFactory.createMethod(...) -- or maybe
romClass.createMethod(..) since it has context where
the existing rom method would provide the position information required by
JomClass to add the new method. The same would be true for blocks, expressions
and statements. Changes
would be collected until committed. Changes would also be able to be rolled
back. For instance refactorings would have a change set of Rom's and would be
committed collectively. If anything failed the changes would be rolled back. That's
the basic idea, Cheers
Mark |
The JAM/JOM subsystems are a set of interfaces and classes used to parse and generate Java code. They can be used separately, but the most power and flexibility are achieved when used together. The main use is the parsing of source or class files to extract information about their contents. Examples of this include the various wizards in the object gallery (File|New) or the UI Designer.
JAM/JOM supersedes JOT as the Two-Way Tools backbone. The JOT package is deprecated; JBuilder 2005 ships with it solely in order not to break existing JOT-dependent OpenTools.
The heart of JAM/JOM is a series of interfaces and classes that describe the Java language in detail.
JAM (Java Abstraction Model) is a read-only layer that provides information about Java classes. The information only includes classes, methods and fields.
JOM (Java Object Model) is a read/write layer that deals with files and the constructs within those files. While JAM only abstracts classes, methods and fields, JOM deals with the details of source files such as position and constructs (statements, blocks, expressions etc).
The two API's work together but are kept separate for a couple of reasons. The main distinction is that JAM deals with the class as its main unit of abstraction, whereas JOM deals with the file as its main level of abstraction. The other big distinction is that, since JAM is not concerned with the writing of source code, it can use either the class file, source file, or both. Jom is only concerned with source code. This frees the user from having to worry about what representation the file comes from; if it's JAM the user doesn't care, and if it's JOM the user knows they are working with the source code.
A typical usage of JAM and JOM is using JAM to find the classes, methods and fields of interest and then converting to JOM when reading and writing source code is required.
JAM is a read-only layer that is basically a reflection layer. It hides details of whether the information comes from class or source. JAM chooses the correct backing based on what is being asked for and what is the fastest correct way of providing that information.
When you are working with JAM/JOM, you are either working with objects (classes, methods, fields) from a file that exists already, or you are creating a new file. If the file exists, it's usually part of your project (say, a selected node). If you are creating a new file, it is usually done in the context of the current project.
To access the JAM subsystem, use the static factory method instance(Project) of the JamFactoryobject, as this example:
Project project; // assume a valid, non-null instance of Project
JamFactory factory = JamFactory.instance(project);
JamClass jamClass = factory.getClass("com.borland.jbuilder.jam.TestJam");
|
The above code will return a jamClass if the class is found on the project paths.
JamFactory will return JamClass from class information if the class is up to date; otherwise it will return a JamClass with the information from the source file. If the information was from class information and the file is edited but not recompiled, then JAM will switch to using the source information when requested for new information.
Example: JAM from JOM alternative
An alternative way to obtain a JAM object from JOM is to use the getJomXXX() method on the relevant jom objects.
JamMethod jamMethod = jomMethod.getJamMethod(); |
To access the JOM subsystem use the static factory method of instance(project) JomFileManager;,
Project project; // assume a valid, non-null instance of Project JomFileManager jomManager = JomFileManager.instance(project); |
A typical usage is to get a JomFile for an existing Url file. This could be done like this:
...
Url fileUrl = new Url("c:/test/Foo.java");
if (fileUrl.exists()) {
// The getJomFile method below will return a JomFile for an existing file.
}
// Obtains a JomFile for a file. (If the fileUrl doesn't exist a new will be created.)
JomFile jomFile = jomManager.getJomFile(fileUrl);
// Use JOM to modify the file:
...
...
|
Another usage is to create the file if it doesn't exist:
...
fileUrl = new Url("c:/test/Foo.java");
// Obtains a JomFile for an existing file or creates a new JomFile object:
JomFile jomFile = jomManager.getJomFile(fileUrl);
// Use JOM to modify the file:
...
jomFile.commit class=(true);
...
|
The content of the source code in the file will not change until one jomFile.commit(...); method is called. JomFile should not be used after a commit on it is performed. A JomFile is intended to be a short-life object. You can use it to generate source code structures (joms) and commit the file. After this point, references to its
JomFile(or any of its children) should not be kept around.
If the source code is modified by other thread/process on disk (or in the VFS buffer), access to jomobjects in this JomFile will throw a ConcurentModificationError.
Example: JOM object from JAM alternative
An alternative way to obtain a JOM object from JAM is to use the getJomXXX() method on the relevant jamobjects.
JomMethod jomMethod = jamMethod.getJomMethod(); |
JAM views a class in a manner similar to how the java.lang.reflect package does. A representation of a Class exists (either from source or from a compiled unit). This Class is composed of member items: fields and methods. In many cases, you'll find working with these higher-level items familiar if you've previously worked with the Reflection API. Reflection provides access to the methods, constructors, and fields of a class. You can glean information on the class itself by using java.lang.Class directly.
JamClassJamClass has methods for discovering its superclasses, interfaces, methods and fields.
The naming conventions are similar to the Reflection API in the JDK. For instance, to
find out all the methods that are declared in the current class, use getDeclaredMethods(), but to find all methods including those of its ancestors, use getMethods(). This convention does not apply to constructors, since only the constructors that are declared in the current class are available. To get the declared constructors, use getConstructors().JamMethodJamMethod provides information about the method you are interested in: return type, the parameter types and names, plus all the modifier information.JamFieldJamField provides information about the field you are interested in: type, name and modifiers.JamTypeJamType represents both primitive types and Object types.
JamType is JamMethodType, which represents a method signature. All JamTypes are intended to represent fully qualified type information. JamType performs all conversions between qualified type names and signatures. Complex types are also represented by JamType, such as arrays, multi-dimension arrays, etc.
JamType as well. Code written with JAM and JOM will be able to take advantage of these features, often without any changes.Using JOM, you can do all that and go much deeper than using reflection alone. Jom has constructs which wrap around the text in the source code, providing for an easy programmatic access to the source code. It also adds a JomFile object, which is the representation for the file that contains the source code. These detail items follow the language guidelines laid out on the Java Language Specification (JLS.)
JomFile.JomClass.JomMethod.JomBlock and its contents are JomStatements or descendants of that class.JOM continues where reflection stops by allowing you to work with the individual statements or expressions that make up the heart of most Java code.
Once a JamClass is obtained as previously described, it can be queried for methods, fields, superclasses and subclasses.
JAM introduces JamType which represents a primitive type or an Objec type. It also recursively represents arrays of a type. There are predefined types for the primitives and String, i.e. JamType.INT or JamType.String. The method asArray() returns a JamType for the type, for instance to get a type for int[].
You can create a type from text (i.e. full type name) or from the type signature (how it's represented in the class files). Here are several ways of creating a JamType:
JamType type;
//Predefined types:
type = JamType.INT;
type = JamType.STRING;
//or specifying the type from the type name (long name for objects):
type = JamType.fromText("short");
type = JamType.fromText("java.lang.String");
//or from the class files signature:
type = JamType.fromSignature("Ljava/lang/String;");
type = JamType.fromSignature("B"); //Byte primitive
//creating arrays:
type = JamType.fromText("int[]");
type = JamType.INT.asArray();
type = type.asArray() //Whatever type is now type[]
|
A difficult problem that JAM addresses is finding methods. Methods have signatures as well as names. Class files deal with the signature of a method, and the compiler and sources deal with the parameter types. Both represent the same thing, but in a completely different way. JAM provides a special JamType, JamMethodType, which represents the method signature (not including the name).
JamMethodType methodType;
//Predefined types:
JamMethodType methodType = new JamMethodType();
//void x();
methodType.addParameter(JamType.INT);
//void x(int i);
methodType.addParameter(JamType.STRING);
//void x(int i, String s);
methodType.setReturnType(JamType.fromText("com.borland.sample.Bar");
//Bar x(int i, String s);
|
Once you have defined the method type, it's easy to find a method from a JamClass.
JamClass jamClass = factory.getClass("com.borland.jbuilder.jam.TestJam");
JamMethod jamMethod = jamClass.getMethod("x", methodType);
|
Finding a field is similar, except only the name is required:
JamClass jamClass = factory.getClass("com.borland.jbuilder.jam.TestJam");
JamField jamField = jamClass.getField("foobar");
|
JOM can parse Java source files. The methods specified by JOM for parsing Java use the naming convention get<XYZ> to extract whatever XYZ represents. These getter methods are generally used to parse the nested nature of Java code and usually return another item that is represented by a JOM.
During parsing, everything works from the outside in. At the outermost level (a JomFile), the parsing procedure is simple. A JomFile contains one or more classes (represented by a JomClass), and each of those classes is composed of variables and methods. Usually, you have a loop that iterates through all the classes in the file. From each of those classes, you can get an array of methods defined in that class and loop through each of these. The implementation of these methods makes things a bit more complicated. For example, a class can have many methods and each method can be composed of several statements, and each of those statements can be a method that is composed of many statements, and so on. At this level, recursion is common.
The base object in the JOM hierarchy is the Jom object. Every Java Language (JL) construct derives from this object. In order to remove any Jom you just need to call the remove method on it.
Every Jom also has an associated beforeComment/afterComment. (In order to obtain them you need to call the appropriate getXXX() methods.) The underlined Jom object of a comment is JomComment. It can be JavaDoc, standard comment, or EOL comment. JomComment allows you to modify the comment text.
JOM has a very high level of granularity. Children of Jom objects (JomMethodsfor JomClass, JomStatement for JomBlock etc.) are created as lazily as possible. The children Joms are created on-demand (class=jomFile.getJom class=(<JomPosition>).) Another way to populate recursively all the children of a Jom(JomClass, JomMethod, JomBlock, JomStatement, etc.) is to call getDeepChildren() method of a JomContainer. Yet another way is to use the FillExistingChildrenVisitor object, as shown below:
...
Jom jomToFill = <some Jom>;
com.borland.jbuilder.jom.FillExistingChildrenVisitor fillVisitor
= new com.borland.jbuilder.jom.FillExistingChildrenVisitor();
fillVisitor.visitJom(jomToFill);
...
|
Using JomContainer's getChildren class=(boolean includePendingRemovals) returns all the children. (Whether to include the one removed from the source is specified by the includePendingRemovals parameter.)
JOM allows you to parse down to the atomic level. Consider a simple example of a for loop:
public void forLoop(){
// a bunch of statements in the method body...
for(int i = 0; i < 5; i++){
System.out.println("Hello Jom!");
}
// even more statements in the method body...
}
|
The for statement is defined in section 14.12 of the Java Language
Specification, which states:
"Theforstatement executes some initialization code, then executes an_expression_, aStatementand some update code repeatedly until the value of the_expression_isfalse."
In our example, the initialization code is: int i = 0. The _expression_ is: i < 5. The statement that is executed is the println() inside the code block of the for loop. The update code is i++.
As the Java Language Specification states, a for loop is a type of statement. This means, at a high level, the for loop in the code would be parsed as being of type JomStatement. This is very generic as many things are JomStatements, so if you are working with a bunch of statements with JOM, you would determine if the type of statement you had encountered was really an instance of JomForLoop using instanceof.
JomForLoop breaks the for statement down even further. The _expression_ portion of a for statement describes the condition that causes the loop to terminate. Any valid Java _expression_ can be used here, ranging from a simple value check to a complex _expression_. JomForLoop.getCondJom() returns a JomExpression object that you could further interrogate to determine what type of an _expression_ you had and what it was composed of, and so on.
Use the following code to see how a simple class is broken down and parsed with JOM:
package com.borland.samples.jom;
import java.awt.*;
public class JotExample extends Component{
static final int PI = 4;
public JomExample(){
// xtor implementation goes here
}
public boolean doesItRock(int x){
int ctr = x;
for(int i = 0; i < ctr; i++){
System.out.println("JBuilder Rocks!");
System.out.println("Oh yes, it does");
}
return true;
}
}
|
For this example, we have an instance of JomFile named jsf that represents this file. To get the package statement to use this code:
JomPackage pkg = jsf.getPackage(); |
To get the import statements from the file, use this code and iterate through the array:
String [] imports = jsf.getImports(); |
Note: You can also use the JomContainer (such as JomFile) methods getChildren(boolean includePendingRemovals) and iterate through the children checking for instanceof JomImport to get the import statements.
To get the class(es) from the file, use this code:
JomClass [] classes = jsf.getClasses(); |
In this example, there's only one class file represented, so you know that your class is the first element in the array of classes returned by JomFile.getClasses(). Therefore, you can use this code to get the specific instance of JomClass:
JomClass jc = classes[0]; |
From the JomClass you can extract information about the class definition as well as its member fields and methods. The basic pattern remains the same; that is, a getter returns a value or object that you can interrogate further for more information.
// info about the class declaration JomModifiers mod = class=jc.getJomModifiers(); JomExtends super = jc.findJomExtends(); JomTypeList imps = findJomImplements |
These methods allow you to find out the modifiers for the class, the super class and the List of interfaces implemented by the class. You could get more information about the superclass by using the methods in JAM to get a JamClass for the JomExtends object.
Where modifiers are used in Java, as, for example a public class or a static final variable, JOM allows access through a getJomModifiers class=() method. getJomModifiers class=() returns an object, which
provides methods for manipulating the modifiers on any JL construct.
If you continue with the parsing of the class, you can get a list of fields and a list of methods contained in the class. Again, you can further parse the results to get to the atomic portions of the fields and methods:
// details on the contents of the class JomField [] fields = jc.getFields(); JomMethod [] methods = jc.getMethods(); |
For a more complete example of using JOM to parse source, see the sample com.borland.samples.opentools.jom.read.ReadingSource. This sample takes the selected file and gathers information about classes and methods using JOM. This information is sent to the message window inside the IDE.
An example of using JAM/JOM to parse source and/or classes is the
sample com.borland.samples.opentools.jom.packagetree.PackageTree.
This sample takes the selected node in the IDE, determines its package, and
then builds a hierarchy tree for all the classes in that package.
Jommodels the source file and its contents. The starting component for Jomis JomFile. JomFileis obtained from the projects JomFileManageror from a JamClass, JamMethod or JamField. Jom is a high level representation of source code and provides many features to simplify the development of code.
Jomprovides a set of advanced features for writing two way code, auto formatting, auto importing, comment support, references, renaming within file and position encapsulation. Also, Jom provides a complete mapping of language constructs to logical Jom constructs.
Some of the important JOM constructs are JomFile, Jom, JomPosition, JomContainer and JomReference.
JomFile
JomFile represents a single source file and provides a logical mapping of the source code. Source code can be safely inspected and modified. Retrieve a JomFile from the JomManagerwith an Url or fully qualified class name, or from a Jam that represents a class, method, or field in a source file.
JomFile is a
JomContainerwhich holds other Joms. The two high level components in a JomFile are a JomClass and a JomImport. JomClass is the container for most other Jom types. When the modifications to a JomFile are complete, a commit writes the changes including (optionally) formatting, importing and adding Javadoc.
JomFile is getting a class and then committing it. This simple program will create the class if it does not already exist:
Url fileUrl = manager.getUrlFromClassName("root.subpackage.MyClass1");
JomFile file = manager.getJomFile(fileUrl);
JomClass jomClass = file.getPrimaryClass();
|
JomPositionJoms within a source file. It translates between compiler positions, buffer positions and line/column positions. All Joms have positions associated with them. Joms can be tested if they are before or after other Joms and using a JomContainer. A JomContaineris any Jom that contains
another Jom's classes, methods, blocks etc. Joms can be used as position markers
within other Joms.JomContainerJoms. Has methods for inserting before or after other Joms and a general method for appending to the container.JomFactoryJoms.When writing source with JOM, you deal with much higher level items like methods or statements, instead of the very fine-grained items available to you when reading source files with JOM. The methods specified by JOM for writing Java code use the naming convention append<XYZ>, insert<XYZ>, add<XYZ> and set<XYZ> where XYZ represents the item you are
creating or modifying. For example, JomClass.getBody class=().append() adds a child to a given class. class=JomMethod.getJomModifiers class=().setFinal(boolean bValue) sets the modifier's final bit for that method.
A convenience method in JomFile is JomClass getPrimaryClass(). This method returns the primary class of a file (the public class, the first top level class.) If the JomFile is empty (for instance, if just being created), this method will return a new JomClass, ready to be edited (using JOM) and later committed.
Notes: JOM is very forgiving of being told to write bad code and will write anything you have told it to write.
JOM expects all the type strings (return types, parameter types etc.) to be fully qualified. The types and appropriate import statements will be automatically generated based on the user code generation properties.
Example: writing code using JOM
Let's create the same class that was used in the parsing section, but we'll build it up in smaller chunks. This simple "starting point" of the class looks like:
package com.borland.samples.jom;
import java.awt.Component;
public class JomExample extends Component{
}
|
Again, we begin with the outermost item, the source file itself. For JOM, the source file is represented by a JomFile that encapsulates the package statements, the imports, and the
class declaration.
Assuming you have an instance of a JomFile named jsf, use the following methods to create the class:
jomClass = JomFactory.createJomClass("com.borland.samples.jom.JomExample", jomFile);
JomExtends jomExtends = jomClass.getJomExtends();
jomExtends.setName class=("java.awt.Component"); // NORES
JomModifiers mod = class=jomClass.getJomModifiers();
mod.setPublic class=();
|
To finish, you need to add the rest of the content (a member variable, a constructor and a method) to the class. This is what the code that is generated will look like:
package com.borland.samples.jom;
import java.awt.Component;
public class JomExample extends Component {
static final int PI = 4;
public JomExample() {
// xtor implementation goes here...
}
public boolean doesItRock(int x){
int ctr = x;
for(int i = 0; i < ctr; i++) {
System.out.println("JBuilder Rocks!");
System.out.println("Oh yes, it does");
}
return true;
}
}
|
To add the field, use code like this:
JomField jf = jcs.getBody().append(JomFactory.createJomField("PI",JamType.INT);
jf.setInitialValue class=("4");
|
To make the field PI both staticand final, do this:
jf.getJomModifiers class=().setStatic(true); jf.getJomModifiers class=().setFinal(true); |
To create the constructor and make it public, do this:
JomConstructor xtor = jcs.append(JomFactory.createJomConstructor(jcs, JomParameterList.EMPTY)); xtor.getJomModifiers().setPublic(); |
To set up the "shell" of the method with a parameter x, use this code:
JomParameterList parList = JomFactory.createJomParameterList();
JomParameter parameter = JomFactory.createJomParameter("x", JamType.INT);
parList.append(parameter);
jomMethod = JomFactory.createJomMethod("doesItRock", parList, JamType.BOOLEAN);
mod = jomMethod.getJomModifiers();
mod.setPublic();
jomClass.append(jomMethod);
|
Add the local variable ctrand assign it the value of the parameter passed to the method:
JomVariable jv = JomFactory.createJomVariable("ctr", JamType.INT);
jv.setInitialValue class=("x");
jomMethod.append class=(jv);
|
Add in the for statement. There are two ways to create the JomForLoop:
JomForLoop) and fill its children Joms appropriately, orJomStatement objects and add it to the method. (Note that JomText, JomStatement and JomExpression can be used to hold any text that needs to be generated in the source code. If the text has more than 255 characters in a row, each row over 255 characters must be broken into smaller rows by JomCommitter.NEWLINE.)In a JomStatement, you can put any arbitrary string and it will be generated in the source code. In the following JomStatement object, we are actually generating two statements: The ForLoopand the return statement:
StringBuffer statementText = new StringBuffer();
statementText.append("for(int i = 0; i < ctr; i++) {");
statementText.append(JomCommitter.NEWLINE);
statementText.append("System.out.println(\"JBuilder Rocks!\");");
statementText.append(JomCommitter.NEWLINE);
statementText.append("System.out.println(\"Oh yes, it does");");
statementText.append(JomCommitter.NEWLINE);
statementText.append("}");
statementText.append(JomCommitter.NEWLINE);
statementText.append("return true;");
jomMethod.appendclass=(JomMaker.createJomStatement(statementText.toString()));
|
For a more complete example of using JOM to write source, see the sample
com.borland.samples.opentools.jom.write.WritingSource. This sample generates a simple class file. Another sample of using JOM to read and write is com.borland.samples.opentools.jom.readwrite.Commenter. This sample takes the selected source node and adds Javadoc comments for top level classes and methods in classes. It only adds them if there isn't a comment there already.
Use JAM to discover the main construct you are interested in. Once you have the appropriate JAM object, you can directly convert to a JOM for manipulating source code.
For instance, assume you have a method named bar that takes an int and returns a Stringin the class com.foobar.Foo. There are 4 simple steps:
JamFactory for the project.JamClass.JamClass for the JamMethod, passing in a JamMethodType and the method name.JomMethod from the JamMethod.Notice that JAM deals in logical constructs entirely -- you don't need to know where files reside on the source path, just the full class name. JamType and JamMethodTypealso simplify the process of defining the method you are searching for.
Project project; // assume a valid, non-null instance of Project
JamFactory factory = class=JamFactory.instance(project);
JamClass jamClass = factory.getClass("com.foobar.Foo");
JamMethodType type = new class=JamMethodType();
type.setReturnType class=(JamType.STRING);
type.addParameter class=(JamType.INT);
JamMethod jamMethod = jamClass.getMethod("bar", type);
JomMethod jomMethod = jamMethod.getJomMethod();
|
Attachment:
JamJomBackground.pdf
Description: Adobe PDF document