Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Preprocessor and Macros
Preprocessor and Macros [message #646794] Fri, 31 December 2010 10:13 Go to next message
No real name is currently offline No real nameFriend
Messages: 25
Registered: April 2010
Junior Member
Hi,

Im almost done with my hacked together grammar, the only thing remaining is how to handle macros.

The problem with macros is, that they can virtually appear anywhere and can contain anything (including operators, etc) therefore are a big pain to include into the parser. Right now, I have them as hidden terminals, but this is only suboptimal as expressions such as these exists:

a = MACRO * MACRO.

Now obviously, if a macro is a hidden terminal, the parser will complain about the multiplication.

Now, I have been thinking to implement a preprocessor myself, by doing the following:

Create a class MyXtextResourceFactory, which implements IResourceFactory and looks like this:


public class MyXtextResourceFactory implements IResourceFactory {

	private Provider<MyXtextResource> provider;

	@Inject
	public XtextResourceFactory(Provider<MyXtextResource> resourceProvider) {
		this.provider = resourceProvider;
	}
	
	public Resource createResource(URI uri) {
		XtextResource xtextResource = provider.get();
		xtextResource.setURI(uri);
		return xtextResource;
	}

}


Additionally, I thought of creating a class called MyXtextResource which would probably extend the lazy linking resource. At that point, I could overwrite the doLoad method and use the preprocessor at this point. This setup would get me the preprocessing done.

Now the problem is that I know little of the backend of xtext, and I would have no idea what consequences this would have. Would it break the syntax highlighting and highlighting in general? Would there be a much better alternative? Ideally, I would like the macros also be covered in the contentassist, but I guess I could implement a preprocessor there as well (e.g. finding all the macros).
Re: Preprocessor and Macros [message #646844 is a reply to message #646794] Sat, 01 January 2011 10:03 Go to previous messageGo to next message
No real name is currently offline No real nameFriend
Messages: 25
Registered: April 2010
Junior Member
I gave that a try and it messes up the highlighting. Ideally I would like to underline the macro if it causes any syntactic error.

Does anyone have any idea on how to go about that?
Re: Preprocessor and Macros [message #646981 is a reply to message #646844] Mon, 03 January 2011 19:30 Go to previous messageGo to next message
No real name is currently offline No real nameFriend
Messages: 25
Registered: April 2010
Junior Member
I have tried to implement the macros in the lexer. Defining macro's works fine and everything else could work fine (e.g. #if macros) but the problems are substitutions.

The way I do inclusions and subsitutions right now, is to change the input CharStream variable to something different and then go back to the old stream.

The biggest problem comes with the fact that the parser seems to use the text and index position for highlighting instead of stop and start index. It wouldnt be able to know otherwise, because all it gets is a Token and not a CommonToken.

As it seems, it doesnt seem to be possible to implement macro substitution without rewriting a lot of the framework.

Unless I have missed something? Maybe someone of you has some idea on how to go about this?

Re: Preprocessor and Macros [message #647055 is a reply to message #646844] Tue, 04 January 2011 11:58 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 1/1/11 11:03 AM, Wattos@gmail.com wrote:
> I gave that a try and it messes up the highlighting. Ideally I would
> like to underline the macro if it causes any syntactic error.
> Does anyone have any idea on how to go about that?

I can imagine a solution where you allow expressions such as
MACRO * MACRO where MACRO is a cross reference to the definition of the
macro (i.e. nothing is hidden).

You can expand the macros when validating, and perform callbacks to the
parser to make it parse the result. You then translate/transform any
errors from this parse into meaningful errors in the source text.

If you need your model to fully reflect the macro-expanded model, you
would need to perform a m2m transformation before code generation or
evaluation.

Just an idea.
- henrik
Re: Preprocessor and Macros [message #647076 is a reply to message #647055] Tue, 04 January 2011 13:54 Go to previous messageGo to next message
Meinte Boersma is currently offline Meinte BoersmaFriend
Messages: 434
Registered: July 2009
Location: Leiden, Netherlands
Senior Member
In addition to Henrik's comments: it's probably a good thing to elevate your macros (or templates?) to first-class citizens in your grammar. That saves a bunch of problems which seem to work against the grain of Xtext anyway and maybe offer you some value in return.

There's some flexibility here as long as you can recognize/define where a macro starts and stops and don't forget to explicitly include all terminals (keywords and terminal rules) in case you have "free text" inside the macros.


Re: Preprocessor and Macros [message #647140 is a reply to message #646794] Tue, 04 January 2011 17:57 Go to previous messageGo to next message
No real name is currently offline No real nameFriend
Messages: 25
Registered: April 2010
Junior Member
Thanks for the responses guys. I really appreciate it.

Another problem which pops up with macros inside the grammer is that macros can appear vistually anywhere. You could have something like:

#define STUPID * 5

then inside the code the following would be valid:

x = 5 VALID.

That becomes even more difficult to parse. Additionally, you could also think of the following

#define FUNC void HelloWord(

and then have in your class body:

FUNC int a ){
};

If I do a separate preprocessing stage before running through the lexer, the highlight will all be messed up (especially if the macro contains new lines Sad )

At this stage, I dont think that a pure macro parsing is possible without changing a lot of the framework. I might be able to get away with a crippled version of macros (e.g. just allow them as expressions and class elements).

If it was up to me, I'd get rid of macros completely, but unfortunately, the langugage is out of my control.

Again, thanks for all the advice given
Re: Preprocessor and Macros [message #712949 is a reply to message #646794] Mon, 08 August 2011 00:14 Go to previous message
Moritz   is currently offline Moritz Friend
Messages: 22
Registered: July 2011
Junior Member
Hi,
I am interested: How did you finally handle the problems you stated? Did you restrain the macros, or did you fully blow up your language with the macros?

Did you consider somehow executing an external preprocessor prior to parsing? Did you try Henrik's suggestion to expand the macros? How did you do it?

Regards, Moritz
Previous Topic: Xtext 2.0 plugin
Next Topic:[XTEND] Standalone compiler
Goto Forum:
  


Current Time: Mon Apr 19 19:26:08 GMT 2021

Powered by FUDForum. Page generated in 0.01765 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top