Eclipse Community Forums: TMF (Xtext) » Creating an unvalidated area of text

Help

Home

Home » Modeling » TMF (Xtext) » Creating an unvalidated area of text

Show: Today's Messages :: Show Polls :: Message Navigator

Creating an unvalidated area of text [message #673830]

Tue, 24 May 2011 21:23

Richard Cookman

Messages: 7
Registered: May 2011

Junior Member

I'd like to start by saying having now used xtext for a few days i'm now well and truely thankful such a language exists as you have helped me save so much time and effort creating eclipse IDE support for my language.

However i'm still strugling with one or two things.

First I want to create an area of text that has no validation at all.

For example

SOMEMARKER{ /* Unvalidated text, anything could go here */... }

Also inside the SOMEMARKER text I could again have a section of validated code.

For example

SOMEMARKER{ ... SOMEOTHERMARKER{ /* Validated text */ } ... }

Any ideas how it might be possible to do this?

I have a very good idea of how I'd do the second if I can get the first to work.

Report message to a moderator

Re: Creating an unvalidated area of text [message #673844 is a reply to message #673830]

Tue, 24 May 2011 23:13

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

The best approach is to solve this at the Lexer level. The problem is to
convey the knowledge to the lexer. The simplest case would naturally be
if your language has very distinct tokens that signal the start and end
of ignored text. If the content of the ignored area is simple enough you
may get away with just a terminal of high precedence (just like the
default terminal for ML_COMMENT).

If that does not work, you may need to write an external lexer. There is
one implementation in mwe2 and I have one in Cloudsmith/geppetto (at
github). Neither implementation deals with "uninterpreted text", but
does similar things to handle strings with interpolated expressions. The
Geppetto external lexer is quite small and it should not be difficult to
figure out how it works even if not understanding everything in the
grammar. (The trickiest problem I had to solve was overlapping terminals
for regular expressions, comments and division).

In the Eclipse b3 project I made an implementation of strings with
interpolation without using an external lexer (I did not know I could
use one at that point... you can take a look at the beelang grammar in
the b3 project as well if interpolation would work for you.

(interpolation == handling strings like "HELLO
${person.getName().toUpper()}, how are you?")

Regards
- henrik

On 5/24/11 11:23 PM, Richard Cookman wrote:
>
> I'd like to start by saying having now used xtext for a few days i'm now
> well and truely thankful such a language exists as you have helped me
> save so much time and effort creating eclipse IDE support for my language.
>
> However i'm still strugling with one or two things.
>
> First I want to create an area of text that has no validation at all.
>
> For example
>
> SOMEMARKER{ /* Unvalidated text, anything could go here */... }
>
> Also inside the SOMEMARKER text I could again have a section of
> validated code.
>
> For example
>
> SOMEMARKER{ ... SOMEOTHERMARKER{ /* Validated text */ } ... }
>
> Any ideas how it might be possible to do this?
>
> I have a very good idea of how I'd do the second if I can get the first
> to work.
>

Report message to a moderator

Re: Creating an unvalidated area of text [message #674441 is a reply to message #673844]

Thu, 26 May 2011 23:06

Richard Cookman

Messages: 7
Registered: May 2011

Junior Member

Thank you Henrik Lindberg. Having a high precedence terminal like a comment isn't very applicable as my language uses '{' '}' for all containment including here.
Also as I mentioned in the example code I need a processed block within the other block, don't think you could exit from a comment in that way (or I maybe wrong there?).
So i'm going to create an external lexer i'm assuming by this you mean doing something along the lines of grammarMixins a-la: the xtext doc grammarMixins

Also if this is the soluting i'm assuming that I create that "external lexer" using the normal x-text grammar file ".xtext".

If this is the case then I will simply have to create another x text project as i'm getting the destinct impression that you can't have two ".xtext" files in the same project each referencing each other. Please correct me if i'm wrong.

That being the case i'd say the documentation is a bit missleading and should really read:

grammar my.SuperGrammar
...
RuleA : "a" stuff=RuleB;
RuleB : "{" name=ID "}";

grammar my.SubGrammar with my.SuperGrammar

Model : (ruleAs+=RuleA)*;

// overrides my.SuperGrammar.RuleB
RuleB : '[' name=ID ']';

Report message to a moderator

Re: Creating an unvalidated area of text [message #674541 is a reply to message #674441]

Fri, 27 May 2011 09:41

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

The external lexer is written in antlr (a .g file), but is integrated
nicely in mwe.
The only difficulty is that tokens/keyword ids must be kept in sync.

You can look at my code at Cloudsmith/Geppetto at github. You find the
external lexer in the project pp.dsl and the mwe workflow in
pp.dsl.generate.

Regards
- henrik

On 5/27/11 1:06 AM, Richard Cookman wrote:
> Thank you Henrik Lindberg. Having a high precedence terminal like a
> comment isn't very applicable as my language uses '{' '}' for all
> containment including here.
> Also as I mentioned in the example code I need a processed block within
> the other block, don't think you could exit from a comment in that way
> (or I maybe wrong there?).
> So i'm going to create an external lexer i'm assuming by this you mean
> doing something along the lines of grammarMixins a-la: the xtext doc
> grammarMixins
>
> Also if this is the soluting i'm assuming that I create that "external
> lexer" using the normal x-text grammar file ".xtext".
>
> If this is the case then I will simply have to create another x text
> project as i'm getting the destinct impression that you can't have two
> ".xtext" files in the same project each referencing each other. Please
> correct me if i'm wrong.
>
> That being the case i'd say the documentation is a bit missleading and
> should really read:
>
> grammar my.SuperGrammar
> ...
> RuleA : "a" stuff=RuleB;
> RuleB : "{" name=ID "}";
>
> grammar my.SubGrammar with my.SuperGrammar
>
> Model : (ruleAs+=RuleA)*;
>
> // overrides my.SuperGrammar.RuleB
> RuleB : '[' name=ID ']';
>

Report message to a moderator

Re: Creating an unvalidated area of text [message #674807 is a reply to message #674541]

Sat, 28 May 2011 16:09

Johannes Stelzer

Messages: 30
Registered: October 2009

Member

I did something similar. With a little hack...

Have a look at may grammar:


any_token returns ecore::EString:
DQ_STRING|SQ_STRING|INT|QualifiedName|';'|'('|')'|'.'|','|ANY;

...
...
...

terminal ANY:
  .;

The any_token rule lists every terminal rule incluging a terminal rule for any character.
And now u can say:

sometext returns ecore::EString:
'SOMEMARKER' '{' any_token* '}';

Hint: you can't use the '{' or '}' in the text....
And the value of sometext will also contain SOMEMARKER { }

Report message to a moderator

Re: Creating an unvalidated area of text [message #674940 is a reply to message #674807]

Sun, 29 May 2011 16:30

Richard Cookman

Messages: 7
Registered: May 2011

Junior Member

Many thanks that hack worked.
I can now have:

SOMEMARKER{ ... Any Text ... { ... Any Text ... } ... { ... SOMEOTHERMARKER { /* Validated Text */ SOMEMARKER{ ... Any Text ... } ... } ... } ... Any Text ... }

Which is really the dynamism I was after, many thanks. Smile

After many ideas, sole searching and trying out different things I found that worked.

I did have a go at something similar to what you suggested before coming to this forum for help but when what i tried didn't work I came here.

In the end I don't think my original string list was comprehensive enough.

In the working version any_token rule includes all my key words and delimiters and the delimiter '#' even though I don't use it as a delimiter (although it is part of another keyword, though I don't understand how it explains why it was failing on the '#' character, think that might be a bit of a quirk)

Atleast it works now anyway.

[Updated on: Sun, 29 May 2011 18:40]

Report message to a moderator

Previous Topic:	Return different properties depending of the EObject type
Next Topic:	Stopping error "Duplicate Constrct 'foo' in Struct 'bar'" for function that take different

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 06:59:22 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter