Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Keywords with whitespaces and same prefix(Lexer issue? )
Keywords with whitespaces and same prefix [message #635874] Thu, 28 October 2010 13:10 Go to next message
Thomas Delissen is currently offline Thomas DelissenFriend
Messages: 11
Registered: September 2010
Location: Eindhoven
Junior Member
Hello,

I am creating a very verbose language, in which I would like to use keywords with spaces in them.

For instance, I have the rule:
CreateTask:
'create' task=[Task] 'with related resource' resource=[Resource]
;


On itself, this works fine. After I have selected the Task from the tasks defined earlier, I can just hit Ctrl+Space and the sentence 'with related resource' is created.

However, at some point, I've introduced a sentence in a completely different part of language as such:

UseMaterial:
'use' material=[Material] 'with' task=[Task]
;


Now, the editor stumbles over the sentence 'with related resource', as it cannot tell the difference between the first part of the keyword 'with related resources' and 'with'.

I think this is an issue with the lexer or something. The weird part is that autocompletion works fine in this case: You press Ctrl+space and after this 'with related resources' appears, but immediately after this an error is raised.

My question is: Is there an easy way to fix this, so that I can still use keywords with spaces in them, or should I avoid keywords with spaces in them?

I know I can use the alternative to make three separate keywords 'with' 'related' and 'resources', but in this case I should press Ctrl+space three times, which is also not desirable.


Kind regards,

Thomas
Re: Keywords with whitespaces and same prefix [message #635940 is a reply to message #635874] Thu, 28 October 2010 16:06 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Keywords with spaces will get you into trouble IMO.

You can easily provide entire sentences as proposals, they are not auto
generated though.

Sentence1 : 'hit' 'monster' 'with' 'axe' ;

Then, when computing a proposal for SENTENCE1, just return the entire
string "hit monster with axe" as the proposal.

In your DSL's ProposalProvider...
public void completeSentence1(EObject model, Assignment assignment,
ContentAssistContext context, ICompletionProposalAcceptor acceptor) {
ICompletionProposal completionProposal = createCompletionProposal(
"hit monster with axe", new StyledString(
"A complete monster hitting sentence"), context);
acceptor.accept(completionProposal);

super.completeBranch_Name(model, assignment, context, acceptor);
}

Where "createCompletionProposal" creates a suitable instance with things
like priority, icon etc.

An additional benefit with separate words is that user can place the
words as far apart as wanted, on different lines, include comments etc.

hit monster with
/* let's confuse the monster with a: */ penguin

- henrik

On 10/28/10 3:10 PM, Thomas Delissen wrote:
> Hello,
> I am creating a very verbose language, in which I would like to use
> keywords with spaces in them.
> For instance, I have the rule:
> CreateTask:
> 'create' task=[Task] 'with related resource' resource=[Resource]
> ;
>
>
> On itself, this works fine. After I have selected the Task from the
> tasks defined earlier, I can just hit Ctrl+Space and the sentence 'with
> related resource' is created.
> However, at some point, I've introduced a sentence in a completely
> different part of language as such:
>
> UseMaterial:
> 'use' material=[Material] 'with' task=[Task]
> ;
>
>
> Now, the editor stumbles over the sentence 'with related resource', as
> it cannot tell the difference between the first part of the keyword
> 'with related resources' and 'with'.
> I think this is an issue with the lexer or something. The weird part is
> that autocompletion works fine in this case: You press Ctrl+space and
> after this 'with related resources' appears, but immediately after this
> an error is raised.
> My question is: Is there an easy way to fix this, so that I can still
> use keywords with spaces in them, or should I avoid keywords with spaces
> in them?
> I know I can use the alternative to make three separate keywords 'with'
> 'related' and 'resources', but in this case I should press Ctrl+space
> three times, which is also not desirable.
Re: Keywords with whitespaces and same prefix [message #636126 is a reply to message #635874] Fri, 29 October 2010 12:19 Go to previous messageGo to next message
Alexander Nittka is currently offline Alexander NittkaFriend
Messages: 1193
Registered: July 2009
Senior Member
Hi,

as Henrik indicates, keywords including white spaces are problematic. The basic problem is that the lexer is greedy in the sense that it tries to construct tokens that are as long as possible. That is, if you have a keyword "with" and one "with related" you will automatically run into trouble because after "with" you will most probably have a space in your file. In this case the lexer thinks "Great, there is a token that starts with 'with' and continues with a space, so this is the token I want here" and will produce a syntax error if you don't continue with "related".

In short: don't use keywords including white spaces, rather adapt code completion or even provide templates for your language that will automatically produce the entire sequence of keywords (as already suggested).

In case you really want exactly one space between each keyword so "a b" is correct whereas "a b" is not, make white spaces non-hidden in the rule and add a "single space keyword" (which is a bit problematic as it overlaps with the WS-rule so you'll have to invest some more work there); note that this means that you have to indicate each position in the rule where a white space may occur.

Alex
Re: Keywords with whitespaces and same prefix [message #636629 is a reply to message #635874] Tue, 02 November 2010 09:40 Go to previous message
Thomas Delissen is currently offline Thomas DelissenFriend
Messages: 11
Registered: September 2010
Location: Eindhoven
Junior Member
Thanks for the advice! I will try too look into overriding the ProposalProvider as Henrik suggested. Indeed, if the content-assist produces the complete sentence even though the grammar has the keywords as: 'hit' 'monster' 'with' 'penguin', my problem is solved.

And if I am not able to do that (I am already having a hard time overriding the ScopeProvider), I will just use underscores and write something like 'hit_monster_with_penguin'. But that would be the easy way out Smile


Kind regards,

Thomas
Previous Topic:Cross reference feature & import question
Next Topic:Adding EnumLiteralDeclaration programmatically
Goto Forum:
  


Current Time: Sat Apr 20 02:34:40 GMT 2024

Powered by FUDForum. Page generated in 0.02624 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top