Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Creating terminal STRING rule with start and end tokens?
Creating terminal STRING rule with start and end tokens? [message #1839800] Mon, 29 March 2021 09:59 Go to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
Is it possible to create the terminal rule STRING such that instead of it using e.g. single quotes to start and stop the string, it uses a long sequence of characters?

The reason is I want the user to be able to write everything inside the string.

Something like this, except this doesn't work.

terminal STRING: 'START' ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'|'START'|'END') )* 'END';


[Updated on: Tue, 30 March 2021 09:15]

Report message to a moderator

Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839804 is a reply to message #1839800] Mon, 29 March 2021 10:35 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7655
Registered: July 2009
Senior Member
Hi

I would expect you to specify the letter sequence more explicitly requiring e.g. 'S' 'T' 'A' 'R' 'T' and so I would expect that 'START' should be reported as an error, but maybe there is support for multi-characters and you just need to enable backtracking.

Regards

Ed Willink
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839825 is a reply to message #1839804] Mon, 29 March 2021 15:08 Go to previous messageGo to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
With backtracking enabled, I've tried terminal STRING: 'S' 'T' 'A' 'R' 'T' ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'| 'S' 'T' 'A' 'R' 'T'| 'E' 'N' 'D') )* 'E' 'N' 'D';

This gave an error: error(100): ../org.xtext.example.sglGrammar/src-gen/org/xtext/example/mydsl/parser/antlr/lexer/InternalSglGrammarLexer.g:0:0: syntax error: buildnfa: <AST>:271:73: expecting EOA, found ''T''

Also tried: terminal STRING: 'S' 'T' 'A' 'R' 'T' ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\' | 'START' | 'END') )* 'E' 'N' 'D';

This gives:
error(204): ../org.xtext.example.sglGrammar/src-gen/org/xtext/example/mydsl/parser/antlr/lexer/InternalSglGrammarLexer.g:271:77: duplicate token type 'END' when collapsing subrule into set
error(204): ../org.xtext.example.sglGrammar/src-gen/org/xtext/example/mydsl/parser/antlr/lexer/InternalSglGrammarLexer.g:271:77: duplicate token type 'END' when collapsing subrule into set
error(204): ../org.xtext.example.sglGrammar.ide/src-gen/org/xtext/example/mydsl/ide/contentassist/antlr/lexer/InternalSglGrammarLexer.g:271:77: duplicate token type 'END' when collapsing subrule into set
error(204): ../org.xtext.example.sglGrammar.ide/src-gen/org/xtext/example/mydsl/ide/contentassist/antlr/lexer/InternalSglGrammarLexer.g:271:77: duplicate token type 'END' when collapsing subrule into set
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839925 is a reply to message #1839825] Thu, 01 April 2021 10:25 Go to previous messageGo to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
?
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839926 is a reply to message #1839925] Thu, 01 April 2021 10:41 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
if anybody had an idea i guess they would have answered
maybe you can look into external lexers or
consider using a datatype rule instead


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Thu, 01 April 2021 10:45]

Report message to a moderator

Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839932 is a reply to message #1839926] Thu, 01 April 2021 12:28 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
you can try

terminal STRING: 'START' -> 'END';

you also need to customize the STRINGValueConverter
or rename the rule to e.g.

terminal STRING2: 'START' -> 'END';


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839959 is a reply to message #1839932] Fri, 02 April 2021 08:01 Go to previous messageGo to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
Thanks, that worked!

Just to be clear, nothing more than something like this is needed to customize the STRINGValueConverter? It seemed to work, but don't know if I'll get in trouble in some other example:

public class UpdatedSTRINGValueConverter extends STRINGValueConverter {
@Override
public String toValue(String string, INode node) {
if (string == null) return null;
return string.substring(5, string.length() - 3);
}
}
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839960 is a reply to message #1839959] Fri, 02 April 2021 09:44 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
would have expected that you customize toString too

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839961 is a reply to message #1839960] Fri, 02 April 2021 09:54 Go to previous messageGo to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
Okay, I'll do that as well. Thanks a lot and have a nice day!
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839978 is a reply to message #1839961] Fri, 02 April 2021 15:52 Go to previous messageGo to next message
M D is currently offline M DFriend
Messages: 33
Registered: January 2021
Member
Found out I needed the default STRING terminal as well. So I've create a terminal rule called STRING_WITH_TOKENS, and I've tried to create a ValueConverter for it, but I can't get it to work. This is what I've got:

Rule:
terminal STRING_WITH_TOKENS: 'START' -> 'END';


Converter:
public class STRING_WITH_TOKENSValueConverter extends DefaultTerminalConverters {
	@Inject StringWithTokens stringWithTokens;
	
	@ValueConverter(rule = "STRING_WITH_TOKENS") //Have also tried the full path: "org.xtext.example.mydsl.SglGrammar.STRING_WITH_TOKENS"
    public IValueConverter<String> STRING_WITH_TOKENS() {
		return stringWithTokens;
	}
        
        private static class StringWithTokens extends AbstractLexerBasedConverter<String> {

			@Override
			public String toValue(String string, INode node) throws ValueConverterException {
				if (string == null) return null;
				return string.substring(5, string.length() - 3);
			}
			
//Don't know if this is correct as well. Worked, when I overwrote the default STRING rule, though.
			@Override
			protected String toEscapedString(String value) {
				return "START" + Strings.convertToJavaString(value, false) + "END";
			}
        	
        }
}


Binding:
public class SglGrammarRuntimeModule extends AbstractSglGrammarRuntimeModule {
	public Class<? extends STRING_WITH_TOKENSValueConverter> bindSTRINGValueConverter() {
    	return STRING_WITH_TOKENSValueConverter.class;
    }
}
Re: Creating terminal STRING rule with randomized start and end tokens? [message #1839979 is a reply to message #1839978] Fri, 02 April 2021 15:58 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
should be

@Override
public Class<? extends IValueConverterService> bindIValueConverterService() {
return STRING_WITH_TOKENSValueConverter.class;
}


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Previous Topic:DSL with python grammar
Next Topic:Keywords available only in specific cases
Goto Forum:
  


Current Time: Tue Apr 23 10:38:52 GMT 2024

Powered by FUDForum. Page generated in 0.04313 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top