| Fuzzy grammar advice [message #1829942] |
Tue, 14 July 2020 10:01  |
Eclipse User |
|
|
|
I'm wondering how to best implement a grammar that skips statements and expressions (and their descendants) and just parses at the surface level of the code. In other words I only want to parse classes/objects with their fields and method names but I don't want to meddle with the expressions in this fuzzy parser version.
The language I'm creating an xtext based LS for, has its methods defined within brackets, as for example:
functionId[ param1 param2 paramN;
...stat/expr...<--- skip all of these
]
And the global function's syntax looks like:
[globalFunctionId param1 param2 paramN;
...stat/expr... <--- skip all of these
] ;
My (flawed) solution so far has been to create a BRACKET_TO_BRACKET terminal:
terminal BRACKET_TO_BRACKET:
'[' -> ']' ';'
;
But the disadvantage is of course that the ']' token can appear within a string statement, such as "[Text]" and break up the parsing halfway.
Luckily though the language doesn't use brackets for array elements so it is probably only the strings that will require special handling.
I'm under the impression that this is hard to do by grammar and that I might need to override the lexer with my own variant of it, but cannot say for sure. It might just be that I haven't thought long enough on how accomplish this yet. But IF I need to override the Lexer Ive noticed that it turns all its rules to final methods making it even harder to override the specific mBRACKET_TO_BRACKET function I want to customize. Where can the final modifier be turned off in case there isn't a better way forward?
(Any suggestions whatsoever in how accomplish the end goal here is most welcome.)
[Updated on: Tue, 14 July 2020 10:12] by Moderator |
|
|
|
|
|
|
| Re: Fuzzy grammar advice [message #1829987 is a reply to message #1829983] |
Wed, 15 July 2020 05:54  |
| Eclipse User |
|
|
|
Hi
I have no experience of custom lexers for Xtext so cannot comment on how easy it is. I shared your reticence about messing up the larger framework. The retokeniser changes very little and my example should demonstrate how to initialize a replacement token with adequate context.
Braces and such like should not be a problem. The retokeniser assumes that the default lexer breaks the text up into sensible tokens so presumably your nested expression will be lexed as a sequence of identifiers and punctuation tokens. The retokeniser just needs to recognise the start token, presumably easy, and the end token, perhaps harder if a string mis-lex occurred, then return a single opaque token for the expression, some residue tokens from the mis-lex and then carry on as normal.
Regards
Ed Willink
|
|
|
Powered by
FUDForum. Page generated in 0.04130 seconds
Home