[cdt-dev] LR parser and token generation

["Mario Pierro" <Mario.Pierro@xxxxxxx>]

Hello,

Another question on LR parser customization...

I am trying to add some custom extensions to the C99 language as
specified in the LR parser plugin. The extensions require both
additional keywords and additional grammar rules.

My ILanguage implementation extends the C99Language class, and provides
the custom C99Parser via its getParser() method. Additional keywords are
added via a custom ICLanguageKeywords implementation (as described in
http://dev.eclipse.org/mhonarc/lists/cdt-dev/msg15788.html) which
extends CLanguageKeywords and adds the new ones.

>From what I understood, my custom parser will process tokens which have
been produced by the CPreprocessor / Lexer classes - as the PDOM parser
does - and use a customized version of the DOMToC99TokenMap class to map
the preprocessor tokens (IToken interface) to the tokens in the
generated C99Parsersym class.

So if the parser defines new tokens, the CPreprocessor needs to know
about them as well. If I got it right, this can be done by having the
language class supply an implementation of
IScannerExtensionConfiguration, which associates the extended keywords
to token ids in the IExtensionToken interface in its addKeyword(char[],
int) method. 

Alternatively, the lexer can ignore the extensions altogether, and the
customized DOMToC99TokenMap class can determine if e.g. an "identifier"
token supplied by the lexer is actually an "extended keyword" token in
the parser.

A customized LR parser will thus be dependent on the tokens generated by
the preprocessor, no matter what its grammar specifies. Circumventing
this might be difficult, some characters might never be recognized as
the Lexer might not be generating any token at all (e.g. the '@' char).

I would like to use the same grammar for the lexer and the parser, so
that the token set is the same. 
Is this possible? Am I getting something terribly wrong here?

Thank you for your help!

/Mario