Re: [jdt-dev] Jikespg syntax description used in ...compiler.batch/gramm

[Alexander Kriegisch <Alexander@xxxxxxxxxxxxxx>]

Hi Stephan.

Thanks for the useful pointers.

BTW, I did not say that it is sophisticated, merely that I am
unexperienced in lexing and parsing or building compilers in general.
Hence the inquiry about documentation.

Most related lexer/parser generators use some flavour (E)BNF, but even
reading BNF is a new thing for me. For instance, I was irritated by the
use of both '::=' and '->' in productions. Only a cursory source code
comment [1] mentions that they both are meant to be "equivalence
symbols", which I understand as them being equivalent and their usage
rather a matter of semantic subtext or even personal developer
preference rather than them actually having different effects.

Thanks for offering to ask specific questions, but in the meantime I was
able to solve the issue [2] at hand in this commit [3]. The regression
test for it in AspectJ is here [4]. Getting it to be LALR(1) - a term I
only vaguely understand at this point - was actually the most difficult
part.

BTW, the way I was able to debug this was to set Parser.DEBUG to true
and compile a JDT Core version with it, running my regression test
against it, watching what was actually being parsed. Is there a more
elegant way to debug grammar problems? And is there an IntelliJ or
Eclipse plugin helping me to highlight the syntax of and navigate or
even code-complete stuff in files like java.g? Especially the navigation
part was difficult, repeatedly jumping around by searching for
production names. Please also note that the AspectJ grammar is more
complex than the plain Java grammar, the latter being just a subset of
the former.

[1] https://github.com/jikespg/jikespg/blob/master/src/lpgparse.c#L3262
[2] https://github.com/eclipse/org.aspectj/issues/20
[3] https://github.com/eclipse/aspectj.eclipse.jdt.core/commit/6b54ee63
[4] https://github.com/eclipse/org.aspectj/commit/5f46d440

--
Alexander Kriegisch
https://scrum-master.de


Stephan Herrmann schrieb am 02.01.2023 um 22:01:
> Hi Alexander,
> 
> Actually, the grammar language isn't very sophisticated, I'd say. At least the 
> portion that JDT uses. The actual grammar rules are a plain-vanilla variant of 
> BNF. Few more things are used like "/.$putCase ... ./" to generate case 
> statements inside Parser.consumeRule() and "/:$readableName ... :/" to provide 
> names for use in error messages. Ah, and of course "/:$compliance ...:/" 
> specifies since which version a given rule is allowed. "$empty" speaks for 
> itself, doesn't it?
> 
> I know that making arbitrary grammars comply to the LALR(1) rules can be very 
> tricky, but that's not a problem of the input language, is it?
> 
> I suggest you ask specific questions, so that those who have been editing java.g 
> over the years (despite the absence of documentation) can provide specific help 
> from their experience.
> 
> best,
> Stephan
> 
> 
> Am 02.01.23 um 10:13 schrieb Alexander Kriegisch:
>> Hi there from the AspectJ team.
>>
>> We are having a little problem with the Java language grammar we adjusted for native AspectJ syntax extensions. I am unexperienced in building lexers and parsers and only partly understand the syntax of the grammar file. In order to tackle the issue at hand, I would like to learn how to read a Jikespg grammar, but cannot find any resource describing it. Is the syntax borrowed from any other tool in the lexer and parser generator realm? Lex, Yacc, ANTLR, you name it. I adjusted a few things in the AspectJ grammar in the past after merging in upstream changes from JDT Core, but I did not change anything substantial there. I really wish to understand better what I am doing there. Therefore, I would be glad to receive any helpful replies. Even small hints will be welcome.
>>
>> Thank you in advance and kind regards
>