Re: [cdt-dev] lpg question

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [cdt-dev] lpg question

From: Mike Kucera <mkucera@xxxxxxxxxx>
Date: Tue, 13 Oct 2009 13:24:39 -0400
Delivered-to: cdt-dev@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/cdt-dev>
List-help: <mailto:cdt-dev-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/cdt-dev>, <mailto:cdt-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/listinfo/cdt-dev>, <mailto:cdt-dev-request@eclipse.org?subject=unsubscribe>

I'm not exactly sure what you're doing, but if you've changed the UPC grammar then probably thats why "shared int" isn't parsing anymore, because it works for me.

I'm not sure why "func" isn't working for you. I think you just have to make sure your grammar is correct. Maybe you should be adding "func" to the front of function_direct_declarator because basic_direct_declarator covers all declarators and not just functions.

I don't think the type strings shown in the oultline view are stored in the PDOM, but I could be wrong about that. The outline view shows what's in the file and I think all that comes from the AST.

Mike Kucera
Software Developer
IBM Toronto
mkucera@xxxxxxxxxx

David Sariel ---10/12/2009 08:02:57 PM---Thanks for the answers. 1) Considering the the 'func' the problem is different. I made the following simple test:

From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	10/12/2009 08:02 PM
Subject:	Re: [cdt-dev] lpg question

Thanks for the answers.
1) Considering the the 'func' the problem is different. I made the following simple test:
    - downloaded a CDT from cvs, added 'func' to the UPCKeywords enum (i.e. in DOMToUPCTokenMap
      it will be recognized as a keyword), changed the rule direct_declarator ::= basic_direct_declarator  to
      'func' basic_direct_declarator. Then I looked at the ast created for the line "shared int i;" The ast has
      CASTProblemDeclaration node for from offset 0 to 10 (i.e. "shared int" was recognized as a problem).
      I tried to track the problem and found that FixedBacktrackingParser.backtrackParse rerults in
     ERROR_ACTION as last action. Am I missing something?
2) Considering the labels in the outline view to be customized by implementors of
ILanguage it is something that requires changes in PDOM database, isn't it?

Thank you,
David

On Sun, Oct 11, 2009 at 5:56 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

I ran into the same problem with UPC, I wanted "shared int" to appear in the outline view. The problem is that CDT doesn't support this currently, and unfortunately I never got around to fixing it. So basically if you want it to work that way you would have to fix CDT. There is an open bug for this:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=196212

The grammar looks correct, so the problem might be that 'func' is not being recognized as a keyword. The tokes are generated by the CDT DOM preprocessor and must be converted from DOM tokens into LPG tokens using the token kinds that were generated by LPG. The problem is that the preprocessor doesn't know about any new keywords you add, and will return them as identifier tokens. The solution is to use the IDOMTokenMap interface. In the UPC token map I check for identifiers that match the new keywords like "shared" and convert them into the correct token type. Take a look at DOMToUPCTokenMap.

Your last question is a bit relative. I think it took me like 6 months to write the original C99 and UPC parsers. But keep in mind that I already had a lot of experience writing parsers at the time. Really, your milage may vary. I'm glad that you're reusing that stuff and I'll try to help as I can. But I know it can be difficult to reuse because the framework is actually very complicated and not documented that well.

Mike Kucera
Software Developer
IBM Toronto

mkucera@xxxxxxxxxx

David Sariel ---10/07/2009 03:54:34 PM---Hi Mike, Thanks a lot for the answer!


From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>

Date:	10/07/2009 03:54 PM

Subject:	Re: [cdt-dev] lpg question

Hi Mike,
Thanks a lot for the answer!

"Reduceing” my new language constructs into ones that already exist in C seems to me the fastest way to make things working. However, I’m still novice and some advice will be appreciated in order to make sure I’m in the right direction:

While trying to “reduce” the language constructs I played a little bit and changed functions to be declared “func foo(in param1, out param2, inout param3,…). In order to make that I used the rule

simple_type_specifier_token
::= interface_type -- Dropping all others (int, char, etc)

interface_type
    ::= 'in'
      | 'out'
      | '

inout

But code completion assumes that all params are of type int and suggests the suggestion “foo(int param1, int param2, int param3): int”. And in outline window I see foo(,,) - with no parameter types at all. The question is where the outline window and code completion decide which type the parameters are.

----------------------------------------------------------------------------------------------------
The second question is that in order to add func declarator I’ve dropped the rule

    direct_declarator
    ::= basic_direct_declarator

And instead added the rules and terminals:

    direct_declarator
    ::= func_directive basic_direct_declarator

    func_directive
      ::= '

func

'

FuncDirective ::= 'func' – new terminal

Also,

func(TK_FuncDirective)

was

added to the UPCKeyword enum

But, while parsing the file that has the only line

i	n	t		f	u	n	k		f	o	o	(	)	{	}
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16

I arrive to this in the first rule reduce:

external_declaration ::= ERROR_TOKEN (Offset: 0, Length: 8 calls consumeDeclarationProblem)

And the following AST was build:

Base Extensible Language AST:

org.eclipse.cdt.internal.core.dom.parser.c.CASTTranslationUnit (0,16)
org.eclipse.cdt.internal.core.dom.parser.c.CASTProblemDeclaration (0,8)
    org.eclipse.cdt.internal.core.dom.parser.c.CASTProblem (0,8)
org.eclipse.cdt.internal.core.dom.parser.c.CASTFunctionDefinition (9,7)
    org.eclipse.cdt.internal.core.dom.parser.upc.ast.UPCASTSimpleDeclSpecifier (0,0)
    org.eclipse.cdt.internal.core.dom.parser.c.CASTFunctionDeclarator (9,5)
      org.eclipse.cdt.internal.core.dom.parser.c.CASTName (9,3) foo
    org.eclipse.cdt.internal.core.dom.parser.c.CASTCompoundStatement (14,2)

Why “int func” is considered a problem? Am I missing something?

----------------------------------------------------------------------------------------------------

And the last question is just in order to assess times: how much time it took to design/write/test UPC?
Thanks a lot for the help
David

On Wed, Sep 30, 2009 at 5:00 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

Hi David,

If I understand what you're asking, the problem is that the CDT AST does not support statements as children to the translation unit node, but your language does.
I think there are a couple options you could explore. You could create some kind of "statement declaration" node that implements IASTDeclaration and add it to the translation unit. Or you could provide your own translation unit node that allows statements. Or you could wrap the statements in a dummy function declaration node.

In any case the consumeTranslationUnit() method can be overridden. Just create your own subclass and override that method and any others you need to, then in the grammar file set the

$build_action_class

macro to the name of your class. I think in an extending parser its ok to make direct reference to any new AST node types that you've created. The main purpose of the node factories is to allow the CDT parsers to share parsing rules between C and C++ while generating the correct AST. You don't have to follow this pattern in your own parser if it gets in your way.

But there is a deeper problem here. At the end of the day CDT only understands its own built-in AST nodes. If you generate an AST that differs significantly from what CDT expects then a lot of the features won't work. I got around this in the UPC parser by having all my new AST nodes implement node interfaces already predefined in CDT. For example UPC has a new kind of for loop that has an extra _expression_ in the header. I created a new kind of AST node for that but it extends CASTForStatement, so CDT usually treats it as a normal for loop. You may find that you have to "reduce" or "desugar" your new language constructs into ones that already exist in C otherwise CDT will get confused. If you really get stuck there is the possibility of making changes directly to CDT to support your language. We've done this in the past with a language called POP C++ or something like that.

Mike Kucera
Software Developer
IBM Eclipse CDT/PTP Team

mkucera@xxxxxxxxxx

David Sariel ---09/29/2009 01:05:20 PM---Hi Mike,


From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>

Date:	09/29/2009 01:05 PM
Subject:	Re: [cdt-dev] lpg question

Hi Mike,
Thanks a lot for the answer!
I have a written a grammar that was received by adding/dropping rules
in UPC/C99 and the generated parser seems reducing rules correctly.

Now I want to reuse as much as possible the parser action classes
(C99BuildASTParserAction, BuildASTParserAction, AbstractParserAction)
in the manner that will cause the
1) Syntax highlighting
2) Code outline
3) Code completion
to work with the new language (I have the part of keyword highlighting
already working).

The problem is that the language I’m working with has no declaration
and type sections that are presented in C99. Bellow are the rules that
bypass the declaration and type rules as I’ve defined:

119 172   directive ::= align_directive               (NEW)
120 173   directive ::= ds_directive                  (NEW)
121 174   directive ::= dcb_directive                (NEW)
122 175   directive ::= dcw_directive                (NEW)
123 176   directive ::= dcl_directive                 (NEW)
124 177   directive ::= list_directive                 (NEW)
125 178   directive ::= nolist_directive              (NEW)
126 179   directive ::= global_directive             (NEW)
127 180   directive ::= const_directive              (NEW)
128 181   directive ::= register_directive           (NEW)
129 182   directive ::= reserve_directive            (NEW)
130 183   directive ::= strict_directive               (NEW)
131 184   directive ::= nonstrict_directive         (NEW)
132 185   directive ::= sync_directive               (NEW)
133 186   directive ::= nosync_directive            (NEW)
134 187   directive ::= functioncall_attr             (NEW)
135 188   directive ::= func_declarator              (NEW)
136 189   directive ::= outofline_directive           (NEW)
137 190   directive ::= struct_or_union_specifier (NEW)
138 191   directive ::= enum_specifier               (NEW)

139 93    expression_statement ::= ;
140 94    expression_statement ::= expression_in_statement ;
141 154   expression_statement ::= directive ;   (NEW)

158 213 section_definition ::= SectionDirective identifier_token
compound_statement (NEW)

159 79    statement ::= labeled_statement
160 80    statement ::= compound_statement
161 81    statement ::= expression_statement
162 82    statement ::= selection_statement
163 83    statement ::= iteration_statement
164 84    statement ::= jump_statement
165 149   statement ::= include_directive              (NEW)
166 150   statement ::= inline_func_definition        (NEW)
167 151   statement ::= func_definition                 (NEW)
168 152   statement ::= section_definition             (NEW)
169 153   statement ::= ERROR_TOKEN

170 92    block_item ::= statement                       (NEW)
171 90    block_item_list ::= block_item
172 91    block_item_list ::= block_item_list block_item

173 211   translation_unit ::= block_item_list
174 212   translation_unit ::=
175 0     $accept ::= translation_unit
176 2     <empty> ::=

Hence, one of the problems I have is how to override the
BuildASTParserAction#consumeTranslationUnit method to add/get
statements to the TranslationUnit instead of
addDeclarations/getDeclaration methods (the problem is that
INodeFactory is annotated by @noextend, @noimplement).

How to overcome this?
Thank you,
David

On Mon, Sep 14, 2009 at 5:34 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

Hi David,

I take it what you're trying to do is have LPG print out which reductions are actually happening during a parse so that you can debug your parser.

As far as I know there is no way to automatically get LPG to do that for you, so you need to do it yourself. You'll notice that the information you need is at the top of the *.l files. So to make it work I wrote a little script that reads the *.l files and generates a Java class that contains a giant String array where the index is the rule number and the element is the rule. Then I added a couple lines of code to the top of the ruleAction() method in the generated parser that uses the array to print out the rules that are being fired.

Mike Kucera
Software Developer
IBM Eclipse CDT Team

mkucera@xxxxxxxxxx

David Sariel ---09/14/2009 04:12:35 AM---Hello,

From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	09/14/2009 04:12 AM
Subject:	[cdt-dev] lpg question

Hello, yacc (bison) supports --verbose and --debug options allowing automatic generation of debug outputs like: Reducing via rule 93 (line 329), statement -> statement_list ... Reducing via rule 185 (line 601), statement_list -> translation unit ... Does lpg analogs (-verbose and -debug switches) allow the same? I'm generating parsers with those options. *.l files indicate that VERBOSE, DEBUG are options in effect. But parser files generated are the same as without -verbose and -debug, i.e. no debugging outputs were generated. Am I missing something? Thanks, David _______________________________________________ cdt-dev mailing list