Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » IMP » Test of generated parser/lexer
Test of generated parser/lexer [message #19445] Thu, 22 May 2008 15:31 Go to next message
Eclipse User
Originally posted by: lars.howhere.se

Hi forum,

I'm developing a parser with LPG, and need to expriment with the content of
the definition files (.g /.gi)
I was wondering if there is any easy way to test the parser during
development?

/Lars
Re: Test of generated parser/lexer [message #19627 is a reply to message #19445] Sat, 24 May 2008 22:51 Go to previous messageGo to next message
Jin Missing name is currently offline Jin Missing name
Messages: 100
Registered: July 2009
Senior Member
re-generating and re-running testcases?

"Lars" <lars@howhere.se> д
Re: Test of generated parser/lexer [message #19672 is a reply to message #19627] Sun, 25 May 2008 05:14 Go to previous messageGo to next message
Eclipse User
Originally posted by: lars.howhere.se

Hi,

For sure executing testcases is a good idea, but how do I create the
testcase in the first place?
Can fragments be generated by LPG somehow or do I have to manually create it
all?

BR

/Lars




"Jin" <jin.phd@gmail.com> skrev i meddelandet
news:g1ak74$sg3$1@build.eclipse.org...
> re-generating and re-running testcases?
>
> "Lars" <lars@howhere.se> д
Re: Test of generated parser/lexer [message #19717 is a reply to message #19672] Tue, 27 May 2008 10:14 Go to previous messageGo to next message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Lars wrote:
> Hi,
>
> For sure executing testcases is a good idea, but how do I create the
> testcase in the first place?
> Can fragments be generated by LPG somehow or do I have to manually create it
> all?

Depends on what you're trying to test, of course. Are you trying to
validate the grammar, the AST's, or some other aspect of the front
end?

We've been talking for a long time about adding some grammar debugging
features to the LPG IDE, like "Generate Sentence" (which generates a
pseudo-random sentence in the sub-language described by the selected
non-terminal), "Show First Set" (which shows the terminals that can
appear in the prefix of a selected non-terminal), and so on. [You can
see some place-holder actions in the LPG IDE's source context menu.]

Unfortunately, we haven't found the bandwidth to do any of this (though
there is some half-baked code in the LPG IDE to start on this). Part of
the problem is that a lot of the necessary analyses to enable this sort
of functionality is buried in the LPG generator (written in C++). We'd
like to avoid duplication of that analysis in the Java-implemented LPG
IDE, but at the same time, we'd like to avoid a messy interface between
the IDE and the generator (which could result from trying to expose the
analyses to the IDE). Another complication is that we'd sort of like to
re-implement the parser generator in Java, but again, we don't have the
bandwidth for that sort of lateral move.

If you're talking about validating the AST's, I assume you're writing
your own.

At the moment, the LPG generator by default creates classes that contain
static data for the parser and scanner tables. Philippe recently did
some work to have the generator create ordinary data files to house the
tables, and to make sure all of the language-specific classes (AST's,
etc.) are pure, with no static data, so they can be loaded and unloaded
cleanly. With those changes, it should be possible to load the necessary

> "Jin" <jin.phd@gmail.com> skrev i meddelandet
> news:g1ak74$sg3$1@build.eclipse.org...
>> re-generating and re-running testcases?
>>
>> "Lars" <lars@howhere.se> дÈëÏûÏ¢ÐÂÎÅ:g14hlv$nno$1@build.eclipse.org...
>>> Hi forum,
>>>
>>> I'm developing a parser with LPG, and need to expriment with the content
>>> of the definition files (.g /.gi)
>>> I was wondering if there is any easy way to test the parser during
>>> development?


--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://www.eclipse.org/imp)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Re: Test of generated parser/lexer [message #19744 is a reply to message #19445] Tue, 27 May 2008 10:25 Go to previous message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Lars wrote:
> Hi forum,
>
> I'm developing a parser with LPG, and need to expriment with the content of
> the definition files (.g /.gi)
> I was wondering if there is any easy way to test the parser during
> development?

[Oops! Previous post when out prematurely; chalk that up to working
on too many different platforms (Mac, Win, Linux) these days.]

Depends on what you're trying to test, of course. Are you trying to
validate the grammar, the AST's, or some other aspect of the front
end?

We've been talking for a long time about adding some grammar debugging
features to the LPG IDE, like "Generate Sentence" (which generates a
pseudo-random sentence in the sub-language described by the selected
non-terminal), "Show First Set" (which shows the terminals that can
appear in the prefix of a selected non-terminal), and so on. [You can
see some place-holder actions in the LPG IDE's source context menu.]

Unfortunately, we haven't found the bandwidth to do any of this (though
there is some half-baked code in the LPG IDE to start on this). Part of
the problem is that a lot of the necessary analyses to enable this sort
of functionality is buried in the LPG generator (written in C++). We'd
like to avoid duplication of that analysis in the Java-implemented LPG
IDE, but at the same time, we'd like to avoid a messy interface between
the IDE and the generator (which could result from trying to expose the
analyses to the IDE). Another complication is that we'd sort of like to
re-implement the parser generator in Java, but again, we don't have the
bandwidth for that sort of lateral move.

If you're talking about validating the AST's, I assume you're writing
your own or are using some pre-existing AST's. [Of course you can trust
LPG to generate perfectly correct AST's that never need testing. ;-)]
If we could generate sentences for a given non-terminal, we'd have some
of what you need, but of course such a generator would most likely be
fairly stupid, and in particular wouldn't respect the many constraints
that many real languages insist on (so almost every test input would
fail in name binding, type checking, or somesuch analysis).

In terms of testing the grammar by simply running the parser on various
inputs, at this point, the biggest obstacle to grammar development may
be the inability to run the parser directly *in the same IDE instance*.

At the moment, the LPG generator by default creates classes that contain
static data for the parser and scanner tables. Philippe recently did
some work to have the generator create ordinary data files to house the
tables, and to make sure all of the language-specific classes (AST's,
etc.) are pure, with no static data, so they can be loaded and unloaded
cleanly. With those changes, it should be possible to load the necessary
classes into the same Eclipse and run it. We haven't yet gotten to that.
I'll ask Philippe to chime in with his thoughts on the subject.

--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://www.eclipse.org/imp)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Re: Test of generated parser/lexer [message #570533 is a reply to message #19445] Sat, 24 May 2008 22:51 Go to previous message
Jin Missing name is currently offline Jin Missing name
Messages: 100
Registered: July 2009
Senior Member
re-generating and re-running testcases?

"Lars" <lars@howhere.se> д
Re: Test of generated parser/lexer [message #570589 is a reply to message #19627] Sun, 25 May 2008 05:14 Go to previous message
Lars is currently offline Lars
Messages: 32
Registered: July 2009
Member
Hi,

For sure executing testcases is a good idea, but how do I create the
testcase in the first place?
Can fragments be generated by LPG somehow or do I have to manually create it
all?

BR

/Lars




"Jin" <jin.phd@gmail.com> skrev i meddelandet
news:g1ak74$sg3$1@build.eclipse.org...
> re-generating and re-running testcases?
>
> "Lars" <lars@howhere.se> д
Re: Test of generated parser/lexer [message #570614 is a reply to message #19672] Tue, 27 May 2008 10:14 Go to previous message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Lars wrote:
> Hi,
>
> For sure executing testcases is a good idea, but how do I create the
> testcase in the first place?
> Can fragments be generated by LPG somehow or do I have to manually create it
> all?

Depends on what you're trying to test, of course. Are you trying to
validate the grammar, the AST's, or some other aspect of the front
end?

We've been talking for a long time about adding some grammar debugging
features to the LPG IDE, like "Generate Sentence" (which generates a
pseudo-random sentence in the sub-language described by the selected
non-terminal), "Show First Set" (which shows the terminals that can
appear in the prefix of a selected non-terminal), and so on. [You can
see some place-holder actions in the LPG IDE's source context menu.]

Unfortunately, we haven't found the bandwidth to do any of this (though
there is some half-baked code in the LPG IDE to start on this). Part of
the problem is that a lot of the necessary analyses to enable this sort
of functionality is buried in the LPG generator (written in C++). We'd
like to avoid duplication of that analysis in the Java-implemented LPG
IDE, but at the same time, we'd like to avoid a messy interface between
the IDE and the generator (which could result from trying to expose the
analyses to the IDE). Another complication is that we'd sort of like to
re-implement the parser generator in Java, but again, we don't have the
bandwidth for that sort of lateral move.

If you're talking about validating the AST's, I assume you're writing
your own.

At the moment, the LPG generator by default creates classes that contain
static data for the parser and scanner tables. Philippe recently did
some work to have the generator create ordinary data files to house the
tables, and to make sure all of the language-specific classes (AST's,
etc.) are pure, with no static data, so they can be loaded and unloaded
cleanly. With those changes, it should be possible to load the necessary

> "Jin" <jin.phd@gmail.com> skrev i meddelandet
> news:g1ak74$sg3$1@build.eclipse.org...
>> re-generating and re-running testcases?
>>
>> "Lars" <lars@howhere.se> дÈëÏûÏ¢ÐÂÎÅ:g14hlv$nno$1@build.eclipse.org...
>>> Hi forum,
>>>
>>> I'm developing a parser with LPG, and need to expriment with the content
>>> of the definition files (.g /.gi)
>>> I was wondering if there is any easy way to test the parser during
>>> development?


--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://www.eclipse.org/imp)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Re: Test of generated parser/lexer [message #570634 is a reply to message #19445] Tue, 27 May 2008 10:25 Go to previous message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Lars wrote:
> Hi forum,
>
> I'm developing a parser with LPG, and need to expriment with the content of
> the definition files (.g /.gi)
> I was wondering if there is any easy way to test the parser during
> development?

[Oops! Previous post when out prematurely; chalk that up to working
on too many different platforms (Mac, Win, Linux) these days.]

Depends on what you're trying to test, of course. Are you trying to
validate the grammar, the AST's, or some other aspect of the front
end?

We've been talking for a long time about adding some grammar debugging
features to the LPG IDE, like "Generate Sentence" (which generates a
pseudo-random sentence in the sub-language described by the selected
non-terminal), "Show First Set" (which shows the terminals that can
appear in the prefix of a selected non-terminal), and so on. [You can
see some place-holder actions in the LPG IDE's source context menu.]

Unfortunately, we haven't found the bandwidth to do any of this (though
there is some half-baked code in the LPG IDE to start on this). Part of
the problem is that a lot of the necessary analyses to enable this sort
of functionality is buried in the LPG generator (written in C++). We'd
like to avoid duplication of that analysis in the Java-implemented LPG
IDE, but at the same time, we'd like to avoid a messy interface between
the IDE and the generator (which could result from trying to expose the
analyses to the IDE). Another complication is that we'd sort of like to
re-implement the parser generator in Java, but again, we don't have the
bandwidth for that sort of lateral move.

If you're talking about validating the AST's, I assume you're writing
your own or are using some pre-existing AST's. [Of course you can trust
LPG to generate perfectly correct AST's that never need testing. ;-)]
If we could generate sentences for a given non-terminal, we'd have some
of what you need, but of course such a generator would most likely be
fairly stupid, and in particular wouldn't respect the many constraints
that many real languages insist on (so almost every test input would
fail in name binding, type checking, or somesuch analysis).

In terms of testing the grammar by simply running the parser on various
inputs, at this point, the biggest obstacle to grammar development may
be the inability to run the parser directly *in the same IDE instance*.

At the moment, the LPG generator by default creates classes that contain
static data for the parser and scanner tables. Philippe recently did
some work to have the generator create ordinary data files to house the
tables, and to make sure all of the language-specific classes (AST's,
etc.) are pure, with no static data, so they can be loaded and unloaded
cleanly. With those changes, it should be possible to load the necessary
classes into the same Eclipse and run it. We haven't yet gotten to that.
I'll ask Philippe to chime in with his thoughts on the subject.

--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://www.eclipse.org/imp)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Previous Topic:Test of generated parser/lexer
Next Topic:IMP on Eclipse 3.3
Goto Forum:
  


Current Time: Wed Aug 20 22:38:17 EDT 2014

Powered by FUDForum. Page generated in 0.01893 seconds