Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Dynamic Languages Toolkit (DLTK) » How useful can DLTK with a parser be?
How useful can DLTK with a parser be? [message #13395] Sat, 18 August 2007 02:18 Go to next message
Auerliano is currently offline AuerlianoFriend
Messages: 149
Registered: July 2009
Senior Member
Hi,

From your python guide, and as it was mentioned earlier in another
post[1], it seems DLTK crucially depends on an 'external parser'. In
other words, if I want to use DLTK to create an IDE for an arbitrary
language, say MATLAB, I will need to have an 'existing' parser or at
least I have to program my own parser. Is that right? In this case, DLTK
seems to be more of a 'user interface' than a 'toolkit'.

If the above is right, and if I want to start using DLTK for my
arbotrary language, where should I start to have a parser? Also, the
question introduces in [1] was left un-answered.



[1]
news://news.eclipse.org:119/3280bf2301d2b66861d57c4f47ed926e $1@www.eclipse.org
Re: How useful can DLTK with a parser be? [message #14399 is a reply to message #13395] Sat, 18 August 2007 02:21 Go to previous messageGo to next message
Auerliano is currently offline AuerlianoFriend
Messages: 149
Registered: July 2009
Senior Member
Oh, I should correct the subject to this:

How useful can DLTK *without* a parser be?
Re: How useful can DLTK with a parser be? [message #14456 is a reply to message #13395] Mon, 20 August 2007 08:11 Go to previous messageGo to next message
Andrei Sobolev is currently offline Andrei SobolevFriend
Messages: 72
Registered: July 2009
Member
Hi Ali,

> From your python guide, and as it was mentioned earlier in another
> post[1], it seems DLTK crucially depends on an 'external parser'. In
> other words, if I want to use DLTK to create an IDE for an arbitrary
> language, say MATLAB, I will need to have an 'existing' parser or at
> least I have to program my own parser. Is that right?

Yes, this is right.
If you want to create fully functional IDE for your language you need to
known language structure, so at least you need a parser. Parser will be
used by model building, search, completion, selection etc.

With DLTK you could use any kind of parser.
For example for Ruby we use JRuby original parser, for TCL we made our
own parser.

> In this case, DLTK
> seems to be more of a 'user interface' than a 'toolkit'.

In general most of DLTK functionality are in core. And of cause we
provide some UI to access core functionality.
>
> If the above is right, and if I want to start using DLTK for my
> arbotrary language, where should I start to have a parser? Also, the
> question introduces in [1] was left un-answered.

You could write your own parser or search for existing parsers.

For your own parser you could use ANTLR v3.x tool. (antlr.org).
It is quite easy to write parser you need.

Thanks.
Re: How useful can DLTK with a parser be? [message #14514 is a reply to message #14399] Mon, 20 August 2007 13:51 Go to previous messageGo to next message
Jae Gangemi is currently offline Jae GangemiFriend
Messages: 28
Registered: July 2009
Junior Member
On 2007-08-17 22:21:11 -0400, Ali <saveez@hotmail.com> said:

> Oh, I should correct the subject to this:
>
> How useful can DLTK *without* a parser be?

depends on the language. i don't have a working parser for my dltk
perl implementation, but i am able to launch and debug code, which is
what is most important to me at the moment.

anything editor based, though, is going to require a parser.

--
-jae
Re: How useful can DLTK with a parser be? [message #14545 is a reply to message #14456] Mon, 20 August 2007 18:02 Go to previous messageGo to next message
Auerliano is currently offline AuerlianoFriend
Messages: 149
Registered: July 2009
Senior Member
Hi Andrei

Andrei Sobolev wrote:
>
> You could write your own parser or search for existing parsers.

If I have an already existing parser in yacc/bison which is basically a
*.y file, could you explain step by step how it is possible to import
this existing parser to DLTK?

>
> For your own parser you could use ANTLR v3.x tool. (antlr.org).
> It is quite easy to write parser you need.
>

If I develop an extension of ANTRL for an arbitrary target language, can
i use the existing binding of ANTLR-DLTK or do I have to also develop a
new binding?
Re: How useful can DLTK with a parser be? [message #14666 is a reply to message #14545] Tue, 21 August 2007 06:18 Go to previous messageGo to next message
Andrei Sobolev is currently offline Andrei SobolevFriend
Messages: 72
Registered: July 2009
Member
Hi Ali,
> Hi Andrei
>
> Andrei Sobolev wrote:
>>
>> You could write your own parser or search for existing parsers.
>
> If I have an already existing parser in yacc/bison which is basically a
> *.y file, could you explain step by step how it is possible to import
> this existing parser to DLTK?

I suppose your parser convert module to some AST like presentation.
Also I suppose they are in Java.
If not you could port them to Java, and use something like jflex or
other.(I've found some flex/bison implementations for Java).

You could use your AST like presentation with DLTK.
Some of DLTK API uses our AST ( org.eclipse.dltk.ast package ).
So in some cases wrapping will be required.

For example we jave Rhino based AST for JavaScript implementation.
For Ruby we write visitor witch translate JRuby AST to our own AST.
Original JRuby AST not very useful for our needs, it is to complicated.

After what your could start with implementing ISourceElementParser to
build model.

I has very simple interface, so you only need to traverse trough your
AST and report elements to model.

You could checkout org.eclipse.dltk.core.tests plugin and obtain
TestSourceElementParser class for simple implementation of
ISourceElementParser interface.

After what you will get working script explorer, outline, and basic
search for declarations.

Andrei.

>
>>
>> For your own parser you could use ANTLR v3.x tool. (antlr.org).
>> It is quite easy to write parser you need.
>>
>
> If I develop an extension of ANTRL for an arbitrary target language, can
> i use the existing binding of ANTLR-DLTK or do I have to also develop a
> new binding?
Re: How useful can DLTK with a parser be? [message #14696 is a reply to message #14666] Wed, 22 August 2007 04:12 Go to previous messageGo to next message
Auerliano is currently offline AuerlianoFriend
Messages: 149
Registered: July 2009
Senior Member
Hi Andrei,

Let's assume that we have some java tool that can parse an expression in
our arbitrary language to a lisp-style expression. So, if we type

1 + 1

in the eclipse ditor, we can pass '1 + 1' as a string to this tool and
get this parsed expression in return:

(+ 1 1)

which is again in string format.


Can this be useful with DLTK? If yes, where should I pass this
lisp-style parse string to?
Re: How useful can DLTK with a parser be? [message #14725 is a reply to message #14696] Wed, 22 August 2007 17:16 Go to previous messageGo to next message
Jae Gangemi is currently offline Jae GangemiFriend
Messages: 28
Registered: July 2009
Junior Member
On 2007-08-22 00:12:09 -0400, Ali <saveez@hotmail.com> said:

> Hi Andrei,
>
> Let's assume that we have some java tool that can parse an expression
> in our arbitrary language to a lisp-style expression. So, if we type
>
> 1 + 1
>
> in the eclipse ditor, we can pass '1 + 1' as a string to this tool and
> get this parsed expression in return:
>
> (+ 1 1)
>
> which is again in string format.
>
>
> Can this be useful with DLTK? If yes, where should I pass this
> lisp-style parse string to?


take a look at the ISourceParser interface - that is what you will
want to implement to integrate w/ your parser. one of the parameters
the 'parse' method takes is the source contents as a char[] array,
which you will most likely need to convert to a String.

from there, however you want to do the actual parsing of the source
code is up to you, but the results of your parsing will need to be
converted into one of the ASTNode subclasses, and then added to the
ModuleDeclaration object that is returned from the 'parse' method.

--
-jae
Re: How useful can DLTK with a parser be? [message #14755 is a reply to message #14725] Thu, 23 August 2007 03:45 Go to previous messageGo to next message
Auerliano is currently offline AuerlianoFriend
Messages: 149
Registered: July 2009
Senior Member
Jae Gangemi wrote:

> take a look at the ISourceParser interface - that is what you will want
> to implement to integrate w/ your parser. one of the parameters the
> 'parse' method takes is the source contents as a char[] array, which you
> will most likely need to convert to a String.
>
> from there, however you want to do the actual parsing of the source
> code is up to you, but the results of your parsing will need to be
> converted into one of the ASTNode subclasses, and then added to the
> ModuleDeclaration object that is returned from the 'parse' method.
>

Hi Jae,

I just had a look at your perlipse source code to see if I can
understand the parsing flow. Documentation of DLTK is fairly poor and
somehow out-of-date, the source code is not documented by comments too,
so it's tedious job to understand the framework simply by hacking.


It seems that this is the parsing flow in perlipse:

(1) There is this perl script /scripts/parseSource.pl which does the
actual parsing in perl.

(2) The main perl parsing thing happens in PerlipseSourceParser.parse()
where perl is externally invoked to run the parsing perl script. I had
to change the perl binary address, as well as the one of the script, to
get the parsing on the track.

(3) An instance of ScriptExecuter runs the parsing script returning the
result as a String object called 'out'.

(3.a) Whatever script I run, this 'out' string is always empty, why?

(4) I did not understand this part: You used xstream to serialise 'out'
as xml. *If* 'out' is not empty, finally we can have the
ModuleDecleration to be connected to the parsed result via
addStatement() and through the xml serialisation object.

Can you also explain that what is the point of passing the parsed string
to a ModuleDeclaration object? More specifically, if the script contains:

1 + 1

and it is parsed to:

(+ 1 1)

then what is the benefit of passing this parsed string to the
ModuleDeclaration object?
Re: How useful can DLTK with a parser be? [message #14813 is a reply to message #14755] Thu, 23 August 2007 14:46 Go to previous message
Jae Gangemi is currently offline Jae GangemiFriend
Messages: 28
Registered: July 2009
Junior Member
On 2007-08-22 23:45:10 -0400, Ali <saveez@hotmail.com> said:

> Hi Jae,
>
> I just had a look at your perlipse source code to see if I can
> understand the parsing flow. Documentation of DLTK is fairly poor and
> somehow out-of-date, the source code is not documented by comments too,
> so it's tedious job to understand the framework simply by hacking.
>
>
> It seems that this is the parsing flow in perlipse:
>
> (1) There is this perl script /scripts/parseSource.pl which does the
> actual parsing in perl.
>
> (2) The main perl parsing thing happens in PerlipseSourceParser.parse()
> where perl is externally invoked to run the parsing perl script. I had
> to change the perl binary address, as well as the one of the script, to
> get the parsing on the track.
>
> (3) An instance of ScriptExecuter runs the parsing script returning the
> result as a String object called 'out'.
>
> (3.a) Whatever script I run, this 'out' string is always empty, why?
>
> (4) I did not understand this part: You used xstream to serialise 'out'
> as xml. *If* 'out' is not empty, finally we can have the
> ModuleDecleration to be connected to the parsed result via
> addStatement() and through the xml serialisation object.

heh - my parser doesn't work right now and i haven't made any serious
efforts to fix it - i'm more interested in getting the debugger working
b/c i am in need of it for the day job.

the basic idea of my parser is the following:

i use a perl script that is a wrapper around PPI (which can parse
perl 'documents') and i have that script output an xml document that i
then feed into xstream, which will then produce a ModuleDeclaration
object that would be returned.

> Can you also explain that what is the point of passing the parsed
> string to a ModuleDeclaration object? More specifically, if the script
> contains:
>
> 1 + 1
>
> and it is parsed to:
>
> (+ 1 1)
>
> then what is the benefit of passing this parsed string to the
> ModuleDeclaration object?

there is no benefit of that - i do not know much about the matlab
language, so i'm not sure what structure looks like, so i'll just give
a quick example w/ using perl.

if i have:

package Foo;

sub bar
{
print "hi\n";
}

1;

if i want 'Foo' to show up in the outline view, along w/ it's method
'bar', i would need to do this (this example will just use the dltk
classes, not the sub-classes i created (the integer values are all
based upon character offsets, and i make no guarentee the numbers sync
up in this example, but you get the general idea).

// length of source document
ModuleDeclaration decl = new ModuleDeclaration(50);

// pkg name, name start, name end, package start, package end
TypeDeclaration pkg = new TypeDeclaration("Foo", 0, 3, 0, 50);

Block body = new Block();
pkg.setBody(body);

// sub name, name, start, name end, sub start, sub end
MethodDeclaration sub = new MethodDeclaration("bar", 16, 19, 14, 45);
body.addStatement(sub);

declaration.addStatement(pkg);

---

i have not spent much more time in the parsing area other then this.
i know that there are additional ASTNode subclasses available that
represent expressions, conditionals, etc and possibily those would be
more of what you're interested in, but i am not sure how they would
need to plug into the overall ModuleDeclaration so that things like the
search, etc work.

--
-jae
Previous Topic:DLTK, Python and ANTLR
Next Topic:DLTK and a D Eclipse IDE
Goto Forum:
  


Current Time: Wed May 29 07:40:55 GMT 2024

Powered by FUDForum. Page generated in 0.04777 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top