Home » Eclipse Projects » Dynamic Languages Toolkit (DLTK) » How useful can DLTK with a parser be?
| How useful can DLTK with a parser be? [message #13395] |
Sat, 18 August 2007 02:18  |
 Auerliano
Messages: 149
Registered: July 2009 |
Senior Member |
|
|
Hi,
From your python guide, and as it was mentioned earlier in another
post[1], it seems DLTK crucially depends on an 'external parser'. In
other words, if I want to use DLTK to create an IDE for an arbitrary
language, say MATLAB, I will need to have an 'existing' parser or at
least I have to program my own parser. Is that right? In this case, DLTK
seems to be more of a 'user interface' than a 'toolkit'.
If the above is right, and if I want to start using DLTK for my
arbotrary language, where should I start to have a parser? Also, the
question introduces in [1] was left un-answered.
[1]
news://news.eclipse.org:119/3280bf2301d2b66861d57c4f47ed926e $1@www.eclipse.org
|
|
| |
| Re: How useful can DLTK with a parser be? [message #14456 is a reply to message #13395] |
Mon, 20 August 2007 08:11  |
Andrei Sobolev
Messages: 72
Registered: July 2009 |
Member |
|
|
Hi Ali,
> From your python guide, and as it was mentioned earlier in another
> post[1], it seems DLTK crucially depends on an 'external parser'. In
> other words, if I want to use DLTK to create an IDE for an arbitrary
> language, say MATLAB, I will need to have an 'existing' parser or at
> least I have to program my own parser. Is that right?
Yes, this is right.
If you want to create fully functional IDE for your language you need to
known language structure, so at least you need a parser. Parser will be
used by model building, search, completion, selection etc.
With DLTK you could use any kind of parser.
For example for Ruby we use JRuby original parser, for TCL we made our
own parser.
> In this case, DLTK
> seems to be more of a 'user interface' than a 'toolkit'.
In general most of DLTK functionality are in core. And of cause we
provide some UI to access core functionality.
>
> If the above is right, and if I want to start using DLTK for my
> arbotrary language, where should I start to have a parser? Also, the
> question introduces in [1] was left un-answered.
You could write your own parser or search for existing parsers.
For your own parser you could use ANTLR v3.x tool. (antlr.org).
It is quite easy to write parser you need.
Thanks.
|
|
| | | | |
| Re: How useful can DLTK with a parser be? [message #14725 is a reply to message #14696] |
Wed, 22 August 2007 17:16 |
Jae Gangemi
Messages: 28
Registered: July 2009 |
Junior Member |
|
|
On 2007-08-22 00:12:09 -0400, Ali <saveez@hotmail.com> said:
> Hi Andrei,
>
> Let's assume that we have some java tool that can parse an expression
> in our arbitrary language to a lisp-style expression. So, if we type
>
> 1 + 1
>
> in the eclipse ditor, we can pass '1 + 1' as a string to this tool and
> get this parsed expression in return:
>
> (+ 1 1)
>
> which is again in string format.
>
>
> Can this be useful with DLTK? If yes, where should I pass this
> lisp-style parse string to?
take a look at the ISourceParser interface - that is what you will
want to implement to integrate w/ your parser. one of the parameters
the 'parse' method takes is the source contents as a char[] array,
which you will most likely need to convert to a String.
from there, however you want to do the actual parsing of the source
code is up to you, but the results of your parsing will need to be
converted into one of the ASTNode subclasses, and then added to the
ModuleDeclaration object that is returned from the 'parse' method.
--
-jae
|
|
|
| Re: How useful can DLTK with a parser be? [message #14755 is a reply to message #14725] |
Thu, 23 August 2007 03:45 |
Auerliano
Messages: 149
Registered: July 2009 |
Senior Member |
|
|
Jae Gangemi wrote:
> take a look at the ISourceParser interface - that is what you will want
> to implement to integrate w/ your parser. one of the parameters the
> 'parse' method takes is the source contents as a char[] array, which you
> will most likely need to convert to a String.
>
> from there, however you want to do the actual parsing of the source
> code is up to you, but the results of your parsing will need to be
> converted into one of the ASTNode subclasses, and then added to the
> ModuleDeclaration object that is returned from the 'parse' method.
>
Hi Jae,
I just had a look at your perlipse source code to see if I can
understand the parsing flow. Documentation of DLTK is fairly poor and
somehow out-of-date, the source code is not documented by comments too,
so it's tedious job to understand the framework simply by hacking.
It seems that this is the parsing flow in perlipse:
(1) There is this perl script /scripts/parseSource.pl which does the
actual parsing in perl.
(2) The main perl parsing thing happens in PerlipseSourceParser.parse()
where perl is externally invoked to run the parsing perl script. I had
to change the perl binary address, as well as the one of the script, to
get the parsing on the track.
(3) An instance of ScriptExecuter runs the parsing script returning the
result as a String object called 'out'.
(3.a) Whatever script I run, this 'out' string is always empty, why?
(4) I did not understand this part: You used xstream to serialise 'out'
as xml. *If* 'out' is not empty, finally we can have the
ModuleDecleration to be connected to the parsed result via
addStatement() and through the xml serialisation object.
Can you also explain that what is the point of passing the parsed string
to a ModuleDeclaration object? More specifically, if the script contains:
1 + 1
and it is parsed to:
(+ 1 1)
then what is the benefit of passing this parsed string to the
ModuleDeclaration object?
|
|
|
| Re: How useful can DLTK with a parser be? [message #14813 is a reply to message #14755] |
Thu, 23 August 2007 14:46 |
Jae Gangemi
Messages: 28
Registered: July 2009 |
Junior Member |
|
|
On 2007-08-22 23:45:10 -0400, Ali <saveez@hotmail.com> said:
> Hi Jae,
>
> I just had a look at your perlipse source code to see if I can
> understand the parsing flow. Documentation of DLTK is fairly poor and
> somehow out-of-date, the source code is not documented by comments too,
> so it's tedious job to understand the framework simply by hacking.
>
>
> It seems that this is the parsing flow in perlipse:
>
> (1) There is this perl script /scripts/parseSource.pl which does the
> actual parsing in perl.
>
> (2) The main perl parsing thing happens in PerlipseSourceParser.parse()
> where perl is externally invoked to run the parsing perl script. I had
> to change the perl binary address, as well as the one of the script, to
> get the parsing on the track.
>
> (3) An instance of ScriptExecuter runs the parsing script returning the
> result as a String object called 'out'.
>
> (3.a) Whatever script I run, this 'out' string is always empty, why?
>
> (4) I did not understand this part: You used xstream to serialise 'out'
> as xml. *If* 'out' is not empty, finally we can have the
> ModuleDecleration to be connected to the parsed result via
> addStatement() and through the xml serialisation object.
heh - my parser doesn't work right now and i haven't made any serious
efforts to fix it - i'm more interested in getting the debugger working
b/c i am in need of it for the day job.
the basic idea of my parser is the following:
i use a perl script that is a wrapper around PPI (which can parse
perl 'documents') and i have that script output an xml document that i
then feed into xstream, which will then produce a ModuleDeclaration
object that would be returned.
> Can you also explain that what is the point of passing the parsed
> string to a ModuleDeclaration object? More specifically, if the script
> contains:
>
> 1 + 1
>
> and it is parsed to:
>
> (+ 1 1)
>
> then what is the benefit of passing this parsed string to the
> ModuleDeclaration object?
there is no benefit of that - i do not know much about the matlab
language, so i'm not sure what structure looks like, so i'll just give
a quick example w/ using perl.
if i have:
package Foo;
sub bar
{
print "hi\n";
}
1;
if i want 'Foo' to show up in the outline view, along w/ it's method
'bar', i would need to do this (this example will just use the dltk
classes, not the sub-classes i created (the integer values are all
based upon character offsets, and i make no guarentee the numbers sync
up in this example, but you get the general idea).
// length of source document
ModuleDeclaration decl = new ModuleDeclaration(50);
// pkg name, name start, name end, package start, package end
TypeDeclaration pkg = new TypeDeclaration("Foo", 0, 3, 0, 50);
Block body = new Block();
pkg.setBody(body);
// sub name, name, start, name end, sub start, sub end
MethodDeclaration sub = new MethodDeclaration("bar", 16, 19, 14, 45);
body.addStatement(sub);
declaration.addStatement(pkg);
---
i have not spent much more time in the parsing area other then this.
i know that there are additional ASTNode subclasses available that
represent expressions, conditionals, etc and possibily those would be
more of what you're interested in, but i am not sure how they would
need to plug into the overall ModuleDeclaration so that things like the
search, etc work.
--
-jae
|
|
|
Goto Forum:
Current Time: Sat Jul 27 10:56:47 GMT 2024
Powered by FUDForum. Page generated in 0.04191 seconds |
] 
Search
Help
Register
Login
Home
