Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Using ANTLR or Xtext for YAML editor
Using ANTLR or Xtext for YAML editor [message #464172] Sun, 02 August 2009 18:28 Go to next message
No real name is currently offline No real nameFriend
Messages: 6
Registered: July 2009
Junior Member
Hi,

I am implementing an YAML (www.yaml.org) editor in Eclipse called YEdit
(http://code.google.com/p/yedit/).

The editor currently uses two parsers. One for the syntax high-lighting
and one to generate and outline view and to check the syntax.

The parser that is used for syntax highlighting is a rule based scanner
that works ok for its purpose, but it is not powerful enough to correctly
parse all YAML. It doesn't seem to be practical to extend the parser so
that it can parse all YAML either.

The parser used for the outline view and the syntax checking correctly
parses all YAML, but is meant to be used for static files. This has been
ok so far, but I have some ideas for new features that requires that the
editor is aware of the structure of the file continuously. Requiring a
complete re-parse of the entire file for each change seems impractical.

For these reason I have plans to either create a new YAML parser in ANTLR
or rewrite the editor to use Xtext. The problem is that it is hard for me
to evaluate which approach will be most sensible and I hope that someone
here could provide some insights. Some questions that are relevant.

1. Can Xtext correctly parse a language where whitespaces are significant?
Like YAML and Python.
2. How often does Xtext update the underlying model for a file?
3. Do ANTLR provide direct support for creating an incremental parser or
is this something that requires a bit of hacking.
4. Do Xtext perform well for large files (10000 lines or more) and complex
grammars?
5. Can error messages for syntax errors be tailored in Xtext to provide
the user with more help?
6. Any recommondations or other things that you can think of?

- Øystein Torget
Re: Using ANTLR or Xtext for YAML editor [message #464678 is a reply to message #464172] Sun, 02 August 2009 20:03 Go to previous messageGo to next message
Dénes Harmath is currently offline Dénes HarmathFriend
Messages: 157
Registered: July 2009
Senior Member
In my opinion, Xtext would be just the perfect choice for YAML! You could
avoid the duplication of parsers since Xtext infers everything from your
grammar definition, and has all the features (and more) that currently
YEdit has. Unfortunately AFAIK there is no EBNF specification for YAML,
but since you managed to write the ANTLR parser, it is not impossible, so
good luck!

Regarding the first question (whitespace significance), this is of course
possible, read Chapter 2 of
http://www.eclipse.org/Xtext/documentation/latest/xtext.html - especially
the section Terminal rules. I don't know the other answers, but I would be
interested in them, too. :)

thSoft
Re: Using ANTLR or Xtext for YAML editor [message #467846 is a reply to message #464172] Mon, 03 August 2009 07:08 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian ZarnekowFriend
Messages: 2940
Registered: July 2009
Senior Member
Hi Øystein,

please find my answer below.
Am 02.08.2009 20:28 Uhr, schrieb Øystein Torget:
> Hi,
>
> I am implementing an YAML (www.yaml.org) editor in Eclipse called YEdit
> (http://code.google.com/p/yedit/).
> The editor currently uses two parsers. One for the syntax high-lighting
> and one to generate and outline view and to check the syntax.
> The parser that is used for syntax highlighting is a rule based scanner
> that works ok for its purpose, but it is not powerful enough to
> correctly parse all YAML. It doesn't seem to be practical to extend the
> parser so that it can parse all YAML either.
> The parser used for the outline view and the syntax checking correctly
> parses all YAML, but is meant to be used for static files. This has been
> ok so far, but I have some ideas for new features that requires that the
> editor is aware of the structure of the file continuously. Requiring a
> complete re-parse of the entire file for each change seems impractical.
>
> For these reason I have plans to either create a new YAML parser in
> ANTLR or rewrite the editor to use Xtext. The problem is that it is hard
> for me to evaluate which approach will be most sensible and I hope that
> someone here could provide some insights. Some questions that are relevant.
> 1. Can Xtext correctly parse a language where whitespaces are
> significant? Like YAML and Python.

That is possible, but I guess it is quite a tough task. As far as I know
the ANTLR book contains an example on how to do this with ANTLR. Well I
think that it is not possibly to apply the trick with Lexer#emit on
Xtext generated grammars too but it should be possible to find a way to
describe indent and dedent.

> 2. How often does Xtext update the underlying model for a file?

That depends on how often you modify your input document. We track the
changes in the document and if the delay between typing and waiting is
long enough, Xtext parses the affected part of the document and merges
to previous semantic model with the newly created objects.

> 3. Do ANTLR provide direct support for creating an incremental parser or
> is this something that requires a bit of hacking.
You can call any ANTLR rule directly from the parser, but it is helpful
to be sure about the required lookahead and the state of the decision
tree. The answer is: is requires a bit of hacking.

> 4. Do Xtext perform well for large files (10000 lines or more) and
> complex grammars?
Xtext's performance does not depend on the complexity of your grammar
(althoug it is definitly helpful to provide a grammar that does not
require backtracking ...). Depending on your linking and validation
algorithms Xtext can handle this size more or less gracefully. We had
tests with 80.000 lines of code and it was possible to edit this files
altough there was some kind of latency.

> 5. Can error messages for syntax errors be tailored in Xtext to provide
> the user with more help?
That is not possible at the moment for error messages from the parser
and the linker. Validation messages and under your control.
> 6. Any recommondations or other things that you can think of?
>
I don't know YAML that well so currently there are no other things that
I can mention except from: Good luck :)
If you have any further questions, feel free to ask them.

> - Øystein Torget
>

Hope that helps,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com
Re: Using ANTLR or Xtext for YAML editor [message #479121 is a reply to message #467846] Sun, 09 August 2009 14:58 Go to previous messageGo to next message
No real name is currently offline No real nameFriend
Messages: 6
Registered: July 2009
Junior Member
Thanks for an informative reply.

I will probably start by getting a better understanding of ANTLR and grammers in general and the try to move over to Xtext when I start in earnest implementing the new version of YEdit.

- Øystein Torget
Re: Using ANTLR or Xtext for YAML editor [message #1063049 is a reply to message #467846] Tue, 11 June 2013 22:17 Go to previous messageGo to next message
Sebastian Sampaoli is currently offline Sebastian SampaoliFriend
Messages: 1
Registered: June 2013
Location: Argentina
Junior Member

Hi Øystein and Sebastian,

Right now, I am working with Xtext and a whitespace-sensitive grammar. The only problem which arose is the one you mentioned in:

> 1. Can Xtext correctly parse a language where whitespaces are
> significant? Like YAML and Python.

Do you have any solution for this issue? I couldn't find any example which can help me, and without solving this issue I can't to go on. I have read there is no practical solution for that kind of issue, but I would like to know even the impractical ones. I would really appreciate if you can help me. Many thanks.

Sebastian.
Re: Using ANTLR or Xtext for YAML editor [message #1389893 is a reply to message #479121] Sat, 28 June 2014 14:31 Go to previous message
Jack Greenfield is currently offline Jack GreenfieldFriend
Messages: 18
Registered: June 2014
Junior Member
Congratulations on the success of yedit.

I see that you did indeed use antlr. Wondering if you tried using xtext. If so, what issues did you encounter? I'm guessing that lexing was a challenge.

The motivation for these questions is that I'd like to leverage xtext for an EMF resource that reads and writes ECore models as YAML. Any insights on using xtext would be helpful.

[Updated on: Mon, 30 June 2014 20:29]

Report message to a moderator

Previous Topic:Best way of traversing a model: bottom up
Next Topic:Infinite Loop in DocumentTokenSource.computeDamageRegion
Goto Forum:
  


Current Time: Thu Nov 27 01:49:00 GMT 2014

Powered by FUDForum. Page generated in 0.03166 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software