Eclipse Community Forums: GMT (Generative Modeling Technologies) » [xtext] Error in the generated PartitionScanner

Home » Archived » GMT (Generative Modeling Technologies) » [xtext] Error in the generated PartitionScanner

[xtext] Error in the generated PartitionScanner [message #384445]

Wed, 06 August 2008 11:24

Eclipse User

by taking a look at the generated antlr grammar, the STRING token takes
care of escaping character \ (which is a really good thing :-)

RULE_STRING :

'"' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'"') )* '"' |
'\'' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'\'') )* '\''

;

However, the generated GeneratedPartitionScanner.java in the .editor
plugin, does not consider the escape character:

public class GeneratedPartitionScanner extends AbstractPartitionScanner {

@Override
public List<IPredicateRule> getRules() {
List<IPredicateRule> rules = new ArrayList<IPredicateRule>();

...
rules.add(new MultiLineRule("\"","\"", string));
rules.add(new MultiLineRule("'","'", string));
return rules;
}

in fact, in the generated editor the highlighting of a string of the shape

"this is \"a test\""

does not generate an error (right!) but it is not highlighted correctly.

The generated partition scanner should consider the escape character:

public class GeneratedPartitionScanner extends AbstractPartitionScanner {

@Override
public List<IPredicateRule> getRules() {
List<IPredicateRule> rules = new ArrayList<IPredicateRule>();

...
rules.add(new MultiLineRule("\"","\"", string, '\\'));
rules.add(new MultiLineRule("'","'", string, '\\'));
return rules;
}

where should I fix this?
if I modify the generated partition scanner directly I assume it will be
overwritten...

thanks
Lorenzo

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

Re: [xtext] Error in the generated PartitionScanner [message #384650 is a reply to message #384445]

Fri, 08 August 2008 02:27

Eclipse User

Hi Lorenzo,

it would be best if you could provide a patch and file it in bugzilla,
so that the issue is fixed for everybody :-)

In general if you want to configure a special partition scanner you
should subclass the generated [LanguageName]Utilities.java class in the
editor project. The subclass should of course go into the src/ folder.
Then change the [LanguageName]EditorPlugin class so that it instantiates
the new subclass instead of the generated class.

In the manually created utility class you can overwrite the method:

@Override
public IPartitionTokenScanner getPartitionScanner() {
// TODO Auto-generated method stub
return super.getPartitionScanner();
}

Note, that this is how you can replace other built-in functionality as
well (e.g. content assist).

hope that helps,
Sven

Lorenzo Bettini schrieb:
> by taking a look at the generated antlr grammar, the STRING token takes
> care of escaping character \ (which is a really good thing :-)
>
> RULE_STRING :
>
> '"' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'"') )* '"' |
> '\'' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'\'') )*
> '\''
>
> ;
>
> However, the generated GeneratedPartitionScanner.java in the .editor
> plugin, does not consider the escape character:
>
> public class GeneratedPartitionScanner extends AbstractPartitionScanner {
>
> @Override
> public List<IPredicateRule> getRules() {
> List<IPredicateRule> rules = new ArrayList<IPredicateRule>();
>
> ...
> rules.add(new MultiLineRule("\"","\"", string));
> rules.add(new MultiLineRule("'","'", string));
> return rules;
> }
>
> in fact, in the generated editor the highlighting of a string of the shape
>
> "this is \"a test\""
>
> does not generate an error (right!) but it is not highlighted correctly.
>
> The generated partition scanner should consider the escape character:
>
> public class GeneratedPartitionScanner extends AbstractPartitionScanner {
>
> @Override
> public List<IPredicateRule> getRules() {
> List<IPredicateRule> rules = new ArrayList<IPredicateRule>();
>
> ...
> rules.add(new MultiLineRule("\"","\"", string, '\\'));
> rules.add(new MultiLineRule("'","'", string, '\\'));
> return rules;
> }
>
> where should I fix this?
> if I modify the generated partition scanner directly I assume it will be
> overwritten...
>
> thanks
> Lorenzo
>

Re: [xtext] Error in the generated PartitionScanner [message #384659 is a reply to message #384650]

Fri, 08 August 2008 05:29

Eclipse User

Sven Efftinge wrote:
> Hi Lorenzo,
>
> it would be best if you could provide a patch and file it in bugzilla,
> so that the issue is fixed for everybody :-)

I'll try to do this, but it might not be easy for someone external to
the development of xtext code itself to try to figure out where to patch ;-)

anyway, I'm trying to download the sources; I assume the right procedure
is the one described in the wiki am I right?

>
> In general if you want to configure a special partition scanner you
> should subclass the generated [LanguageName]Utilities.java class in the
> editor project. The subclass should of course go into the src/ folder.
> Then change the [LanguageName]EditorPlugin class so that it instantiates
> the new subclass instead of the generated class.
>
> In the manually created utility class you can overwrite the method:
>
> @Override
> public IPartitionTokenScanner getPartitionScanner() {
> // TODO Auto-generated method stub
> return super.getPartitionScanner();
> }
>

OK, I managed to figure this out and already did that :-)

Note that if you override a standard rule, as for instance the single
comment rule, the generated partition scanner is not updated
accordingly... so you end up with an highlighting scheme that is not
consistent with the parser... was that intentional or is it a bug?

> Note, that this is how you can replace other built-in functionality as
> well (e.g. content assist).
>
> hope that helps,
> Sven
>
> Lorenzo Bettini schrieb:
>> by taking a look at the generated antlr grammar, the STRING token takes
>> care of escaping character \ (which is a really good thing :-)
>>
>> RULE_STRING :
>>
>> '"' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'"') )*
>> '"' |
>> '\'' ( '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\') | ~('\\'|'\'')
>> )* '\''
>> ;
>>
>> However, the generated GeneratedPartitionScanner.java in the .editor
>> plugin, does not consider the escape character:
>>
>> public class GeneratedPartitionScanner extends AbstractPartitionScanner {
>>
>> @Override
>> public List<IPredicateRule> getRules() {
>> List<IPredicateRule> rules = new ArrayList<IPredicateRule>();
>>
>> ...
>> rules.add(new MultiLineRule("\"","\"", string));
>> rules.add(new MultiLineRule("'","'", string));
>> return rules;
>> }
>>
>> in fact, in the generated editor the highlighting of a string of the
>> shape
>>
>> "this is \"a test\""
>>
>> does not generate an error (right!) but it is not highlighted correctly.
>>
>> The generated partition scanner should consider the escape character:
>>
>> public class GeneratedPartitionScanner extends AbstractPartitionScanner {
>>
>> @Override
>> public List<IPredicateRule> getRules() {
>> List<IPredicateRule> rules = new ArrayList<IPredicateRule>();
>>
>> ...
>> rules.add(new MultiLineRule("\"","\"", string, '\\'));
>> rules.add(new MultiLineRule("'","'", string, '\\'));
>> return rules;
>> }
>>
>> where should I fix this?
>> if I modify the generated partition scanner directly I assume it will be
>> overwritten...
>>
>> thanks
>> Lorenzo
>>

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

Re: [xtext] Error in the generated PartitionScanner [message #384663 is a reply to message #384650]

Fri, 08 August 2008 12:47

Eclipse User

This is a multi-part message in MIME format.
--------------070706060603020106050406
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Sven Efftinge wrote:
> Hi Lorenzo,
>
> it would be best if you could provide a patch and file it in bugzilla,
> so that the issue is fixed for everybody :-)
>

there seem to be problems with bugzilla, so for the moment I'm posting
the patch here

hope this helps
Lorenzo

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

--------------070706060603020106050406
Content-Type: text/x-diff;
name="scanner_with_escape_chars_for_strings.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: inline;
filename="scanner_with_escape_chars_for_strings.patch"

### Eclipse Workspace Patch 1.0
#P xtext.generator
Index: main/src/org/openarchitectureware/xtext/editor/Utilities.xpt
============================================================ =======
RCS file: /cvsroot/architecturware/oaw_v4/xtext/xtext.generator/main/s rc/org/openarchitectureware/xtext/editor/Utilities.xpt,v
retrieving revision 1.6.2.3.2.1
diff -u -r1.6.2.3.2.1 Utilities.xpt
--- main/src/org/openarchitectureware/xtext/editor/Utilities.xpt 31 May 2008 12:57:18 -0000 1.6.2.3.2.1
+++ main/src/org/openarchitectureware/xtext/editor/Utilities.xpt 8 Aug 2008 16:16:50 -0000
@@ -103,8 +103,8 @@
rules.add(new MultiLineRule("/*","*/", comment));
rules.add(new SingleLineRule("//", "", comment));
�ENDIF-�
- rules.add(new MultiLineRule("\"","\"", string));
- rules.add(new MultiLineRule("'","'", string));
+ rules.add(new MultiLineRule("\"","\"", string, '\\'));
+ rules.add(new MultiLineRule("'","'", string, '\\'));
return rules;
}

--------------070706060603020106050406--

Re: [xtext] Error in the generated PartitionScanner [message #384664 is a reply to message #384650]

Fri, 08 August 2008 12:47

Eclipse User

This is a multi-part message in MIME format.
--------------030806020307050809080108
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Sven Efftinge wrote:
> Hi Lorenzo,
>
> it would be best if you could provide a patch and file it in bugzilla,
> so that the issue is fixed for everybody :-)
>

there seem to be problems with bugzilla, so for the moment I'm posting
the patch here

hope this helps
Lorenzo

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

--------------030806020307050809080108
Content-Type: text/x-diff;
name="scanner_with_escape_chars_for_strings.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: inline;
filename="scanner_with_escape_chars_for_strings.patch"

### Eclipse Workspace Patch 1.0
#P xtext.generator
Index: main/src/org/openarchitectureware/xtext/editor/Utilities.xpt
============================================================ =======
RCS file: /cvsroot/architecturware/oaw_v4/xtext/xtext.generator/main/s rc/org/openarchitectureware/xtext/editor/Utilities.xpt,v
retrieving revision 1.6.2.3.2.1
diff -u -r1.6.2.3.2.1 Utilities.xpt
--- main/src/org/openarchitectureware/xtext/editor/Utilities.xpt 31 May 2008 12:57:18 -0000 1.6.2.3.2.1
+++ main/src/org/openarchitectureware/xtext/editor/Utilities.xpt 8 Aug 2008 16:16:50 -0000
@@ -103,8 +103,8 @@
rules.add(new MultiLineRule("/*","*/", comment));
rules.add(new SingleLineRule("//", "", comment));
�ENDIF-�
- rules.add(new MultiLineRule("\"","\"", string));
- rules.add(new MultiLineRule("'","'", string));
+ rules.add(new MultiLineRule("\"","\"", string, '\\'));
+ rules.add(new MultiLineRule("'","'", string, '\\'));
return rules;
}

--------------030806020307050809080108--

Re: [xtext] Error in the generated PartitionScanner [message #384665 is a reply to message #384664]

Fri, 08 August 2008 12:52

Eclipse User

Lorenzo Bettini wrote:
> Sven Efftinge wrote:
>> Hi Lorenzo,
>>
>> it would be best if you could provide a patch and file it in bugzilla,
>> so that the issue is fixed for everybody :-)
>>
>
> there seem to be problems with bugzilla, so for the moment I'm posting
> the patch here
>

OK: https://bugs.eclipse.org/bugs/show_bug.cgi?id=243608

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

Re: [xtext] Error in the generated PartitionScanner [message #384779 is a reply to message #384665]

Wed, 13 August 2008 11:23

Eclipse User

This also has already been applied by Patrick.

Thank you!

Lorenzo Bettini schrieb:
> Lorenzo Bettini wrote:
>> Sven Efftinge wrote:
>>> Hi Lorenzo,
>>>
>>> it would be best if you could provide a patch and file it in
>>> bugzilla, so that the issue is fixed for everybody :-)
>>>
>>
>> there seem to be problems with bugzilla, so for the moment I'm posting
>> the patch here
>>
>
> OK: https://bugs.eclipse.org/bugs/show_bug.cgi?id=243608
>

Re: [xtext] Error in the generated PartitionScanner [message #384781 is a reply to message #384779]

Wed, 13 August 2008 12:35

Eclipse User

thank you for xtext :-)

Sven Efftinge wrote:
> This also has already been applied by Patrick.
>
> Thank you!
>
>
> Lorenzo Bettini schrieb:
>> Lorenzo Bettini wrote:
>>> Sven Efftinge wrote:
>>>> Hi Lorenzo,
>>>>
>>>> it would be best if you could provide a patch and file it in
>>>> bugzilla, so that the issue is fixed for everybody :-)
>>>>
>>>
>>> there seem to be problems with bugzilla, so for the moment I'm
>>> posting the patch here
>>>
>>
>> OK: https://bugs.eclipse.org/bugs/show_bug.cgi?id=243608
>>

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

Re: [xtext] Error in the generated PartitionScanner [message #384880 is a reply to message #384781]

Fri, 22 August 2008 04:22

Eclipse User

It seems to me that the problem is that the STRING definition is
incorrect. See this thread for my explanation of why I think this:

http://www.openarchitectureware.org/forum/viewtopic.php?show topic=9640

Also, should I be posting on this newsgroup or that one?

Thanks,
-- Scott

Re: [xtext] Error in the generated PartitionScanner [message #384890 is a reply to message #384880]

Fri, 22 August 2008 07:50

Eclipse User

For the currently released xtext (shipped with oAW 4.3) it's best to use
the forum over at openarchitectureware.org.
If people have questions about the new Xtext (the one currently
developed under TMF) they should aks here (eclipse.modeling.tmf).
Since there are no releases yet, there are no questions :-)

regards,
Sven

Scott schrieb:
> It seems to me that the problem is that the STRING definition is
> incorrect. See this thread for my explanation of why I think this:
>
> http://www.openarchitectureware.org/forum/viewtopic.php?show topic=9640
>
> Also, should I be posting on this newsgroup or that one?
>
> Thanks,
> -- Scott
>

Re: [xtext] Error in the generated PartitionScanner [message #384901 is a reply to message #384880]

Sat, 23 August 2008 13:10

Eclipse User

Scott wrote:
> It seems to me that the problem is that the STRING definition is
> incorrect. See this thread for my explanation of why I think this:
>
> http://www.openarchitectureware.org/forum/viewtopic.php?show topic=9640
>
> Also, should I be posting on this newsgroup or that one?
>

As I also answered to that thread:

Actually, concerning the bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=243608, which
now should be fixed in the CVS, the strings with backslashed quotes used
to work correctly as before (the parser used to parse it correctly), it
was the highlighter that was wrong...

--
Lorenzo Bettini, PhD in Computer Science, DI, Univ. Torino
ICQ# lbetto, 16080134 (GNU/Linux User # 158233)
HOME: http://www.lorenzobettini.it MUSIC: http://www.purplesucker.com
http://www.myspace.com/supertrouperabba
BLOGS: http://tronprog.blogspot.com http://longlivemusic.blogspot.com
http://www.gnu.org/software/src-highlite
http://www.gnu.org/software/gengetopt
http://www.gnu.org/software/gengen http://doublecpp.sourceforge.net

Re: [xtext] Error in the generated PartitionScanner [message #618902 is a reply to message #384445]

Fri, 08 August 2008 02:27

Eclipse User