Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » GMT (Generative Modeling Technologies) » [TCS] Symbols, operators, tokens, and lexer section
[TCS] Symbols, operators, tokens, and lexer section [message #379828] Tue, 02 October 2007 16:28 Go to next message
Eclipse UserFriend
Originally posted by: rivera.lcc.uma.es

Hi all,

I have been reading some questions (and answers) here in the newsgroup
and i have still some doubts about lexer section, symbols, operators and
tokens.

* Can i write all the lexer section using tokens (in TCS language)? If
so, where can i find information about how to use tokens?
* Are declared symbols tokens by itself?
* What kind of symbols do i have to declare?All of them?Only the ones
used as keywords in templates?Only the ones used to recognize tokens?
If for example "<" and ">" are used to recognize a special token (lexer
section), do i have to declare them?
* And what about operators? are just used to give priorities to some
"symbols"?

Sorry for asking so many things together, but I'm a begineer a little
bit confused :S
Thanks in advance,
Jose E. Rivera
Re: [TCS] Symbols, operators, tokens, and lexer section [message #379858 is a reply to message #379828] Sat, 06 October 2007 20:18 Go to previous message
Frédéric Jouault is currently offline Frédéric JouaultFriend
Messages: 572
Registered: July 2009
Senior Member
Hi,

> I have been reading some questions (and answers) here in the newsgroup
> and i have still some doubts about lexer section, symbols, operators and
> tokens.
>
> * Can i write all the lexer section using tokens (in TCS language)? If
> so, where can i find information about how to use tokens?

Tokens still have some shortcomings which makes using them only a bit
difficult right now.


> * Are declared symbols tokens by itself?

A lexer rule is created for both declared symbols and tokens.
So, there are similar.


> * What kind of symbols do i have to declare?All of them?Only the ones
> used as keywords in templates?Only the ones used to recognize tokens?
> If for example "<" and ">" are used to recognize a special token (lexer
> section), do i have to declare them?

You should declare all the symbols that you use, but only if there are
used in the syntax.

For instance, if double quotes (") are used to delimit strings, there
are not syntactical entities by themselves, but part of STRING.
Therefore, there should not appear in the list of symbols.


> * And what about operators? are just used to give priorities to some
> "symbols"?

Operators are used to represent expression-like constructs.

Technically, you can detect that you need operators if there is a
left-recursion that you cannot factorize simply. Then, TCS operator
support can help you.

Do not hesitate to use the TCS syntax definitions from the TCS Zoo as
examples:
http://wiki.eclipse.org/TCS/Zoo


> Sorry for asking so many things together, but I'm a begineer a little
> bit confused :S

No problem :-).


Regards,

Frédéric Jouault
Re: [TCS] Symbols, operators, tokens, and lexer section [message #608056 is a reply to message #379828] Sat, 06 October 2007 20:18 Go to previous message
Frédéric Jouault is currently offline Frédéric JouaultFriend
Messages: 572
Registered: July 2009
Senior Member
Hi,

> I have been reading some questions (and answers) here in the newsgroup
> and i have still some doubts about lexer section, symbols, operators and
> tokens.
>
> * Can i write all the lexer section using tokens (in TCS language)? If
> so, where can i find information about how to use tokens?

Tokens still have some shortcomings which makes using them only a bit
difficult right now.


> * Are declared symbols tokens by itself?

A lexer rule is created for both declared symbols and tokens.
So, there are similar.


> * What kind of symbols do i have to declare?All of them?Only the ones
> used as keywords in templates?Only the ones used to recognize tokens?
> If for example "<" and ">" are used to recognize a special token (lexer
> section), do i have to declare them?

You should declare all the symbols that you use, but only if there are
used in the syntax.

For instance, if double quotes (") are used to delimit strings, there
are not syntactical entities by themselves, but part of STRING.
Therefore, there should not appear in the list of symbols.


> * And what about operators? are just used to give priorities to some
> "symbols"?

Operators are used to represent expression-like constructs.

Technically, you can detect that you need operators if there is a
left-recursion that you cannot factorize simply. Then, TCS operator
support can help you.

Do not hesitate to use the TCS syntax definitions from the TCS Zoo as
examples:
http://wiki.eclipse.org/TCS/Zoo


> Sorry for asking so many things together, but I'm a begineer a little
> bit confused :S

No problem :-).


Regards,

Frédéric Jouault
Previous Topic:Conversion from GME paradigms and models...
Next Topic:[TCS - Transformations Outside Eclipse ]
Goto Forum:
  


Current Time: Thu Mar 28 11:21:00 GMT 2024

Powered by FUDForum. Page generated in 0.51518 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top