After a longer time of trial and error I found a way to override the lexer that is generated by Xtext using some internal knowledge how Xtext seems to work and think. I found
a way to trace the tokens that are generated by the lexer. This makes it far easier to understand what goes wrong in a DSL definition.
a way to insert, split, and redefine tokens in the token stream, giving them additional synthetic terminal types.
a way to insert tokens with zero length making it possible to keep the inserted tokens in sync with the editor.
What is the best way to share the information? After searching the web I think there is some demand to find some answers to these questions and I would preferably like to add something to the Xtext documentation instead of putting it somewhere on the web in a separate blog.
Karsten Thoms Messages: 563 Registered: July 2009 Location: Dortmund, Germany
Hi Markus! The Xtext manual is written in a manner to describe general concepts and common extension machanisms. You are solving a very specific use case. We usually don't add such things to the manual.
To share your solution approch you might consider creating a minimal sample project on GitHub and write the challenge and the specific solution to the project's README.MD. You may write a blog post about that, too, and pointing to your example project.