Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » After Update to 2.11: Encoding Problem
After Update to 2.11: Encoding Problem [message #1755443] Fri, 03 March 2017 12:02 Go to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
We have a generator that creates files based on our dsl. One of our generator has a javadoc with special charaters ('ö') hardcoded within the generator.
Up to xtext 2.10.0 also the generated file contains the special character in correct encoding (UTF-8).
After migration to xtext 2.11.0 the special character is not encoded correctly in the generate file.

So far i have checked
- my PDE-Workspace
- my workspace using the feature.
- second (brand new) workspace using the feature (even on another machine)
- the generator xtend file; the generator java file
- the generated file
but all of them are encoded in UTF-8. But still the generated file has the special character not correctly encoded.

What make things worse:
If i test my feature from my PDE-workpace with the help of "Run as .. -> Eclipse Application" everything works fine (UTF-8 encoded file; special character encoded correctly)

Any ideas?

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755445 is a reply to message #1755443] Fri, 03 March 2017 12:13 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
Hi,

did you (remote) debug org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2.generateFile(String, String, CharSequence)
is the encoding calculated there correct?
is the charsequence correct?
does the postprocessor behave correct?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

[Updated on: Fri, 03 March 2017 12:17]

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755456 is a reply to message #1755445] Fri, 03 March 2017 13:55 Go to previous messageGo to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
Just followed your proposal: Find attached a screenshot showing the debugging of a generator that has special characters set. As you can see, the generator java class (derived from xtend implementation of the generator) has the special character set correct. But after these string get appended to the StringConcatenation, the encoding seems to get strange.

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755458 is a reply to message #1755456] Fri, 03 March 2017 14:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
hmmm,

which is your platform encoding?

during debugging you do a StringConcatenation.toString.

this might do the bad visuals.
but the interesting part should be where the string is actually written to the file.

=> which is the encoding xtext uses to write the file

StringInputStream newContent = getInputStream(contentsAsString, encoding); //<-- what is encoding here. is it maybe windows ascii or something like that?

further debugging at org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2.generateFile(IFile, InputStream, IFile, CharSequence, OutputConfiguration)
would be interesting as well.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755462 is a reply to message #1755458] Fri, 03 March 2017 14:33 Go to previous messageGo to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
Sorry. Forgot to mention that. Encoding in the EclipseResourceFileSystemAccess2.generateFile(String, String, CharSequence) method is UTF-8.
But the provided parameter content (type CharSequence) already contains the mis-encoded special character. You see this as bad visual?

What do you mean with platform encoding? Workspace encoding is set to UTF-8 always, while operation system default encoding is CP1252.

[Updated on: Fri, 03 March 2017 14:36]

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755463 is a reply to message #1755462] Fri, 03 March 2017 14:39 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
the interesting part/question is: what is inside the generator class file.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755464 is a reply to message #1755463] Fri, 03 March 2017 14:44 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
=> again the question: how is the jar built that is deployed into eclipse

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755465 is a reply to message #1755464] Fri, 03 March 2017 14:45 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
(and i mean the classes jar, not the source jar)

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755468 is a reply to message #1755464] Fri, 03 March 2017 14:49 Go to previous messageGo to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
We export the feature with all plugins from within eclipse.
Right click on feature - project --> Export --> Deployable features --> Next -> Set Destination Directory to local folder and Set Option 'Package as individual JAR..' -> Finish

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755469 is a reply to message #1755468] Fri, 03 March 2017 14:51 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
can you turn the umlaut thing to a variable in the generator so that you can inspect it in the debugger?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755471 is a reply to message #1755469] Fri, 03 March 2017 15:04 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14735
Registered: July 2009
Senior Member
p.s. i tried to reproduce this and was not able to do so

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755474 is a reply to message #1755469] Fri, 03 March 2017 15:12 Go to previous messageGo to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
The variable value is also in a strange encoding. See ..DuringDebug Screenshot.

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755475 is a reply to message #1755471] Fri, 03 March 2017 15:13 Go to previous messageGo to next message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
Christian Dietrich wrote on Fri, 03 March 2017 10:04
p.s. i tried to reproduce this and was not able to do so

That is bad (for me). I will try to setup my feature workspace again.

Report message to a moderator

Re: After Update to 2.11: Encoding Problem [message #1755544 is a reply to message #1755475] Sun, 05 March 2017 13:06 Go to previous message
Klaus H. is currently offline Klaus H.Friend
Messages: 14
Registered: July 2009
Junior Member
Problem solved. Seem to be neither caused by xtext nor by update xtext 2.11.0.

Solution: When exporting the plugin as above mentioned, you have to check the button "Use class files compiled in the workspace".

Explanation: My operation system (windows 7 in german) has a default encoding set to windows-1252 (aka CP 1252). All settings in my eclipse workspace are set to UTF-8. But when exporting the feature (incl. plugins) some background process seem to compile the sources again. And this hidden compile does not consider the workspace settings. By checking "Use class files compiled in the workspace" during the export process there seems to be no "background compile". Instead the compiled files from withing eclipse (which consideres the workspace setting) are used.

Please correct my explanation where necessary.

Many thanks for your help.

[Updated on: Sun, 05 March 2017 16:36]

Report message to a moderator

Previous Topic:xtext intellij build stuck in downloadIdea task
Next Topic:How to run automatically generated java file has a method main in Xtext generator ?
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Sun Dec 01 20:48:33 GMT 2024

Powered by FUDForum. Page generated in 0.06273 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top

Sign up to our Newsletter

A fresh new issue delivered monthly