Eclipse Community Forums: TMF (Xtext) » After Update to 2.11: Encoding Problem

Home » Modeling » TMF (Xtext) » After Update to 2.11: Encoding Problem

After Update to 2.11: Encoding Problem [message #1755443]

Fri, 03 March 2017 07:02

Eclipse User

We have a generator that creates files based on our dsl. One of our generator has a javadoc with special charaters ('ö') hardcoded within the generator.
Up to xtext 2.10.0 also the generated file contains the special character in correct encoding (UTF-8).
After migration to xtext 2.11.0 the special character is not encoded correctly in the generate file.

So far i have checked
- my PDE-Workspace
- my workspace using the feature.
- second (brand new) workspace using the feature (even on another machine)
- the generator xtend file; the generator java file
- the generated file
but all of them are encoded in UTF-8. But still the generated file has the special character not correctly encoded.

What make things worse:
If i test my feature from my PDE-workpace with the help of "Run as .. -> Eclipse Application" everything works fine (UTF-8 encoded file; special character encoded correctly)

Any ideas?

Attachment: generatorWithSpecialCharacters.png
(Size: 55.03KB, Downloaded 182 times)
Attachment: generatedFileWithEncodingFailure.png
(Size: 18.28KB, Downloaded 127 times)

Re: After Update to 2.11: Encoding Problem [message #1755445 is a reply to message #1755443]

Fri, 03 March 2017 07:13

Eclipse User

Hi,

did you (remote) debug org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2.generateFile(String, String, CharSequence)
is the encoding calculated there correct?
is the charsequence correct?
does the postprocessor behave correct?

[Updated on: Fri, 03 March 2017 07:17] by Moderator

Re: After Update to 2.11: Encoding Problem [message #1755456 is a reply to message #1755445]

Fri, 03 March 2017 08:55

Eclipse User

Just followed your proposal: Find attached a screenshot showing the debugging of a generator that has special characters set. As you can see, the generator java class (derived from xtend implementation of the generator) has the special character set correct. But after these string get appended to the StringConcatenation, the encoding seems to get strange.

Attachment: generatorJavaFileWithEncodingProblem.png
(Size: 156.72KB, Downloaded 139 times)

Re: After Update to 2.11: Encoding Problem [message #1755458 is a reply to message #1755456]

Fri, 03 March 2017 09:12

Eclipse User

hmmm,

which is your platform encoding?

during debugging you do a StringConcatenation.toString.

this might do the bad visuals.
but the interesting part should be where the string is actually written to the file.

=> which is the encoding xtext uses to write the file

StringInputStream newContent = getInputStream(contentsAsString, encoding); //<-- what is encoding here. is it maybe windows ascii or something like that?

further debugging at org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2.generateFile(IFile, InputStream, IFile, CharSequence, OutputConfiguration)
would be interesting as well.

Re: After Update to 2.11: Encoding Problem [message #1755462 is a reply to message #1755458]

Fri, 03 March 2017 09:33

Eclipse User

Sorry. Forgot to mention that. Encoding in the EclipseResourceFileSystemAccess2.generateFile(String, String, CharSequence) method is UTF-8.
But the provided parameter content (type CharSequence) already contains the mis-encoded special character. You see this as bad visual?

What do you mean with platform encoding? Workspace encoding is set to UTF-8 always, while operation system default encoding is CP1252.

[Updated on: Fri, 03 March 2017 09:36] by Moderator

Re: After Update to 2.11: Encoding Problem [message #1755463 is a reply to message #1755462]

Fri, 03 March 2017 09:39

Eclipse User

the interesting part/question is: what is inside the generator class file.

Re: After Update to 2.11: Encoding Problem [message #1755464 is a reply to message #1755463]

Fri, 03 March 2017 09:44

Eclipse User

=> again the question: how is the jar built that is deployed into eclipse

Re: After Update to 2.11: Encoding Problem [message #1755465 is a reply to message #1755464]

Fri, 03 March 2017 09:45

Eclipse User

(and i mean the classes jar, not the source jar)

Re: After Update to 2.11: Encoding Problem [message #1755468 is a reply to message #1755464]

Fri, 03 March 2017 09:49

Eclipse User

We export the feature with all plugins from within eclipse.
Right click on feature - project --> Export --> Deployable features --> Next -> Set Destination Directory to local folder and Set Option 'Package as individual JAR..' -> Finish

Re: After Update to 2.11: Encoding Problem [message #1755469 is a reply to message #1755468]

Fri, 03 March 2017 09:51

Eclipse User

can you turn the umlaut thing to a variable in the generator so that you can inspect it in the debugger?

Re: After Update to 2.11: Encoding Problem [message #1755471 is a reply to message #1755469]

Fri, 03 March 2017 10:04

Eclipse User

p.s. i tried to reproduce this and was not able to do so

Re: After Update to 2.11: Encoding Problem [message #1755474 is a reply to message #1755469]

Fri, 03 March 2017 10:12

Eclipse User

The variable value is also in a strange encoding. See ..DuringDebug Screenshot.

Attachment: generatorWithUmlautProperty.png
(Size: 71.78KB, Downloaded 129 times)
Attachment: generatorWithUmlautPropertyDuringDebug.png
(Size: 148.09KB, Downloaded 137 times)

Re: After Update to 2.11: Encoding Problem [message #1755475 is a reply to message #1755471]

Fri, 03 March 2017 10:13

Eclipse User

Christian Dietrich wrote on Fri, 03 March 2017 10:04

p.s. i tried to reproduce this and was not able to do so

That is bad (for me). I will try to setup my feature workspace again.

Re: After Update to 2.11: Encoding Problem [message #1755544 is a reply to message #1755475]

Sun, 05 March 2017 08:06

Eclipse User

Problem solved. Seem to be neither caused by xtext nor by update xtext 2.11.0.

Solution: When exporting the plugin as above mentioned, you have to check the button "Use class files compiled in the workspace".

Explanation: My operation system (windows 7 in german) has a default encoding set to windows-1252 (aka CP 1252). All settings in my eclipse workspace are set to UTF-8. But when exporting the feature (incl. plugins) some background process seem to compile the sources again. And this hidden compile does not consider the workspace settings. By checking "Use class files compiled in the workspace" during the export process there seems to be no "background compile". Instead the compiled files from withing eclipse (which consideres the workspace setting) are used.

Please correct my explanation where necessary.

Many thanks for your help.

[Updated on: Sun, 05 March 2017 11:36] by Moderator

Previous Topic:	xtext intellij build stuck in downloadIdea task
Next Topic:	How to run automatically generated java file has a method main in Xtext generator ?

Goto Forum:

-=] Back to Top [=-

Current Time: Mon May 12 21:12:14 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter