Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Multiple LanguageServers vs one JVM process(How to design multiuser webapp integrated with XText ?)
Multiple LanguageServers vs one JVM process [message #1790760] Fri, 15 June 2018 20:37 Go to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Hi,

we are working on web application where users can write simple expressions via web forms. This application can be used simultaneously by few users. Application doesn't allow users to edit the same resource. We'be build web editor for our expression DSL with Monaco + LSP4J + XText.

When we started the project we didn't know too much about XText and LSP so we do it quite naively (?). When user opens the web page with expressions to edit (each expression is separate EMF resource) we start new LanguageServer. When user changes the script to edit we don't start new server, application changes the underlying resource. Code below is executed at the beginning of each user "session" (!= HttpSession), during this session user can edit many resources (one at a time):

LanguageServerImpl languageServer = injector.<LanguageServerImpl>getInstance(LanguageServerImpl.class);
Launcher<LanguageClient> launcher = LSPLauncher.createServerLauncher(languageServer,in,out);
languageServer.connect(launcher.getRemoteProxy());
launcher.startListening();


This works fine as long as we test it on a developer machine with just one concurrent user (only on language server is created at a time).

Now we are wondering if this approach will work with many concurrent users (test/prod environment)? All users have access to the same resources, but as I said application guards against concurrent modification of the same resource.

As far as I know XText today (still very poorly) XText(EMF?) uses static fields (registers) and is not designed to multithreaded environment ? if this is true how can we leverage power of XText in such architecture ? Do we have to lunch separate JVM process from our webapp for each "session" ? Or separate thread with dedicated classloader to "shield" one XTextServer from another ?

If my fears will be confirmed, "language server" as a name is a little confusing, because it suggests of multiuser support out-of-the-box (like ftp server, http server , chat server etc) :(

Daniel
Re: Multiple LanguageServers vs one JVM process [message #1790761 is a reply to message #1790760] Fri, 15 June 2018 20:46 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
since the lsp guys dont read here you may better ask this at: github.com/eclipse/lsp4j

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790762 is a reply to message #1790761] Fri, 15 June 2018 20:55 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
btw: how did you solve the adaption of websocket to in/out (i have done this is experiments only so far and it is ugly since most apis dont offer a stream level support)
and i stuble over the problem that the monaco language client does not send/expect the content length headers so i head to strip/add them manually


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790764 is a reply to message #1790762] Fri, 15 June 2018 21:17 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Looking at the API: LSPLauncher.createServerLauncher(languageServer,in,out) I don't know if there is an option to share launcher among users.

I wonder if it is possible to share XtextServer ? - should I create one instance per user or just one global instance and share it with many LSPLauncher ?- I found also that EMF is thread safe https://wiki.eclipse.org/EMF/FAQ#Is_EMF_thread-safe.3F)

Regarding WebScoket and LSP streaming API (input/output streams) we do this with our custom in/out stream implementations (warping same SpringBoot stuff) - its not a code I'm proud of because it is a little tricky. We still have some minor problems to solve (exceptions are thrown when we type too quickly in the editor:) ), but If you are interested we can share some code - we integrated LSP4J with SpringBoot WebSocket support - witch doesn't expose in/out streams :(

We also have to manually work around content-length header problem :) We lost many hours on this issue because LSP4J failed silently (as far as I remember) - there is issue for this on github, I think.
Re: Multiple LanguageServers vs one JVM process [message #1790767 is a reply to message #1790764] Fri, 15 June 2018 21:38 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
emf is not thread safe in all situations unfortunately " EMF does not, itself, ensure thread-safety"
yes the lsp4j github issue was created by me

i dont know if there exist concept for multiuser thus the hint to ask in an lsp4j issue (problem is more a conceptual/general thing)


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790938 is a reply to message #1790767] Wed, 20 June 2018 11:41 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Hi,

we have another conceptual problem - we don't know if such functionality is provided by XText/LanguageServer or we should implement this by our own.

Below is simplified version of our scenario:
1)User opens web page with list of resources. Each of this resources is a script of our expression DSL stored in DB (or file on disk - its irrelevant for this discussion)
2)User chooses one of this resources and edit it via Monaco editor. Script edition is supported (error checking, validation etc) by LanguageServer/XText
3)User chooses another script (resource) to edit (content of this resource shuold be loaded to editor)
4)After editing one or few resources user clicks "save" and all changed resources are persisted.

So far we worked on step 2) - we created expression grammar, integrated XText + LSP4J + Monaco etc. (We don't have any persistence mechanism so each time user started with empty Monato editor. )
Now we start working on 1) , 3) and 4) and we don't know if LanguageServer/XText provides any support in this area ?

For example, if we(for test purpose only) created file on disk with our DSL and initialized Monaco with uri pointing to this file, content of this file was parsed by Xtext and LSP but not content was visible in an editor. (if DSL was incorrect monaco correctly displayed validation errors but without content).
Another problem, is of retriving text content of each XtextResource when user clicks save button. XText resources contains AST (EObjects) not text entered in Monaco.

I have feeling that we miss some basic knowledge about XText/EMF/LanguageServer and we don't know where to start. Another option is that we are trying to do something that Xtext/LSP is not supporting out of the box. What we tried so far is to "bind" XTexty resources "physical recources" (files, records is DB) via URIHandlers/Resources/ResourceFactories but none of this worked.

When I've read LSP specification (about didOpen message) I concluded that content of file being opened is send from Monacto to LSP not vice-versa. Am I right ? (Its opossite what we assumed)
Whats more, we found that in LSP4J code that didSave message is not implemented.
I also found this https://www.eclipse.org/Xtext/documentation/330_web_support.html#getting-text-content but Its not about LSP :(

That's why we suspect that tasks 1) 3) and 4) from above list should be implemented by our own.
Before we start working on them I wonted to be sure that we don't miss anything ?

[Updated on: Wed, 20 June 2018 11:50]

Report message to a moderator

Re: Multiple LanguageServers vs one JVM process [message #1790959 is a reply to message #1790938] Wed, 20 June 2018 15:37 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
i dont think this is xtext specific.
as ls basically are designed to work with the same filesystem in client and backend for everything else you have to mimic that behaviour yourself.
its the same with other language servers.

=> during work the client sends changes to backend but appart from that the sharing of the files
has to happen over another channel (.e.g. using custom uri handling and processing for that uris)
=> saving is a client side thing
=> opening is a client side thing.


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790960 is a reply to message #1790959] Wed, 20 June 2018 15:38 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
p.p.s: as already said: typefox does not read the forum so you may be better of to ask this in github lsp4j (or xtext-core with mentioning miro and sven explicitely - still a very low chance of getting an answer)

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790962 is a reply to message #1790960] Wed, 20 June 2018 15:59 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
p.p.s:
in a single file scenario you might be able to mimic that by initializte from client with no uri and work with the open request only, but i dont know how well this is supported (have not tried it yet)

problematic with that is that all open files would go into one virtual project and see each other
=> will work only if
- you dont have refs between files
- you adapt scoping to ignore global scope
- you adapt the language servers "virtual project" handling


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Wed, 20 June 2018 17:13]

Report message to a moderator

Re: Multiple LanguageServers vs one JVM process [message #1790967 is a reply to message #1790962] Wed, 20 June 2018 16:28 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
did some experiments on that and at least in unit test it seems to work

 languageServer.initialize(new InitializeParams => [])
		val urix = 'inmemory:/foo/bar.testlang';
		val model = '''type Foo { }     xxxx'''
		open(urix, model);
		println(diagnostics.get(urix))
		languageServer.didChange(new DidChangeTextDocumentParams => [
			textDocument = new VersionedTextDocumentIdentifier => [
				uri = urix
				version = 2
			]
			contentChanges = #[
				new TextDocumentContentChangeEvent => [
					range = new Range(new Position(0, model.length - 5), new Position(0, model.length))
					text = ""
				]
			]
		])
		println(diagnostics.get(urix))


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1790997 is a reply to message #1790967] Thu, 21 June 2018 08:39 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Ok, so you confirmed what I suspected (open/save it not LS responsibility).

Regarding code above, what is "diagnostics" object ? How to obtain/create it ? Where we can find same information about such classes and LS internals ?

In our case we have refs between resources, and scopeProvider must be aware of EObjects from different files.

If LS is not responsible for open/save and we have cross refs between I imagine such solution:

1)When LS starts we should add existing resources to resourceset - to be parsed by Xtext (EObjects will be in index and cross refs and scoping will work)
2)When user starts editing one of this resources we should load content of this file into Monaco editor
3)After editing few resources, when users saves all of (modified/unmodified) resources (in one operation) we should get text content of each resource and persist is (disk/db) by our own.

Few questions about this scenario
In step 2) when we have to load content to monaco - we should get text content of resource being edited, we cant load id from persistent storage because this resource may be dirty (user can edit the same resource few times before save and we should obtain latest, still transient version).
Is it possible to get text content of xtext resource ? Is it "diagnostics.get(urix)" for ? If few files where modified before save, does Xtext/LS keep content for each of them and this solution will work ? If a file was loaded by LS (step1) and not yet edited if it also work ?

If this sollution will not work, I imagine we will finish with 3 "states" of each resource:
a)persisted unparsed (db/files) - updated after users clicks "save all"
b)transient-parsed (resources managed by XText/LSP) - contains eobjects - loaded in step 1) and updated by LS
c)transient-unparsed (text content of each resource to be able to load it to editor) - we will have to update them by our own (they are independent of xtext resources)

What I worry about is that the same resource have two representations b) and c) and we have to write code to keep them synchronized (when user type something in a editor LS will update parsed state and our code should update unparsed state). Do you see better solution ? If we have to do this, we should somehow intercept didChange operation and after LS changes XTextResource (b) we should update (c), any idea how implement such "interception" ?

If there is a solution not keeping separate set of unparsed resources (c) how can we access all resources from LS ? I asked you in separate topic, and solution you provided worked, but in this scenario we don't have access to another resource from the same resourceset we can't call: resouce.getResourceSet().

Hope, you understand me.I I appreciate your help so far.
Daniel
Re: Multiple LanguageServers vs one JVM process [message #1791001 is a reply to message #1790997] Thu, 21 June 2018 09:35 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
have a look at the tests in xtext-core.
the diagostics objects collects the validation results based on the open/change requests.
thus it allows to simulate what happens.

the inizialize project gets and uri. its then part of the server side to do whatever is needed to initially parse the project

yoiu have to care about all syncing and proper handling yourself


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792531 is a reply to message #1791001] Thu, 19 July 2018 07:44 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Hi,
we have another question :)

As I previously mentioned each of our resources represents one expression. For each resource we expect correct type to be "returned" by such expression (string/number/date etc).

Now we want to validate if expression "returns" expected type. I don't know where we should store such information (expected type for resource) to be easily accessible from our validator. I saw that EObjectDescription has userDataMap but this represents EObjects stored inside resource. We need something similar for whole resource. Is it possible ?
Re: Multiple LanguageServers vs one JVM process [message #1792534 is a reply to message #1792531] Thu, 19 July 2018 08:19 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
hi i dont understand that. please explain. and please use a different thread for different issues

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792543 is a reply to message #1792534] Thu, 19 July 2018 09:59 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Ok, I'll use separate thread next time.

I want to write type validator. Validator should check if expression like "a.b.c" returns expected type. Expected type is defined on resource level. For example we have 3 expressions (=resources). For each of this expression we know what returned type should be. We want to write validator and check if type returned by expression is the one expected by a resource. Validator should know what is expected type for this resource and we don't know what is the best place to store such information (there is no userDataMap on resource).

Is it clear now ?
Re: Multiple LanguageServers vs one JVM process [message #1792544 is a reply to message #1792543] Thu, 19 July 2018 10:23 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
what does determine the expected type of a resource?


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792612 is a reply to message #1792544] Fri, 20 July 2018 08:20 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Our resources represents mappings between data structures (these structures are not defined in the grammar, only mappings between them are). We developed language to map one data structure to another one (like xslt, xquery, etc).

If I have structure:
Struct{
Int a;
String b;
}

I can map field "a" and "b" to some other structure. For each of this fields I have to write mapping formula (in our DSL). So we will have two mappings (two XText resources), one for mapping field "a" and one for field "b"

For example:
For field "a" it cant be something like: "someOtherStruct.someField.someNestedField".

Now we want to write type validator to check if this formula "returns" Int type (look at Struct definition). We know type of someOtherStruct, someField and someNestedField but we have to compare it to the expected type of field a and we dont know where we should store this information to be easily accessible from Xtext validator.

Re: Multiple LanguageServers vs one JVM process [message #1792625 is a reply to message #1792612] Fri, 20 July 2018 10:30 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
yes but i dont undestand where your problem doing that is: so where is the place you dont get the information you do need

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792628 is a reply to message #1792625] Fri, 20 July 2018 10:45 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
The problem is that I don't know where I can store such information.

I would like to store this information inside EMF/Xtext objects - to be part of LS.

For EObjects there is EObjectDecription with userDataMap but our fields and structures are not part of EMF model. I looked at Resource and ResourceDescrptions classes but haven't found anything similar to userDataMap from EObjectDescription class.

Of course I can store this information outside LS and access this from validator class but it looks not elegant ;)
Re: Multiple LanguageServers vs one JVM process [message #1792638 is a reply to message #1792628] Fri, 20 July 2018 11:57 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
i still dont understand. of you have cross references you can simply follow them in your validator. or you use the userdata.
=> i have no clue where your actual problem is


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792719 is a reply to message #1792638] Mon, 23 July 2018 08:08 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
From the example above:

If you want to create mapping for field "a" in our editor you should write "someOtherStruct.someField.someNestedField" (not "a=someOtherStruct.someField.someNestedField"). So in our AST we don't have cross reference to "a" field.

In general we have two data structures "input" and "output" and we created DSL to map "output" to "input" fields. In our DSL user never references "output" structure directly (like a=....).
For each field in the "output" structure there is editor(=XtextResource) to enter the mapping from "input" (yes, we have cross references to "input" structure) .Now, when we have references to input structure in our DSL we know the type of "someOtherStruct", "someField" and "someNestedField" => we know that whole expression will return for example integer type. The problem is that "output" structure is not part of our grammar neither EMF model so we don't have cross reference to "a" field to compare its type with type of the expression - and that was your suggestion If I understood correctly.

Because output structure is not part of our DSL model (it is not needed because there it is not possible to reference to output from DSL) we need to store somewhere information of expected type of whole mapping(one mapping = one XtextResource) . The problem is, that we don't know were to store this information to access it in the validator.
Re: Multiple LanguageServers vs one JVM process [message #1792725 is a reply to message #1792719] Mon, 23 July 2018 09:15 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
still the question: where/when do you know which information to store? you can store stuff in the resource in the resourceset, in eadapers, in the validator, in the validator context, in a custom service that you call from the validator, ...
=> i dont know what fits your needs best


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Mon, 23 July 2018 09:29]

Report message to a moderator

Re: Multiple LanguageServers vs one JVM process [message #1792742 is a reply to message #1792725] Mon, 23 July 2018 12:56 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
We know that from the beginning. When user uploads "output" structure (ie XSD file) we create empty resource(empty mapping) for each field of this structure and we know the expected type for this mapping.
Re: Multiple LanguageServers vs one JVM process [message #1792743 is a reply to message #1792742] Mon, 23 July 2018 13:04 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
so you could add it e.g as adapter to the resource?

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792789 is a reply to message #1792743] Tue, 24 July 2018 06:46 Go to previous messageGo to next message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Maybe I could, but I don't know how, do you have same code snippet ?

I haven't found anything about adapters so far. Does adapter can store some extra info about xtext resource to be easily accessible from XTextValidator ?

In this example http://www.vogella.com/tutorials/EclipseEMFNotification/article.html adapters are used as listeners for EMF model changes. I don't understand how such listener can help me with described problem.

[Updated on: Tue, 24 July 2018 06:53]

Report message to a moderator

Re: Multiple LanguageServers vs one JVM process [message #1792798 is a reply to message #1792789] Tue, 24 July 2018 08:46 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12388
Registered: July 2009
Senior Member
xtext uses adapters all over the place

class MyAdaper extends AdaperImpl {

}
//write
MyAdaper myAdapter = ....
resource.eAdapers().add(myAdaper);

//read
for (Adaper : resource.adapers) {
if (adaper instanceof ....) {
// here
}
}


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Multiple LanguageServers vs one JVM process [message #1792821 is a reply to message #1792798] Tue, 24 July 2018 13:00 Go to previous message
Daniel Lipski is currently offline Daniel LipskiFriend
Messages: 20
Registered: May 2018
Junior Member
Ok, I misunderstood functionality of adapters - I thought that it is kind of listener of EMF model changes.

In your example I can add field to MyAdapter such as (Type expextedType) and treat this adapter as a place to store extra info about resource.

I'll try. Thanks a lot !
Previous Topic:Parameterized unit test
Next Topic:Whitespace without newlines
Goto Forum:
  


Current Time: Fri Nov 16 19:49:15 GMT 2018

Powered by FUDForum. Page generated in 0.03362 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top