Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » N-to-N code generation
N-to-N code generation [message #1859358] Fri, 02 June 2023 09:42 Go to next message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
The `IGenerator2` seems to assume 1-to-N code generation, that is, for one DSL file there may be any number of generated output files. This makes sense for supporting incremental builds.

However, we have use cases where this relation differs, and I'm not sure about the best approach to handle these.

=== Use case 1: generate a package structure per namespace. ===

Our DSL files start with the declaration of a namespace, much like in Java, e.g. `namespace com.foo.bar`. Multiple DSL files might have the same namespace.

One of the languages we generate to is Python, which uses a convention of having one `__init__.py` file per namespace. How does this fit into Xtext's generation infrastructure?

Example: suppose we have three DSL files.
- `a.dsl`, with namespace `com.foo.bar`
- `b.dsl`, with namespace `com.foo.bar`
- `c.dsl` with namespace `com.foo.util`
I would like to generate the following layout:
com
  - foo
    - bar
      - __init__.py
      - a.py
      - b.py
    - util
      - __init__.py
      - c.py


This is definitely our most important use case, for which we currently don't have a workaround.

=== Use case 2: generating project files. ===

It would be useful to have a way of generating project files such as a file for dependency management.

To take the example of Python again (but this is relevant for almost all languages), we would like to automatically generate a `pyproject.toml` file or a `requirements.txt` file for defining a list of dependencies, and potential metadata such as the project name and version.

Questions:
- How does this fit into Xtext's generation infrastructure? Are there hooks that fire once before all generation takes place and after all generation has finished?
- Is it possible to access context of the project such as the name and the version during generation? (e.g., the name/version of the Maven project when generating using the `xtext-maven-plugin`, or the name of the `ProjectDescription` when generating using the Xtext language server)

Right now we have a workaround of providing these files statically per project/target language.

=== Use case 3: generating patches. ===

Our language is constantly evolving, and being time pressured we find it sometimes convenient to temporarily generate patches for our dependencies. Right now we do so in the `afterGenerate` hook, which is called for every input DSL file, meaning we regenerate these patches many many times although a single generation would suffice.

Context

The use cases we want to support is generation via Maven (e.g., the `xtext-maven-plugin`) and generation via Xtext's language server.

What I've tried so far is extending the `IGenerator2` interface and providing a `beforeAllGenerate` and `afterAllGenerate` hook, which have access to the entire resource set. I've experimented with an extension of the `xtext-maven-plugin` which will actually use these calls, but which will will probably not fit into incremental builds. Additionally, I've noticed that the `IGenerator2` instance is not reused by the builder, but a new instance is created for each DSL file, meaning it is not possible to maintain state in the code generator.

These are difficult problems that might be solvable, but I just wonder:
- is there out-of-the-box Xtext support for this?
- are there alternative approaches for these use cases?
- have people encountered similar problems? What were their solutions?
I did see this StackOverflow post, but there wasn't really a solution: https://stackoverflow.com/questions/43904786/xtext-generation-from-multiple-model-files

[Updated on: Fri, 09 June 2023 12:45]

Report message to a moderator

Re: N-to-N code generation [message #1859359 is a reply to message #1859358] Fri, 02 June 2023 09:46 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14722
Registered: July 2009
Senior Member
maybe you can simply add an edadapter as marker to the resourceset
and check that marker in all generators
and then generate if the marker is not there.
as long as you dont use parallel generation this should work fine.

incremental deletes wont work in eclipse then i guess.
so you might have to remove that feature then


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

[Updated on: Fri, 02 June 2023 09:55]

Report message to a moderator

Re: N-to-N code generation [message #1859362 is a reply to message #1859359] Fri, 02 June 2023 10:20 Go to previous messageGo to next message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
Yeah, I guess that could work for implementing a `beforeAllGenerate` hook. Is it possible to disable incremental deletes in the language server as well?

EDIT: I guess this EAdaptor marker would have to be cleared between changes as well (for the language server use case). Not sure how that would work.

[Updated on: Fri, 02 June 2023 10:24]

Report message to a moderator

Re: N-to-N code generation [message #1859363 is a reply to message #1859362] Fri, 02 June 2023 10:25 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14722
Registered: July 2009
Senior Member
does the resourceset stay the same in lsp?
you need to read the code.
org.eclipse.xtext.ide.server.ProjectManager.newBuildRequest(List<URI>, List<URI>, List<Delta>, CancelIndicator) seems to create new

org.eclipse.xtext.generator.OutputConfiguration.isCleanUpDerivedResources()
about derived resources
of course you can overrride everythign using guice


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

[Updated on: Fri, 02 June 2023 10:38]

Report message to a moderator

Re: N-to-N code generation [message #1859365 is a reply to message #1859363] Fri, 02 June 2023 12:23 Go to previous messageGo to next message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
Quote:
org.eclipse.xtext.ide.server.ProjectManager.newBuildRequest(List<URI>, List<URI>, List<Delta>, CancelIndicator) seems to create new

Interesting, I did not realize that. That would indeed make the EAdaptor marker option feasible.

Quote:
org.eclipse.xtext.generator.OutputConfiguration.isCleanUpDerivedResources()

Having glanced at the code, setting this option to `false` would disable incremental deletes, is that right?

I think I'll use that workaround for now. In the future, I might look into proper support for a `IGenerator2::forAllGenerate` hook in Xtext's incremental builders (`IncrementalBuilder.InternalStatefulIncrementalBuilder` for Xtext's language server; `StandaloneBuilder` for the `xtext-maven-plugin`). I don't see this feature conflicting with incremental builds in a fundamental way. Conceptually, I could see it working like this:
- add a method `forAllGenerate(ResourceSet input, IFileSystemAccess2 fsa, IGeneratorContext context)` to the `IGenerator2` interface.
- in the incremental builders, associate files generated with this method with a synthetic URI `synthetic:/forAllGenerate` representing the whole resource set.
- if any resource is modified, added or deleted, consider the resource set to be modified. I.e., clean all generated files associated to `synthetic:/forAllGenerate` and rerun `forAllGenerate`.
- the resource set is considered "added" once initially. This triggers the first run of `forAllGenerate`.
- the resource set is never considered "deleted".

@ChristianDietrich, do you think Xtext would be open to this kind of contribution?

[Updated on: Fri, 02 June 2023 12:24]

Report message to a moderator

Re: N-to-N code generation [message #1859366 is a reply to message #1859365] Fri, 02 June 2023 13:14 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14722
Registered: July 2009
Senior Member
I don't think. This not a common usecase

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com
Re: N-to-N code generation [message #1859367 is a reply to message #1859366] Fri, 02 June 2023 14:19 Go to previous messageGo to next message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
Hm, strange, I would think for example everyone generating to Python would run into this problem at some point.

But that's fine, I can extend the functionality in our own repository.
Re: N-to-N code generation [message #1859368 is a reply to message #1859367] Fri, 02 June 2023 15:05 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14722
Registered: July 2009
Senior Member
The question. How to solve the incremental delete usecase

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com
Re: N-to-N code generation [message #1859404 is a reply to message #1859368] Mon, 05 June 2023 16:14 Go to previous messageGo to next message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
Well, I think my proposal in the above takes care of that. The idea is to link generated resources to an artificial URI `synthetic:/forAllGenerate`. This URI (representing the whole resource set) is never considered deleted, so incremental delete will never be triggered on these files, which is how it should be.
Re: N-to-N code generation [message #1859407 is a reply to message #1859404] Mon, 05 June 2023 17:08 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14722
Registered: July 2009
Senior Member
But if you delete the last element in a package the package shall be deleted

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com
Re: N-to-N code generation [message #1859428 is a reply to message #1859407] Wed, 07 June 2023 11:31 Go to previous message
Simon Cockx is currently offline Simon CockxFriend
Messages: 69
Registered: October 2021
Member
That's covered by this case:

Quote:
- if any resource is modified, added or deleted, consider the resource set to be modified. I.e., clean all generated files associated to `synthetic:/forAllGenerate` and rerun `forAllGenerate`.

[Updated on: Fri, 09 June 2023 12:47]

Report message to a moderator

Previous Topic:Xtext 2.31.0 is out
Next Topic:Detect index out of sync
Goto Forum:
  


Current Time: Wed Oct 09 23:40:27 GMT 2024

Powered by FUDForum. Page generated in 0.05193 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top