Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Future of Managed Build
On Wed, 22 Jan 2020 at 20:15, Jan Baeyens <jan@xxxxxxxxxx> wrote:

Hi

I was hoping for more responses. I guess I'm the only Jantje on the maillist ;-)

See my comments below

Jantje and not Jan please :-D


Sorry Jante.
 

Op 20/01/2020 om 19:52 schreef Jonah Graham:
(I changed the subject because Jan brings up lots of important stuff that extends far beyond LLVM/clang integration.)

Jan,

Your observations are all very accurate. In particular the lack of sensible/easy instantiation of the build model. This is an area that many people have spoken about recently, now the challenge is how to move forward. I think in the Eclipse Ecosystem such a build model should be represented in EMF models. Alexander Federov and William Riley have both been making some discussions in that regard.
I agree on the fact that the toolchain description should be a model. That is how I visualize it myself
The components on the model I currently see are files, filters,grouping, tools, states.
However: The current extension points solution has the feature of options in the project properties. I can't recall how it works exactly but this is something I do not want to loose.
One could consider describing the options at the tool level as an extension point referenced by the model. However this would mean that a extension point needs to be instantiated for each tool, which is probably not wanted. Also, in Arduino world the build/compile/archive/link options are locked and decided on by the build options provided by the framework which are completely different from the gnu compile options. So I don't think having a tool extension point will work for that.

I have no experience with emf modelling so I don't know it is capable enough. I also have no experience getting this in java classes. Help will be needed if we go this way. Learning EMF is now on my todo :-)
Is there some doc from the discussion from Alexander Federov and William Riley?

@William - can you share your current state?

@Alexander - do you have some guidance based on your past experiences that you can share?
 

The good makefiles is an interesting point - do other tools such and Meson and CMake make "good makefiles"?
Does CMake make makefiles? I have no clue. I'm wondering myself. That is why I request input. Personally I think it is a waste of resource because I think people who can write makefiles will want to write them themselves. People who can't write makefiles fluently just want to build. I think they don't even care whether it is make, cmake, internal builder.
And as to deploy (as was mentioned in one of the links you shared) ... I don't think we are up to that yet.
At one point Eclipse CDT folk implemented a new builder that (essentially) reimplements Make called the Internal Builder.
From where CDT is/was, that seems pretty logic to me. I wanted to use that with Sloeber so I only had to provide the tools of the toolchain and didn't need to provide a make.exe. However I never got it to work. One of the biggest pro's is that you would use the same make on all oses.
What we need is a way forward - and I don't think anyone is excited about evolving as opposed to replacing the existing Managed Make / GNU Makefile Generator code. The current code is very brittle, used by lots and lots of extenders so any change, even small requires a very large amount of work to decide if it is safe. However perhaps I am wrong about this and I should be convinced otherwise.
As I already said: I like the extension point stuff. But (and this is new) there is quite a lot of makefile contamination in there.
And I do not like you can't use outputs from tool A as input for tool B.

This is one of the challenges - the current code has a general source inputs to one output executable. Anything else is bolted on and was not really in scope of the original implementation IIUC.
 
And I only got sloeber to work due to the makefile contamination :-s
After having looked at the problem domain and the current implementations seriously -with evolving on my mind- I think evolving is not feasible
So I'm currently looking for the good and bad things of the managed make and I hope we can leave the bad things behind (but not what we learned) and implement the good things in something new.


 
To answer your questions:
1)Inside the broad CDT world. Does anyone know of other build models like described in the Managed Build System Extensibility Document in the eclipse help->CDT-plugin developer guide->programmers guide->CDT DOM?

(for clarity for others, the MBS Extensibilty Document is not part of the CDT DOM - the link in online help is https://help.eclipse.org/topic/org.eclipse.cdt.doc.isv/guide/mbs/extensibilityGuide/Managed_Build_Extensibility.html?cp=14_0_1)

I don't know of other build models - in the broad CDT world AFAIK almost everyone either uses MBS or has custom Makefiles or other tools to manage build settings.
I thought autotools was something like that and wasn't there a autotools component in CDT?

There is an autotools component in CDT. It uses MBS too as does the non-Managed Makefile support. Of course both of them don't use Eclipse's GNUMakefileGenerator


2)Did you ever build a build model based on "Managed Build System Extensibility Document" or others? Can you share your experiences/thoughts?

Yes, the build model is what you are using in Sleober :-)
Sorry for the confusion. I'm looking for input on what a managed build system should look like for people willing to "design toolchains" or input for external build tools. This is also why I asked Martin in the Cmake support thread. Basically I'm looking for user requirements.

3)Suppose I build a buildmodel instantiator (and a makefile generator for testing) does anyone think someone will use it?

This is an area that I cannot answer. I think having a good instantiation of the build model that is easier to use than what we have today would be nice. But as I outlined above, I think that the development effort may best be spent on a step change to new technologies.
Thinking about this again I don't think there will be many users. If we build a good internal builder on top of this....there will be even less.
This makes me doubt about EMF being a good way to go. Seems like overkill when taking the little usage I see into account.
Who sees usage opportunities?

4)Could this buildmodel instantiator become part of CDT?

Yes it absolutely can if it is a testable unit of code. If it is useful to you it will probably be useful to others.

Cool.

off-course it must be testable. Sloeber runs several thousands of functional test before releasing. I'm all for testing
My current thinking is that the main module should take a ( targetPath, project files,project,buildModel) and produce a instantiated build model that should be serializable which should be consistent in output. I guess you guys know how to setup a project for testing, I don't.

I have lots of second thoughts about the targetpath, project files and the project. It is just very first thinking and I want to list the important things (I am not talking api right now). From my experiences with gnuMakeFile I want the project in all api calls. I needed to overload and cast the name providers to add the project to be able to read environment variables.
So I know that technically the targetPath and the project files can and should be taken from the project, what I try to express is that even if you have those you'll still need the project to call the name providers and to get build options.

Thanks Jan for re-starting this discussion here. There have been lots of discussions on these issues in the past, on the mailing list and some written down in the Wiki*. What is needed is someone to lead this effort and a real commitment of development resources to make it happen.

If you have resources, feel free to assign them :-) until then I fear you will have to deal with me rofl :-)

I think I'll start with the EMF study and design following toolchains;

Library (debug/release)
executable (debug /release)
Arduino
Please add as crazy as possible ideas

That will be a first go/no go reflection moment

Then I'll try to implement a basic instantiated build model maker and a makefile maker.
This will be a second go/no go reflection moment.

As I'm not a java developer and I'm not a cdt contributor and I'm doing this to get it into CDT I would like someone who knows the eclipse/CDT way to have a look from time to time (candidates please speak up).
I don't yet know how I will handle version control and sharing though (ideas please).


* I have been reviewing all the Wiki pages. Some pages that I have looked at on this topic are:

These are all on this same broad discussion and nowhere is a full resolution.

Jonah



~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Sat, 18 Jan 2020 at 08:51, Jan Baeyens <jan@xxxxxxxxxx> wrote:

Hi

I'm happy to there is lots of attention for the managed build capability of CDT right now :-).
I have been playing with the gnu makefile generator for some years now and I have been impressed with the capabilities provided by the extension points (called build model below).
I realize I only understand a very small portion of the problem domain. For instance I don't know cMake, LLVM, LLVM-GCC, LLVM + JIT... but I also feel I can contribute.
What I do know is this; Whatever tool you use to build, you will always have to combine source code files with commands (executable and parameters).
The way to combine (and the commands) can be very easy or very complex.

When I look at the gnu makefile generator I think the build model is a very good basis to describe "how to combine source code files with commands".
If I look at the implementation of the gnu makefile generator I see 3 problematic design decisions:

1)No instantiation of the build model.
2)Trying to make "good makefiles"
3)Working with strings

I would like to sell the lack of "instantiation of the build model"  as a current weakness in CDT and the addition of a "instantiation of the build model" as the solution.
And please see my questions below.

1)No instantiation of the build model.

Suppose I have the super simple build model
convert each *.c to *.o with the command g++ -o {output} {input}
convert all *.o to {project}.exe with the command link  {inputs} {output}

With a project myExample, containing the files input.c and test.c; 

A instantiation of the build model would look like

Input files (input.c);output files (input.o);command (g++ -o input.o input.c)
Input files (test.c);output files (test.o);command (g++ -o test.o test.c)
Input files (input.o; test.o);output files (myExample.exe);command (link test.o input.o myExample.exe)

My thinking is that with this instantiation of the build model one can easily

1) make a makefile
2) run the commands
3) make a cmake file
....

Even though gnuMakefileGenrator does all this work; it doesn't instantiate the build model in any way, the code goes from build model directly to makefile in pieces.
If we could agree on a data model for the instantiation of the build model we could avoid lots of duplication and have a great addition to CDT.

2)Trying to make "good makefiles"

The second problem I see with gnuMakefileGenerator is that it tries to generate good makefiles. Hints to this are:
1)The possibility to not expand environment variables in the makefile
2)When using nameproviders the makefile becomes completely different

Though it is noble to make readable/maintainable makefiles, it also requires lots of extra resources (like testing) and a high level of knowledge about makefiles for future maintainers.
Using only one "type" of makefiles one can easily reduce:
1)amount of code
2)amount of tests
3)amount of training

3)Working with strings

The code is all based around strings and doesn't use classes for files and so on. I don't think this is due to lack of skill but a simple consequence of the design decisions I already mentioned.
This however adds to the maintenance nightmare.

My questions

My fingers want to tackle this problem but to avoid needless work I feel it is important for me to have a better understanding of how this fits into the cdt community. As such I have following questions.
1)Inside the broad CDT world. Does anyone know of other build models like described in the Managed Build System Extensibility Document in the eclipse help->CDT-plugin developer guide->programmers guide->CDT DOM?
2)Did you ever build a build model based on "Managed Build System Extensibility Document" or others? Can you share your experiences/thoughts?
3)Suppose I build a buildmodel instantiator (and a makefile generator for testing) does anyone think someone will use it?
4)Could this buildmodel instantiator become part of CDT?

Best regards

Jantje

Op 18/01/2020 om 4:50 schreef Marc-Andre Laperle:
Hi,

From what I remember, the intention of the preference page was to be able to add the system headers which a prebuilt clang did not know about at all and you had to specify manually. This preference page would be global to all projects and configurations. But this is not the way normally managed build projects are configured. In fact, it did this global setting by adding the include paths to every projects and every configuration, before every build and regardless of the toolchain! Also, post-build it would always do a full workspace refresh which I had to comment out because people were complaining quickly that Eclipse was unusable while this plugin was installed.

In the midst of the enthusiasm of having LLVM support, I don’t think this plugin was fully tested or thoroughly reviewed because there were/are glaring issues. I personally ended up using the GCC toolchain and just replacing the command with clang and clang++ which worked fine (especially once many OS/distros had a prebuilt clang fully configured with system paths, etc) so the motivation of fixing the LLVM toolchain was/is low. Still, I think it would be a nice project to go back and clean up the LLVM plugin (…or remove it). A first obvious clean-up would be to remove the preference page and any concept of global include path and libs. Another step would be to remove the many tools and toolchain that I don’t believe people use or are still maintained (LLVM-GCC, LLVM + JIT, etc). Then there is general code clean-up (there shouldn’t be obvious comment for every line).

Cheers,
Marc-André

On Jan 17, 2020, at 12:20 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Joost,

Thank you for your analysis up until this point.

In my new role as CDT project lead there are a number of areas that I am simply not familiar with (yet!) and this is one of them. 

I can try to provide some background, and I will be very happy to review any patches you provide!

To answer some of your questions:

The LLVM plug-ins were added in Bug 338553 and haven't changed much since then in the particular area of the code you have highlighted. 

From a quick look at this it seems there is a confusion in this code between settings passed to the LLVM compiler and those passed to the CDT indexer and which tool is "in charge". In simple cases that difference probably doesn't matter, but it clearly quickly goes wrong (the case in Bug 527757 is particularly obviously bad).


> - Is there any (high level) design documentation available for the LLVM managedbuilder stuff? Or for GCC?

Have a look at https://wiki.eclipse.org/CDT/designs/MBS it contains design documentation for when managed build was first added and has a number of links that may be useful. 

> - Why is there a CDT LLVM preference page? What is/was it's purpose?

I don't know. There may be answers in the Petri's thesis, but the link to it is dead (see comment).

> There is no such page for GCC?

Because it probably shouldn't exist for LLVM either?

> - I cannot find UI for LLVM/Clang project properties. Can someone
enlighten me?
> - The same for the GCC project properties?

It is in the Project Properties -> C/C++ Build -> Settings. (see attached file to confirm we are talking about the same thing)

Thank you for taking the time to look at this. There is certainly a lot of little bits needed to get a C++ Managed Build project to "just work" and I look forward to supporting you where I can.

Thanks
Jonah




~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 17 Jan 2020 at 09:11, Joost Kraaijeveld <J.Kraaijeveld@xxxxxxxxxx> wrote:
Hi,

My immediate personal problem is solved by placing a comment on line 62
of LlvmResourceListener.java to prevent
LlvmPreferenceStore.addStdLibUnix() being called.


I suspect that the same should be done on line 59 but I have no Windows
with MinGW to test it with.

--
Groeten,

Joost Kraaijeveld
Hommelseweg 123
6821 LD Arnhem
06-51855277

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev
<Screenshot_2020-01-17_12-16-40.png>_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cdt-dev

Back to the top