| Re: [cdt-dev] Handling huge C/C++ project in eclipse |
I agree with the issue with listeners. But that's more an issue with miss behaving plug-ins that are registered to too wide an event queue, or a problem with how Eclipse manages events to see if we can do it better. We need to treat Eclipse exactly how it is, a really big event driven system.
I use an uber IDE that includes the Eclipse SDK, BlackBerry and QNX tools, CDT, Web tools for _javascript_, Android. I do get odd hesitations at times, especially after saving a file. I really need to find out what's causing that, mind you it's probably garbage collection. Which brings up the memory issues mentioned earlier.
Doug.
From: kesselhaus <kesselhaus@xxxxxxx>
Date: Mon Jul 21 2014 19:47:56 GMT-0400 (EDT)
To: cdt-dev@xxxxxxxxxxx <cdt-dev@xxxxxxxxxxx>
Subject: Re: [cdt-dev] Handling huge C/C++ project in eclipse
I guess that, like in automotive projects nowadays, you have so many different files/types within one project:
a) C code, which might be handwritten, or generated by tools
b) Model files (may it be Rhapsody or EA or ... or even some Eclipse based Modeling Tool
c) Configuration files for AUTOSAR MCAL, which are configured an Eclipse based tool (currently extern)
d) Configuration files for AUTOSAR BSW modules, which are configured by an Eclipse based tool (also currently extern)
e) Configuration files for ComStack (AUTOSAR or CANbedded) configured by external Tool
f) Scripts for conversion or other automation stuff around the build environment (python, perl ..)
g) XML based files generated by one tool and pre/post processed by others
So, we are talking here about the following tool/plugins:
CDT, Artop, EATOP, EMF & Modelling Tools, Xtext/Xpand/Xtend + Workflow tools, XML editor, Pydev, EPIC ...
It's not like in old days, where you just hack on your plain C files and maybe some makefiles to get something done, we have a whole bunch of other stuff now included.
That's why I also find it kind of strange, that Eclipse foundation does take everything what it can get as contribution and is happy about that many projects, but also does some steering about the directions. Just take a look on the many modelling and generation plugins available, and what other plugins depend on what.
Now, the ARTOP and EATOP go Xtext/Xpand, but Polarsys (Papyrus UML etc.) going Acceleo.
And IDE means now what? .. Integrated Development Environment ...
We had already a lot of trouble to get the c), d) and e) Tools somehow cooperate, to generate the files, where the actual sandbox with eclipse+CDT is working.
And, the question is also, do you have one Eclipse instance or how many?
Let's count the Java based tools, that get my machine down @work:
Lotus Notes
MKS Integrity
Eclipse + CDT
MCAL Configuration & Generation Tool
BSW Configuration & Generation Tool
Cantata .. well, yeah, nice to have the Unit & Code Coverage tests in a separate tool
So, taking Lotus Notes aside, all the other tools are needed for just one project.
Not to mention, that I have now at least 10..20 projects open in one workspace, since
1) I need several different branches of a single project
2) I have several projects open in parallel (yeah, sometimes I have to look into other projects and their branches, or are working in multiple projects at the same time)
3) Lots of times I have more than 99 files open ( C Files, XML Files, scripts, makefiles etc.) because of 1) and 2) just before
Sorry about the rant, but I'm already getting lost on my desktop ... I just wanted an I.D.E.
Am 21.07.2014 20:12, schrieb Vishal Gupta:
Hi,
If you ask me, then yes i exactly do what you have told----------------------------------------------------------------------------------------------------------------------------------------------------Have several separate Eclipse installations, which have only the plugins needed for the specific usage?
e.g. one for C/C++, one for maybe Modeling tasks, etc. pp.--------------------------------------------------------------------------------------------------------------------
This is the one way how i stop eclipse from becoming non-responsive .In my personal opinion as the number of plugins increases in a particular eclipse installation, the more number of listeners gets added. And whenever a change happens in the workspace all these listeners gets activated. Which actually makes eclipse non-responsive.
Vishal
On Mon, Jul 21, 2014 at 4:00 AM, Doug Schaefer <dschaefer@xxxxxxx> wrote:
Also, the only out of memory condition I've run into has been the CDT. The Boost source always causes that as it's indexing. I need to take a week and debug it. Andrew E did a lot of the leg work there but his contract ended.
And, yes -Xmx1024 seems to be the only way to get by these days without Java constantly garbage collecting, but I refuse to go more than that so I can see where problems occur.
Doug.
From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Doug Schaefer [dschaefer@xxxxxxx]
Sent: Sunday, July 20, 2014 6:19 PM
To: kesselhaus; cdt-dev@xxxxxxxxxxx
Subject: Re: [cdt-dev] Handling huge C/C++ project in eclipse
It's easy to blame Eclipse 4. But I'd like to see some real memory analysis that shows why it's consuming as much as it does. Then we can act on fixing it.
Sent from my BlackBerry 10 smartphone on the Rogers network.
Hi,
I guess, you are not the only one.
Since Eclipse 4.x hit the ground, Eclipse looks like getting bloated, lacking and consuming much more RAM.
Not everyone has the luck, that he gets that fast machines with lots of RAM.
My working machine e.g. "only" has 8GB of RAM.
But Eclipse isn't the only tool I use, and some other programs also consume 500MB RAM and more.
And more and more programs here are also based on Java, may that be Lotus Notes, MKS Integrity Client, AUTOSAR Configuration tools ...
So, I wonder, what the actual usage of Eclipse should be.
Have several separate Eclipse installations, which have only the plugins needed for the specific usage?
e.g. one for C/C++, one for maybe Modeling tasks, etc. pp.
Or does it depend on the workspace and it's contained projects?
But switching all the time between the workspaces isn't that nice either.
I'm not a Java developer as you might already know, so I also have no clue, what these VM settings actually have what effect.
I only found out, that one should set the Xmx to something like 1024M on Windows. But I still get sometimes an Out-of-Memory, or Heapspace error, and sometimes, Eclipse just needs several seconds before it is actually reacting on actions I do, like calling the context menu on a project or such simple things.
So, what should one do?
I guess, this is not just a problem of CDT itself, but the whole Eclipse zoo of plugins.
What are the intentions also of the Eclipse foundation? Are there any activities now to get the 4.x stream to perform like the old 3.x stream?
Am 26.06.2014 00:29, schrieb Doug Schaefer:
I've been worried that we've increased memory requirements in the last couple of releases. Love to understand why that is once I get it other things out of the way. Boost is the best example.
Sent from my BlackBerry 10 smartphone on the Rogers network.
I do have a really fast computer, though.not. Perhaps there are better ones. I mostly just guessed.This seems pretty big, but it does complete. I don't know if these numbers are sensible orthe indexing seemed to complete.You might think about changing your eclipse.ini file.I found I ran out of memory with Luna, but when I set these values in my eclipse.ini file,
I have a much larger project than that. I have about 6000 files, with about 84 Mbytes,
and 2.5M lines of code, including comments. The indexer runs on this in about an hour.
-XX:MaxPermSize=256m
-Xms1G
-Xmx4G
On Wed, Jun 25, 2014 at 2:05 PM, Vishal Gupta <vishal.vit@xxxxxxxxx> wrote:
Thanks Ronald and Doug .
After reading your comments i am feeling like I am stuck again with my huge code base on which i wan't to use Eclipse in it's original form.
Bringing my huge code base on my local machine or on my remote server is not the problem. The problem arises when the indexing start, even if i assign 4GB of RAM for eclipse still it throws out of memory while indexing (both on my windows and Linux box). I did try all possible permutation and combinations of allocating memory to eclipse, but no success.
So i thought of doing some weird changes in eclipse.
Main component in eclipse which i want to use is CDT, so my scope was first limited to that.
The two things which i identified is the bottle neck in eclipse for huge code base are1. Eclipse recursive refresh. Which itself takes long time for a huge code base. And once it completes, it notifies all the resource change listeners. This is when the hell break lose with CDT.2. CDT indexer which is the most memory intensive, that too when it tries to index the huge code base in one go, it throws out of memory most of the time. And till the time it's running, the eclipse is not usable at all. Even if it does a complete indexing there is no guarantee that the Eclipse will not throw out of memory (My personal feeling is that CDT maintains a lot of information in memory).
So i put my hack in eclipse code.
1. I stopped the INFINITE_DEPTH refresh in eclipse.2. Created a custom recursive refresh call "On demand" (via command). And allowed the Depth One refresh call on Tree expansion from Explorer.3. Stopped the recursive refresh calls from CDT components.4. Plugged in a pre-build index file which i created by running cdt indexer in headless mode on my server via CDT extension point. The pre-build index file size is 4GB.5. To make call hierarchy and open deceleration usable, i just patched a refresh call for the Bindings found from pre-build index file.
Cool now i can create a C/C++ project without worrying about the code base size. Project gets created in no time. Eclipse don't give out of memory since it doesn't deal with complete source tree. i can refresh only those resources recursively which i want to look into or want to modify in my code base. And still i can utilize most of the functionality of CDT require during coding.
This is quite an ugly approach. But works for me since the Refresh and Indexing happens on demand.The major drawback is that i have modified the Eclipse Refresh mechanism.
So i was looking into Remote project, but again it has its own problem as mentioned in the mail thread below. Synchronize Project will again bring back the complete source tree indexing (in one go) back to my local machine and for sure Eclipse will give me out of memory.
I have changed the subject of the mail and have put cdt-dev mailing list also in the "To" list if anyone can suggest me something cleaner.
Thanks,Vishal
On Wed, Jun 25, 2014 at 2:20 AM, Doug Schaefer <dschaefer@xxxxxxx> wrote:
This has been a very interesting discussion. It certainly reminds us why the remote project support was added in the first place. I personally think that it's such a hard problem, there probably isn't a good solution for hooking up an IDE to such an extremely large code base located on a remote machine, as much as we want there to be.
But one take away I have that might make synchronized projects work out is that you're desktop probably has as much power as the server you're running the indexer on, especially since the indexer is single threaded. It'll probably take the same amount of time to index on either and then you're left with the remote tooling doing expensive look ups on the server and the latency in passing results back to the workstation. I'd be excited to hear if there was much success using it that way and kill my doubt.
Also, Gigs of source isn't such a big thing any more when we have terabyte drives in our laptops. It may be an expensive first time configuration to get all that copied over to the workstation, but if the synchronizer has done it's job well, things should be very fast after that.
Anyway, just another opinion for the pile. You get to choose your path, but I'm afraid the remote project one isn't well travelled and I'm not as confident on where it leads.
Doug.
From: ptp-dev-bounces@xxxxxxxxxxx [ptp-dev-bounces@xxxxxxxxxxx] on behalf of Roland Schulz [roland@xxxxxxx]
Sent: Tuesday, June 24, 2014 4:41 PM
To: Parallel Tools Platform general developers
Subject: Re: [ptp-dev] Remote Project
I recommend to use synchronized project and tune the local indexer. See e.g. http://stackoverflow.com/questions/9565125/whats-the-recommended-eclipse-cdt-configuration-for-big-c-project-indexer-ta for tips of how to do that.
Roland
On Tue, Jun 24, 2014 at 11:54 AM, Vishal Gupta <vishal.vit@xxxxxxxxx> wrote:
Ideally telling Yes i want to get the Indexing done on the server.And i am hopefull that it will ease the load on the local machine.
Yes i want to get everything index, but currently even my smaller project is not getting indexed
On Tue, Jun 24, 2014 at 9:11 PM, Roland Schulz <roland@xxxxxxx> wrote:
Hi,
do you want to use the index? Do you need everything indexed?
Roland
On Tue, Jun 24, 2014 at 8:22 AM, Vishal Gupta <vishal.vit@xxxxxxxxx> wrote:
Hello Greg,
Thanks for the feedback.
Let me give you a background why i thought of giving a try to the remote project.
1. I have a very huge code base ( in GBs)2. Using Eclipse to import the complete code base on the local machine is crazy. Eclipse can't withstand such a huge code base and the indexer makes things non-usable.3. So the solution was to checkout the code base on server and I tried to use the Remote Project to access it. The benefit i thought was that the indexing (which is the most memory intensive operation will happen on the server).
Now as you are suggesting to use the Synchronized project, my biggest nightmare "indexing" will again be happening on the local machine. Which will again make eclipse non-usable for my code base. Another thing is to sync the code base initially, my personal feeling is that it will take ages to do initial syncing between geographic different location.
I am very much open to contribute code if you people can guide me to a solution.
Thanks,Vishal
On Tue, Jun 24, 2014 at 5:16 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Hi Vishal,
Many of the problems you describe are also apparent even when accessing a remote project in the same geographic location, or even the same local network, and tend to be inherent in the nature of remote projects. It may be possible to speed up directory listings as you suggest, but this will not do much to speed up the overall performance of remote projects.
As a result of these issues, we have transitioned to another type of project called synchronized projects. These projects maintain both a local and remote copy of the source files. Apart from the time it takes to do the initial synchronization, these projects tend to work much faster with Eclipse, since the indexer and other expensive operations only happen on the local copy of the files.
Synchronized projects are available in both the Kepler and soon-to-be-released Luna versions of Eclipse for Parallel Application Developers package.
Ongoing development of remote projects has pretty much ceased, so it is unlikely there will be any future updates unless you would like to contribute something.
Regards,
Greg
On Jun 24, 2014, at 6:59 AM, Vishal Gupta wrote:
> Hello ,
>
> I have just started using Remote Project.
> I am accessing my project in a geographically different location. (Remote machine the server is in USA and local machine is in India)
> My initial impressing is that it's very slow to access even a small
> project with 1000 files (around few MB). I am using the "remote tool" configuration
> to access my project on remote machine.
> I tried looking into it and my initial finding is :
>
> Project Creation itself take a long time as the .project file and .setting folder is created and updated on the remote machine. Could it be possible to create the .project on the local machine ?
> I think the same happens with RSE connections where a hidden project is created on the local system for the connections and the configuration files are stored in it.There may be better solution for sure.
>
> For me the refresh of the files and folder is also very slow, my initial feeling is that the "ls" operation via sftp connection is taking more time. Any possibility to speed it up?
>
> The Indexing for my CProject is not working. The PDOM files is getting created on the remote system but the indexing is not happening. Do i need to do anything extra ?
> I checked the status of the Remote Indexing operation and it only had info and no error.
> I am still digging into the problem. Just wanted to check if i have missed anything ?
>
>
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev