Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Nebula » Big data grid memory usage
Big data grid memory usage [message #1759084] Thu, 06 April 2017 06:55 Go to next message
Jack Borsin is currently offline Jack BorsinFriend
Messages: 3
Registered: April 2017
Junior Member
Hi

I'm working on a simple tool, in it I have a grid that displays a huge amount of rows, sometimes reaching 1 million and more, it's working fine with SWT.VIRTUAL, but I'm seeing a major issue, it's using too much memory sometimes reaching 700MB and more.

The grid is creating all the GridItems(Rows) at once, so once I launch the grid, I get 1M row/GridItem objects created, which seems to be the cause behind the memory issue.

Tried later on to use jTable, but performance is quite sluggish as it needs to create the items once they're visible, and on top of that it doesn't provide some options I want that work with Nebula Grid.


Is there any way to optimize this behavior by Nebula Grid using GridVisibleRange or otherwise? Or should I be using a different thing altogether?

Thanks
Re: Big data grid memory usage [message #1759144 is a reply to message #1759084] Thu, 06 April 2017 21:57 Go to previous messageGo to next message
Wim Jongman is currently offline Wim JongmanFriend
Messages: 432
Registered: July 2009
Senior Member

Hi Jack,

This is an interesting topic.

First, the way all Eclipse virtual tables/trees work is by supplying all possible rows in advance which act as some kind of proxy for the real row that will only replace it when the user actually wants to look at it.

The only benefit that this has is, AFAICT, updating the scroll thumb and actually allowing the user to yank it down to the bottom. Pretty useless for more than a couple of thousand rows if you ask me.

So if you tell the Grid that it is lazy for 1M rows, it will happily create those rows. Which obviously comes with a memory penalty.

You have to use some trickery to make the user believe he can actually see 1M rows event though everyone knows he will never read. One way is to load only load, let's say, a 1000 rows and then if the user scrolled to the last line to add another thousand. We use a strategy of have the last line a link: "Load the next 1000 rows".

Anything useful can only be supplied with filters and "position to".

You can also take a look at the Nebula pagination widget. It displays one screenful of data and the user can then click on the next page.

Hope this helps.

Cheers,

Wim
Re: Big data grid memory usage [message #1759165 is a reply to message #1759144] Fri, 07 April 2017 08:07 Go to previous messageGo to next message
Ian Mayo is currently offline Ian MayoFriend
Messages: 20
Registered: July 2009
Junior Member
Jack,
I also experienced your issue: SWT.VIRTUAL mode didn't offer the performance improvement I was hoping for.

Eclipse Nebula NatTable includes a virtual data model that separates the data model from the visualisation model:
https://eclipse.org/nattable/documentation.php?page=basics

Regards,
Ian
Re: Big data grid memory usage [message #1759261 is a reply to message #1759144] Sun, 09 April 2017 02:14 Go to previous messageGo to next message
Jack Borsin is currently offline Jack BorsinFriend
Messages: 3
Registered: April 2017
Junior Member
Thanks for the reply and the great explanation.

I was thinking about doing something like that, making some sort of buffer of 1K/10K rows, that will get updated when the user scrolls, but I feel this might bring some issues later on as some demands are required, users wanting a specific area of the grid that's being split into 2 halves by the 1K segmentation, or users wanting to jump around, so I thought I would look into other alternatives before I delve into that, me not being much of an expert in SWT doesn't help too Smile

I will also look into the pagination you mentioned.

Wim Jongman wrote on Thu, 06 April 2017 21:57
Hi Jack,

This is an interesting topic.

First, the way all Eclipse virtual tables/trees work is by supplying all possible rows in advance which act as some kind of proxy for the real row that will only replace it when the user actually wants to look at it.

The only benefit that this has is, AFAICT, updating the scroll thumb and actually allowing the user to yank it down to the bottom. Pretty useless for more than a couple of thousand rows if you ask me.

So if you tell the Grid that it is lazy for 1M rows, it will happily create those rows. Which obviously comes with a memory penalty.

You have to use some trickery to make the user believe he can actually see 1M rows event though everyone knows he will never read. One way is to load only load, let's say, a 1000 rows and then if the user scrolled to the last line to add another thousand. We use a strategy of have the last line a link: "Load the next 1000 rows".

Anything useful can only be supplied with filters and "position to".

You can also take a look at the Nebula pagination widget. It displays one screenful of data and the user can then click on the next page.

Hope this helps.

Cheers,

Wim

Re: Big data grid memory usage [message #1759262 is a reply to message #1759165] Sun, 09 April 2017 10:20 Go to previous message
Jack Borsin is currently offline Jack BorsinFriend
Messages: 3
Registered: April 2017
Junior Member
Thanks for the reply.

It seems like this issue is felt by some users, as I noticed while searching about it, I hope later on an update can bring some better optimization for such niche needs.

I looked into NatTable, but I felt it might be too complicated for my noobish self, and I recall trying out the sample program they provided, it didn't feel very smooth when you have many columns visible like I do, and I'm worried whatever work I put into it will be deemed not good enough.

Did you use NatTable to solve this issue or something else?

Thanks again and I appreciate the replies.

ian mayo wrote on Fri, 07 April 2017 08:07
Jack,
I also experienced your issue: SWT.VIRTUAL mode didn't offer the performance improvement I was hoping for.

Eclipse Nebula NatTable includes a virtual data model that separates the data model from the visualisation model:
https://eclipse.org/nattable/documentation.php?page=basics

Regards,
Ian

Previous Topic:TableCombo sends a selection event on each opening of drop down box
Next Topic:Problems running Fortran code
Goto Forum:
  


Current Time: Tue Nov 13 19:49:55 GMT 2018

Powered by FUDForum. Page generated in 0.02040 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top