| Big data grid memory usage [message #1759084] |
Thu, 06 April 2017 06:55  |
 Jack Borsin
Messages: 3
Registered: April 2017 |
Junior Member |
|
|
Hi
I'm working on a simple tool, in it I have a grid that displays a huge amount of rows, sometimes reaching 1 million and more, it's working fine with SWT.VIRTUAL, but I'm seeing a major issue, it's using too much memory sometimes reaching 700MB and more.
The grid is creating all the GridItems(Rows) at once, so once I launch the grid, I get 1M row/GridItem objects created, which seems to be the cause behind the memory issue.
Tried later on to use jTable, but performance is quite sluggish as it needs to create the items once they're visible, and on top of that it doesn't provide some options I want that work with Nebula Grid.
Is there any way to optimize this behavior by Nebula Grid using GridVisibleRange or otherwise? Or should I be using a different thing altogether?
Thanks
|
|
|
| Re: Big data grid memory usage [message #1759144 is a reply to message #1759084] |
Thu, 06 April 2017 21:57  |
|
Hi Jack,
This is an interesting topic.
First, the way all Eclipse virtual tables/trees work is by supplying all possible rows in advance which act as some kind of proxy for the real row that will only replace it when the user actually wants to look at it.
The only benefit that this has is, AFAICT, updating the scroll thumb and actually allowing the user to yank it down to the bottom. Pretty useless for more than a couple of thousand rows if you ask me.
So if you tell the Grid that it is lazy for 1M rows, it will happily create those rows. Which obviously comes with a memory penalty.
You have to use some trickery to make the user believe he can actually see 1M rows event though everyone knows he will never read. One way is to load only load, let's say, a 1000 rows and then if the user scrolled to the last line to add another thousand. We use a strategy of have the last line a link: "Load the next 1000 rows".
Anything useful can only be supplied with filters and "position to".
You can also take a look at the Nebula pagination widget. It displays one screenful of data and the user can then click on the next page.
Hope this helps.
Cheers,
Wim
|
|
|
|
| Re: Big data grid memory usage [message #1759261 is a reply to message #1759144] |
Sun, 09 April 2017 02:14 |
Jack Borsin
Messages: 3
Registered: April 2017 |
Junior Member |
|
|
Thanks for the reply and the great explanation.
I was thinking about doing something like that, making some sort of buffer of 1K/10K rows, that will get updated when the user scrolls, but I feel this might bring some issues later on as some demands are required, users wanting a specific area of the grid that's being split into 2 halves by the 1K segmentation, or users wanting to jump around, so I thought I would look into other alternatives before I delve into that, me not being much of an expert in SWT doesn't help too 
I will also look into the pagination you mentioned.
Wim Jongman wrote on Thu, 06 April 2017 21:57Hi Jack,
This is an interesting topic.
First, the way all Eclipse virtual tables/trees work is by supplying all possible rows in advance which act as some kind of proxy for the real row that will only replace it when the user actually wants to look at it.
The only benefit that this has is, AFAICT, updating the scroll thumb and actually allowing the user to yank it down to the bottom. Pretty useless for more than a couple of thousand rows if you ask me.
So if you tell the Grid that it is lazy for 1M rows, it will happily create those rows. Which obviously comes with a memory penalty.
You have to use some trickery to make the user believe he can actually see 1M rows event though everyone knows he will never read. One way is to load only load, let's say, a 1000 rows and then if the user scrolled to the last line to add another thousand. We use a strategy of have the last line a link: "Load the next 1000 rows".
Anything useful can only be supplied with filters and "position to".
You can also take a look at the Nebula pagination widget. It displays one screenful of data and the user can then click on the next page.
Hope this helps.
Cheers,
Wim
|
|
|
| Re: Big data grid memory usage [message #1759262 is a reply to message #1759165] |
Sun, 09 April 2017 10:20 |
Jack Borsin
Messages: 3
Registered: April 2017 |
Junior Member |
|
|
Thanks for the reply.
It seems like this issue is felt by some users, as I noticed while searching about it, I hope later on an update can bring some better optimization for such niche needs.
I looked into NatTable, but I felt it might be too complicated for my noobish self, and I recall trying out the sample program they provided, it didn't feel very smooth when you have many columns visible like I do, and I'm worried whatever work I put into it will be deemed not good enough.
Did you use NatTable to solve this issue or something else?
Thanks again and I appreciate the replies.
ian mayo wrote on Fri, 07 April 2017 08:07Jack,
I also experienced your issue: SWT.VIRTUAL mode didn't offer the performance improvement I was hoping for.
Eclipse Nebula NatTable includes a virtual data model that separates the data model from the visualisation model:
https://eclipse.org/nattable/documentation.php?page=basics
Regards,
Ian
|
|
|
Powered by
FUDForum. Page generated in 0.03072 seconds
] 
Search
Help
Register
Login
Home
