Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Platform » TableViewer terribly inefficient when loading a lot of data
TableViewer terribly inefficient when loading a lot of data [message #660295] Thu, 17 March 2011 17:25 Go to next message
David  Pérez is currently offline David Pérez
Messages: 228
Registered: July 2009
Senior Member
Hi,

I'm working a lot with TableViewer's in virtual mode (SWT.VIRTUAL style for the Table), and fetching only the visible items from the database (or other data sources).

It can potentially contain thoudsands of rows.

I have optimized so that data is fetched on the fly.

I think that none works in this way, because I've faced with a lot of bugs and inefficiencies, both in JFace and SWT. Am I right?

As an example of a bug, is that the AbstractTableViewer maintains in memory all the elements, in inner class VirtualManager.

On the other hand, Table class (at least under Win32), maintains in memory all the rendered TableItem's. This worses the memory problem.

I have solved all this inefficiencies.
I can contribute them, if desired, as a patch.

Now, I have discovered another one, that I have to solve right now:

AbstractTableViewer#setItemCount() forces to create all TableItem's.

In my test case, itemCount is 1,000,000 items and it takes many minutes to do so.

class AbstractTableViewer {
.....

public void setItemCount(int count) {
int oldCount = doGetItemCount();
if (count < oldCount) {
for (int i = count; i < oldCount; i++) {
Item item = doGetItem(i);
if (item.getData() != null) {
disassociate(item);
}
}
}
....
}
}

Does e4 improve a little this?
Re: TableViewer terribly inefficient when loading a lot of data [message #660303 is a reply to message #660295] Thu, 17 March 2011 17:47 Go to previous messageGo to next message
Thomas Schindl is currently offline Thomas Schindl
Messages: 5327
Registered: July 2009
Senior Member
[...]
>
> Does e4 improve a little this?

No - e4 currently only address problems above the SWT/JFace level (=the
workbench)

My only suggestion is to rethink your UI concept.

Tom
Re: TableViewer terribly inefficient when loading a lot of data [message #660374 is a reply to message #660303] Fri, 18 March 2011 07:57 Go to previous messageGo to next message
David  Pérez is currently offline David Pérez
Messages: 228
Registered: July 2009
Senior Member
Thanks Tom.

I just wanted to emphasize the fact that the VIRTUAL tables don't work well, with large quantities of data, and can supply some patches.

I think I don't need to rethink my UI concept, just a few patches and it works quite well. The biggest effort is already done.

Tom Schindl wrote on Thu, 17 March 2011 13:47
[...]
>
> Does e4 improve a little this?

No - e4 currently only address problems above the SWT/JFace level (=the
workbench)

My only suggestion is to rethink your UI concept.

Tom

Re: TableViewer terribly inefficient when loading a lot of data [message #660380 is a reply to message #660374] Fri, 18 March 2011 08:19 Go to previous messageGo to next message
Thomas Schindl is currently offline Thomas Schindl
Messages: 5327
Registered: July 2009
Senior Member
David, we are always happy to get patches. You are completey correct
about your findings on SWT.VIRTUAL and Viewers.

This is an area not heavily used (most likely because the Eclipse IDE
has no real need for it I guess) and needs some TLC.

Since I left the JFace-Viewer area to work on other more important
things for our on RCP-Applications (=e4) there's no one really pushing
JFace-Viewers forward in new directions.

Tom

Am 18.03.11 08:57, schrieb David Pérez:
> Thanks Tom.
>
> I just wanted to emphasize the fact that the VIRTUAL tables don't work
> well, with large quantities of data, and can supply some patches.
>
> I think I don't need to rethink my UI concept, just a few patches and it
> works quite well. The biggest effort is already done.
>
> Tom Schindl wrote on Thu, 17 March 2011 13:47
>> [...]
>> > > Does e4 improve a little this?
>>
>> No - e4 currently only address problems above the SWT/JFace level (=the
>> workbench)
>>
>> My only suggestion is to rethink your UI concept.
>>
>> Tom
>
>
Re: TableViewer terribly inefficient when loading a lot of data [message #660393 is a reply to message #660374] Fri, 18 March 2011 09:47 Go to previous messageGo to next message
David  Pérez is currently offline David Pérez
Messages: 228
Registered: July 2009
Senior Member
Good news Smile , with my latest patch, TableViewer can support millions of rows.

I have changed setItemCount() so that it doesn't try to create non-existant TableItems. It used to take over 10 minutes, and now just some milliseconds and the memory consumption is also good.

David Pérez wrote on Fri, 18 March 2011 03:57
Thanks Tom.

I just wanted to emphasize the fact that the VIRTUAL tables don't work well, with large quantities of data, and can supply some patches.

I think I don't need to rethink my UI concept, just a few patches and it works quite well. The biggest effort is already done.

Tom Schindl wrote on Thu, 17 March 2011 13:47
[...]
>
> Does e4 improve a little this?

No - e4 currently only address problems above the SWT/JFace level (=the
workbench)

My only suggestion is to rethink your UI concept.

Tom



Re: TableViewer terribly inefficient when loading a lot of data [message #660815 is a reply to message #660380] Mon, 21 March 2011 17:58 Go to previous messageGo to next message
David  Pérez is currently offline David Pérez
Messages: 228
Registered: July 2009
Senior Member
OK, Tom.

When I have time, I'll create a bug in Bugzilla and attach the patches.

I only work with Win32, and cannot test all the platforms, but my changes should be platform independent.

David

Tom Schindl wrote on Fri, 18 March 2011 04:19
David, we are always happy to get patches. You are completey correct
about your findings on SWT.VIRTUAL and Viewers.

This is an area not heavily used (most likely because the Eclipse IDE
has no real need for it I guess) and needs some TLC.

Since I left the JFace-Viewer area to work on other more important
things for our on RCP-Applications (=e4) there's no one really pushing
JFace-Viewers forward in new directions.

Tom


Re: TableViewer terribly inefficient when loading a lot of data [message #758659 is a reply to message #660393] Thu, 24 November 2011 07:01 Go to previous messageGo to next message
John Missing name is currently offline John Missing name
Messages: 2
Registered: November 2011
Location: San Diego
Junior Member
Hi David,

I am interested your patch which solved setitemcount with tableviewer when huge amount of rows are involved. Can I share with your patch which you posted back in 2009?


Thanks and happy thanksgiving!


John
Re: TableViewer terribly inefficient when loading a lot of data [message #840136 is a reply to message #660393] Mon, 09 April 2012 18:05 Go to previous messageGo to next message
John Missing name is currently offline John Missing name
Messages: 2
Registered: November 2011
Location: San Diego
Junior Member
Hi David,

Would you please provide the link to get your patch regarding tableviewer inefficiency (setiemcount) with huge amount of data?


Thanks,

John
Re: TableViewer terribly inefficient when loading a lot of data [message #871175 is a reply to message #840136] Fri, 11 May 2012 09:36 Go to previous messageGo to next message
gaurav dubey is currently offline gaurav dubey
Messages: 37
Registered: July 2009
Member
Hi David,

Can you please provide me the link of the patch which will allow SWT table to store large amount of rows, I tried with virtual table, but no success.

Thanks in advance.
Re: TableViewer terribly inefficient when loading a lot of data [message #991115 is a reply to message #660393] Mon, 17 December 2012 12:02 Go to previous message
Andrew Nesheret is currently offline Andrew Nesheret
Messages: 29
Registered: July 2009
Junior Member
David érez wrote on Fri, 18 March 2011 05:47
Good news Smile , with my latest patch, TableViewer can support millions of rows.

I have changed setItemCount() so that it doesn't try to create non-existant TableItems. It used to take over 10 minutes, and now just some milliseconds and the memory consumption is also good.



Hi David, report submitted https://bugs.eclipse.org/bugs/show_bug.cgi?id=396736
You are welcome to provide patches.
Previous Topic:How to install ICompletionListener (to all TextEditors)
Next Topic:eclipse debug hot code replacement fails
Goto Forum:
  


Current Time: Fri Sep 19 05:51:45 GMT 2014

Powered by FUDForum. Page generated in 0.03164 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software