Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Nebula » [CompositeTable] a question of size
[CompositeTable] a question of size [message #50986] Mon, 24 March 2008 21:07 Go to next message
David Kyle is currently offline David KyleFriend
Messages: 125
Registered: July 2009
Senior Member
Hi,

Is it realistic to use a CompositeTable for an extremely large table? A bit
of playing around with CompositeTableSnippet4 shows that it can handle
millions of rows. Hurrah! But it cannot handle 20,000 columns. Even a small
500 column table caused out of resource issues.

I noticed that
http://www.eclipse.org/nebula/widgets/compositetable/composi tetable.php
mentions

"4. Requests only visible data using a virtual table API for maximum
scalability and performance."

Has any progress been made on this point? My only other option is to use
Albireo and implement a Swing JTable.

Thanks,
David Kyle
http://richclientplatform.blogspot.com
Re: [CompositeTable] a question of size [message #51042 is a reply to message #50986] Tue, 25 March 2008 20:09 Go to previous messageGo to next message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
Hi David,

This is a really good question, and hits one of the last remaining
CompositeTable limitations head-on.

I'd like to ask the community for feedback on this, because we're
starting to work on addressing this right now.

First: What kind of scalability do you realistically need? I've heard
of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
that's ~280 visible columns. Does anyone need more scalability than
that?

Here are some approaches for addressing this limitation:

1) Tweak the per-process resource limit in the Windows registry. This
obviously just moves the threshold, but it might be enough. However,
some IT teams might balk at widespread registry editing. Does anyone
have experience with this solution?

2) Make CompositeTable (optionally) virtual in the horizontal as well as
the vertical direction. This would require API on the row objects to
create/dispose cell objects as well as much more complicated horizontal
scrolling logic.

3) Support Draw2d widgets in row objects. Since Draw2d basically
reimplements Swing inside an SWT Canvas, you don't wind up with the huge
resource requirements, you get to use the Draw2d hit testing code, and
the (small) library of Draw2d widgets. At least Draw2d has a Text
implementation, probably the most important widget. Thoughts?

4) Create an abstract Row implementation that uses the KTable approach.
ie: Draw cells using renderers and edit using CellEditors (kind of like
JFace). This requires reimplementing a few UI toolkit abstractions like
hit testing, and makes it hard to implement "always-on" editing. But
it's doable.

Personally, I'm leaning towards choice #3, but I'd like to know what the
community thinks and why.

We are in the early stages of evaluating solutions, so any feedback at
this point is very welcome.


Thoughts?


Best regards,

Dave Orme

On Mon, 2008-03-24 at 15:07 -0600, David Kyle wrote:
> Hi,
>
> Is it realistic to use a CompositeTable for an extremely large table? A bit
> of playing around with CompositeTableSnippet4 shows that it can handle
> millions of rows. Hurrah! But it cannot handle 20,000 columns. Even a small
> 500 column table caused out of resource issues.
>
> I noticed that
> http://www.eclipse.org/nebula/widgets/compositetable/composi tetable.php
> mentions
>
> "4. Requests only visible data using a virtual table API for maximum
> scalability and performance."
>
> Has any progress been made on this point? My only other option is to use
> Albireo and implement a Swing JTable.
>
> Thanks,
> David Kyle
> http://richclientplatform.blogspot.com
>
>
Re: [CompositeTable] a question of size [message #51068 is a reply to message #51042] Wed, 26 March 2008 08:15 Go to previous messageGo to next message
Roland Tepp is currently offline Roland TeppFriend
Messages: 336
Registered: July 2009
Senior Member
Hello,

I'd start by arguing that if the table has so many columns, there is
something wrong with the design and the design should thus be
reconsidered. At least this is my opinions and I do tend to prefer clean
interfaces over cluttered ones (I consider tables with more than 5
columns generally a "cluttered" interface)...

On the other hand, if there must be gazillion columns I'd opt for
something in between choices #3 and #4.

That is - if CompositeTable is used essentially as a larger, more
scalable Table implementation, there should be some generic API to
support that use case. Preferably such that would not require too much
internal knowledge of the implementation details unless one is trying to
display some esoteric content...

Basically I'd envision that the api would have special Row
implementation that would contain lightweight "column" items, that are
responsible for displaying and editing of their content. Row
implementation would handle lazily loading and unloading of the column
items as they become visible or get scrolled off the screen.

The implementation of this row might easily use Draw2d as row item
implementation, but I'm a bit afraid of the extra dependency this would
intoduce, not every user is interested in (me, for example - unless I
can use Draw2D widgets with a non-columnar row item layout as well).


Dave Orme kirjutas mulle midagi seesugust:
> Hi David,
>
> This is a really good question, and hits one of the last remaining
> CompositeTable limitations head-on.
>
> I'd like to ask the community for feedback on this, because we're
> starting to work on addressing this right now.
>
> First: What kind of scalability do you realistically need? I've heard
> of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
> that's ~280 visible columns. Does anyone need more scalability than
> that?
>
> Here are some approaches for addressing this limitation:
>
> 1) Tweak the per-process resource limit in the Windows registry. This
> obviously just moves the threshold, but it might be enough. However,
> some IT teams might balk at widespread registry editing. Does anyone
> have experience with this solution?
>
> 2) Make CompositeTable (optionally) virtual in the horizontal as well as
> the vertical direction. This would require API on the row objects to
> create/dispose cell objects as well as much more complicated horizontal
> scrolling logic.
>
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?
>
> 4) Create an abstract Row implementation that uses the KTable approach.
> ie: Draw cells using renderers and edit using CellEditors (kind of like
> JFace). This requires reimplementing a few UI toolkit abstractions like
> hit testing, and makes it hard to implement "always-on" editing. But
> it's doable.
>
> Personally, I'm leaning towards choice #3, but I'd like to know what the
> community thinks and why.
>
> We are in the early stages of evaluating solutions, so any feedback at
> this point is very welcome.
>
>
> Thoughts?
>
>
> Best regards,
>
> Dave Orme
>
> On Mon, 2008-03-24 at 15:07 -0600, David Kyle wrote:
>> Hi,
>>
>> Is it realistic to use a CompositeTable for an extremely large table? A bit
>> of playing around with CompositeTableSnippet4 shows that it can handle
>> millions of rows. Hurrah! But it cannot handle 20,000 columns. Even a small
>> 500 column table caused out of resource issues.
>>
>> I noticed that
>> http://www.eclipse.org/nebula/widgets/compositetable/composi tetable.php
>> mentions
>>
>> "4. Requests only visible data using a virtual table API for maximum
>> scalability and performance."
>>
>> Has any progress been made on this point? My only other option is to use
>> Albireo and implement a Swing JTable.
>>
>> Thanks,
>> David Kyle
>> http://richclientplatform.blogspot.com
>>
>>
>

--
Roland Tepp
Re: [CompositeTable] a question of size [message #51096 is a reply to message #51042] Thu, 27 March 2008 14:25 Go to previous messageGo to next message
David Kyle is currently offline David KyleFriend
Messages: 125
Registered: July 2009
Senior Member
> I'd like to ask the community for feedback on this, because we're
> starting to work on addressing this right now.
>
> First: What kind of scalability do you realistically need? I've heard
> of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
> that's ~280 visible columns. Does anyone need more scalability than
> that?

280 visible columns would be an extreme for us. Most of our users look at
about 15 columns to a maximum of about 125 columns across multiple monitors.

> Here are some approaches for addressing this limitation:
>
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?

I would be most comfortable with this solution.

FYI: Our tables are read only. We just need a selection service. For the
time being I'm currently looking at the Albireo project to use a Swing
JTable. In the future, I would be very happy to rip out Swing JTable and
replace it with a Nebula component.

David Kyle
http://richclientplatform.blogspot.com
Re: [CompositeTable] a question of size [message #51152 is a reply to message #51042] Thu, 27 March 2008 16:08 Go to previous messageGo to next message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
On Tue, 2008-03-25 at 15:09 -0500, Dave Orme wrote:
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?

Yesterday I checked my understanding about Draw2d and found it to be
incorrect about having a Text implementation. Draw2d is display only
for now. Apparently, GEF-based editors that allow in-place editing are
putting an SWT Text on top of the Draw2d canvas...


Dave
Re: [CompositeTable] a question of size [message #51180 is a reply to message #51068] Thu, 27 March 2008 16:50 Go to previous messageGo to next message
David Kyle is currently offline David KyleFriend
Messages: 125
Registered: July 2009
Senior Member
Roland,

We have a project in house where:

trace geometry database is (place a Dr. Evil pinky near mouth)
1,000,000,000 rows by 10 columns
trace header values: 500 rows by 2,500 columns per shot. There are
just under 400,000 shots.
trace sample values: 1,500 rows by 2,500 columns per shot. There are just
under 400,000 shots.

The size of a shot can vary up to 10,000 columns on some projects. In about
three years we expect to see 20,000 columns per shot.

The data volumes are overwhelming. It can takes us a few hundred of years of
computer time to process a large project and our clients can take years to
analyze the data.

I'm all for reconsidering how we present our data. The EclipseCon 2008 UI
and design BoF's were a big help. But we do need to present the data in a
raw form (ie: table) to our users.

David
http://richclientplatform.blogspot.com
Re: [CompositeTable] a question of size [message #51234 is a reply to message #51068] Fri, 28 March 2008 16:52 Go to previous messageGo to next message
Peter Centgraf is currently offline Peter CentgrafFriend
Messages: 52
Registered: July 2009
Member
Roland Tepp wrote:
> That is - if CompositeTable is used essentially as a larger, more
> scalable Table implementation, there should be some generic API to
> support that use case.

I think there may be a conceptual mismatch here. AFAIK, CompositeTable
will never be _more_ efficient than the native Table using TableItems
and the SWT.VIRTUAL style. CT uses more resources to provide a richer
UI than is possible with Table.

At least on Windows, TableItem is a very lightweight implementation, and
most of the drawing logic is based on a large virtual canvas. From what
I can tell, the scaling limits for Table are all based on the maximum
number of pixels that can be addressed by a 32 bit integer, which is a
pretty impressive number.

CompositeTable attempts to implement a similar UI concept using
heavyweight controls arranged on a smaller drawing surface. The scaling
limits for CT are based on the number of heavyweight controls that the
OS can efficiently manage, which is a much smaller number -- an OS-wide
global limit of 10,000 is the default for Windows XP. This is the total
number of controls allowed by the OS in all applications combined.

If you are using CT with a layout similar to a basic Table, each visible
row will consume at least 1 + C control handles, where C is the number
of columns. This assumes that you are using basic SWT controls like
Text that use only 1 handle each. More complex composite controls can
use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
therefore consume O(C*V) handles, where V is the number of visible rows.
The use case described by David Kyle would break the OS limits with a
single visible row.

The solution proposed by Dave Orme would use a Canvas to draw each row,
rather than heavyweight widgets. This would reduce the number of
handles to O(V), but would also forfeit the major benefits of using CT,
such as "always on" editing. With large monitors and narrow rows, even
O(V) controls might be too many for Windows.

Ultimately, I doubt that any custom table widget implementation will be
more efficient than the highly-optimized controls provided by the OS
vendors. If you need to display more data than the standard SWT Table
will support, you have two options: write a custom table that trades
more scale for less responsiveness by redrawing without the virtual
canvas or change the UI design to fit the working set within reasonable
constraints.

IMHO, the second option is much better. Human eyes and minds cannot
process that much visual information simultaneously, so it is not
effective to present it that way. You have entered the realm of
multi-dimensional visualization, which is still an active research area
for Human-Computer Interaction academics. Perhaps try a Context-Detail
approach using a 2D heat map for the context and a standard Table for
the detail. Even rendering a single pixel per cell would require a
large, high-res display for your data sets, and it might still have to
scroll.

P.S. David, if you are interested in exploring alternative UI design
ideas, please contact me directly by email.

--
Peter
Re: [CompositeTable] a question of size [message #51374 is a reply to message #51234] Mon, 07 April 2008 08:18 Go to previous messageGo to next message
Roland Tepp is currently offline Roland TeppFriend
Messages: 336
Registered: July 2009
Senior Member
Come to think of it, I tend to agree with you, although not in so many
words :)


As to Kyle's use case - I wander, why does he not use the native Table
implementation (with VIRTUAL support, of course...)

In his case, he also might want to implement his own viewer, as to my
experience, current implementations of Table viewers are somewhat
inefficient in that they do not release offscreen TableItems, once
initialized...

Peter Centgraf kirjutas mulle midagi seesugust:
> Roland Tepp wrote:
>> That is - if CompositeTable is used essentially as a larger, more
>> scalable Table implementation, there should be some generic API to
>> support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter

--
Roland Tepp
Re: [CompositeTable] a question of size [message #51402 is a reply to message #51234] Tue, 08 April 2008 02:41 Go to previous messageGo to next message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
My initial implementation of CompositeTable was based on the SWT Table.
However, SWT doesn't guarantee that the order of events will remain
consistent across platforms. I found this to be problematic when trying
to work across platforms while basing the control on SWT's Table.

This led to the idea of not using SWT's Table at all, but building
something out of real SWT controls that looks like a Table. After all,
computer graphics are just a big optical illusion anyway.

This was quite a few years ago (4, I think?), so maybe things have
improved since then.

I'd be interested to hear of peoples' cross-platform experiences trying
to build a rich editing experience on top of SWT's table.


Best regards,

Dave Orme

On Fri, 2008-03-28 at 12:52 -0400, Peter Centgraf wrote:
> Roland Tepp wrote:
> > That is - if CompositeTable is used essentially as a larger, more
> > scalable Table implementation, there should be some generic API to
> > support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter
Re: [CompositeTable] a question of size [message #51430 is a reply to message #51180] Tue, 08 April 2008 02:47 Go to previous messageGo to next message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
On Thu, 2008-03-27 at 10:50 -0600, David Kyle wrote:
> The size of a shot can vary up to 10,000 columns on some projects. In
> about
> three years we expect to see 20,000 columns per shot.

If CompositeTable were virtual in the horizontal as well as the vertical
direction, would that help you? In other words, how big is your
viewport?

The problem we have is that we may only have 50 rows visible, but some
of our clients might display a table across 7 or so monitors, so at over
200 columns visible in the viewport, the SWT handle usage is still
rather excessive.

BTW, I was in the UI design BOF too. Too bad we didn't meet. :-)


Regards,

Dave
Re: [CompositeTable] a question of size [message #51457 is a reply to message #51234] Tue, 08 April 2008 02:47 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
Peter,

Do you have pointers to any good resources on alternate ways of
displaying large amounts of tabular data?


Regards,

Dave

On Fri, 2008-03-28 at 12:52 -0400, Peter Centgraf wrote:
> Roland Tepp wrote:
> > That is - if CompositeTable is used essentially as a larger, more
> > scalable Table implementation, there should be some generic API to
> > support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter
Re: [CompositeTable] a question of size [message #589124 is a reply to message #50986] Tue, 25 March 2008 20:09 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
Hi David,

This is a really good question, and hits one of the last remaining
CompositeTable limitations head-on.

I'd like to ask the community for feedback on this, because we're
starting to work on addressing this right now.

First: What kind of scalability do you realistically need? I've heard
of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
that's ~280 visible columns. Does anyone need more scalability than
that?

Here are some approaches for addressing this limitation:

1) Tweak the per-process resource limit in the Windows registry. This
obviously just moves the threshold, but it might be enough. However,
some IT teams might balk at widespread registry editing. Does anyone
have experience with this solution?

2) Make CompositeTable (optionally) virtual in the horizontal as well as
the vertical direction. This would require API on the row objects to
create/dispose cell objects as well as much more complicated horizontal
scrolling logic.

3) Support Draw2d widgets in row objects. Since Draw2d basically
reimplements Swing inside an SWT Canvas, you don't wind up with the huge
resource requirements, you get to use the Draw2d hit testing code, and
the (small) library of Draw2d widgets. At least Draw2d has a Text
implementation, probably the most important widget. Thoughts?

4) Create an abstract Row implementation that uses the KTable approach.
ie: Draw cells using renderers and edit using CellEditors (kind of like
JFace). This requires reimplementing a few UI toolkit abstractions like
hit testing, and makes it hard to implement "always-on" editing. But
it's doable.

Personally, I'm leaning towards choice #3, but I'd like to know what the
community thinks and why.

We are in the early stages of evaluating solutions, so any feedback at
this point is very welcome.


Thoughts?


Best regards,

Dave Orme

On Mon, 2008-03-24 at 15:07 -0600, David Kyle wrote:
> Hi,
>
> Is it realistic to use a CompositeTable for an extremely large table? A bit
> of playing around with CompositeTableSnippet4 shows that it can handle
> millions of rows. Hurrah! But it cannot handle 20,000 columns. Even a small
> 500 column table caused out of resource issues.
>
> I noticed that
> http://www.eclipse.org/nebula/widgets/compositetable/composi tetable.php
> mentions
>
> "4. Requests only visible data using a virtual table API for maximum
> scalability and performance."
>
> Has any progress been made on this point? My only other option is to use
> Albireo and implement a Swing JTable.
>
> Thanks,
> David Kyle
> http://richclientplatform.blogspot.com
>
>
Re: [CompositeTable] a question of size [message #589133 is a reply to message #51042] Wed, 26 March 2008 08:15 Go to previous message
Roland Tepp is currently offline Roland TeppFriend
Messages: 336
Registered: July 2009
Senior Member
Hello,

I'd start by arguing that if the table has so many columns, there is
something wrong with the design and the design should thus be
reconsidered. At least this is my opinions and I do tend to prefer clean
interfaces over cluttered ones (I consider tables with more than 5
columns generally a "cluttered" interface)...

On the other hand, if there must be gazillion columns I'd opt for
something in between choices #3 and #4.

That is - if CompositeTable is used essentially as a larger, more
scalable Table implementation, there should be some generic API to
support that use case. Preferably such that would not require too much
internal knowledge of the implementation details unless one is trying to
display some esoteric content...

Basically I'd envision that the api would have special Row
implementation that would contain lightweight "column" items, that are
responsible for displaying and editing of their content. Row
implementation would handle lazily loading and unloading of the column
items as they become visible or get scrolled off the screen.

The implementation of this row might easily use Draw2d as row item
implementation, but I'm a bit afraid of the extra dependency this would
intoduce, not every user is interested in (me, for example - unless I
can use Draw2D widgets with a non-columnar row item layout as well).


Dave Orme kirjutas mulle midagi seesugust:
> Hi David,
>
> This is a really good question, and hits one of the last remaining
> CompositeTable limitations head-on.
>
> I'd like to ask the community for feedback on this, because we're
> starting to work on addressing this right now.
>
> First: What kind of scalability do you realistically need? I've heard
> of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
> that's ~280 visible columns. Does anyone need more scalability than
> that?
>
> Here are some approaches for addressing this limitation:
>
> 1) Tweak the per-process resource limit in the Windows registry. This
> obviously just moves the threshold, but it might be enough. However,
> some IT teams might balk at widespread registry editing. Does anyone
> have experience with this solution?
>
> 2) Make CompositeTable (optionally) virtual in the horizontal as well as
> the vertical direction. This would require API on the row objects to
> create/dispose cell objects as well as much more complicated horizontal
> scrolling logic.
>
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?
>
> 4) Create an abstract Row implementation that uses the KTable approach.
> ie: Draw cells using renderers and edit using CellEditors (kind of like
> JFace). This requires reimplementing a few UI toolkit abstractions like
> hit testing, and makes it hard to implement "always-on" editing. But
> it's doable.
>
> Personally, I'm leaning towards choice #3, but I'd like to know what the
> community thinks and why.
>
> We are in the early stages of evaluating solutions, so any feedback at
> this point is very welcome.
>
>
> Thoughts?
>
>
> Best regards,
>
> Dave Orme
>
> On Mon, 2008-03-24 at 15:07 -0600, David Kyle wrote:
>> Hi,
>>
>> Is it realistic to use a CompositeTable for an extremely large table? A bit
>> of playing around with CompositeTableSnippet4 shows that it can handle
>> millions of rows. Hurrah! But it cannot handle 20,000 columns. Even a small
>> 500 column table caused out of resource issues.
>>
>> I noticed that
>> http://www.eclipse.org/nebula/widgets/compositetable/composi tetable.php
>> mentions
>>
>> "4. Requests only visible data using a virtual table API for maximum
>> scalability and performance."
>>
>> Has any progress been made on this point? My only other option is to use
>> Albireo and implement a Swing JTable.
>>
>> Thanks,
>> David Kyle
>> http://richclientplatform.blogspot.com
>>
>>
>

--
Roland Tepp
Re: [CompositeTable] a question of size [message #589140 is a reply to message #51042] Thu, 27 March 2008 14:25 Go to previous message
David Kyle is currently offline David KyleFriend
Messages: 125
Registered: July 2009
Senior Member
> I'd like to ask the community for feedback on this, because we're
> starting to work on addressing this right now.
>
> First: What kind of scalability do you realistically need? I've heard
> of RCP apps spanning up to 7 monitors. Given ~40 columns per monitor,
> that's ~280 visible columns. Does anyone need more scalability than
> that?

280 visible columns would be an extreme for us. Most of our users look at
about 15 columns to a maximum of about 125 columns across multiple monitors.

> Here are some approaches for addressing this limitation:
>
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?

I would be most comfortable with this solution.

FYI: Our tables are read only. We just need a selection service. For the
time being I'm currently looking at the Albireo project to use a Swing
JTable. In the future, I would be very happy to rip out Swing JTable and
replace it with a Nebula component.

David Kyle
http://richclientplatform.blogspot.com
Re: [CompositeTable] a question of size [message #589155 is a reply to message #51042] Thu, 27 March 2008 16:08 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
On Tue, 2008-03-25 at 15:09 -0500, Dave Orme wrote:
> 3) Support Draw2d widgets in row objects. Since Draw2d basically
> reimplements Swing inside an SWT Canvas, you don't wind up with the huge
> resource requirements, you get to use the Draw2d hit testing code, and
> the (small) library of Draw2d widgets. At least Draw2d has a Text
> implementation, probably the most important widget. Thoughts?

Yesterday I checked my understanding about Draw2d and found it to be
incorrect about having a Text implementation. Draw2d is display only
for now. Apparently, GEF-based editors that allow in-place editing are
putting an SWT Text on top of the Draw2d canvas...


Dave
Re: [CompositeTable] a question of size [message #589159 is a reply to message #51068] Thu, 27 March 2008 16:50 Go to previous message
David Kyle is currently offline David KyleFriend
Messages: 125
Registered: July 2009
Senior Member
Roland,

We have a project in house where:

trace geometry database is (place a Dr. Evil pinky near mouth)
1,000,000,000 rows by 10 columns
trace header values: 500 rows by 2,500 columns per shot. There are
just under 400,000 shots.
trace sample values: 1,500 rows by 2,500 columns per shot. There are just
under 400,000 shots.

The size of a shot can vary up to 10,000 columns on some projects. In about
three years we expect to see 20,000 columns per shot.

The data volumes are overwhelming. It can takes us a few hundred of years of
computer time to process a large project and our clients can take years to
analyze the data.

I'm all for reconsidering how we present our data. The EclipseCon 2008 UI
and design BoF's were a big help. But we do need to present the data in a
raw form (ie: table) to our users.

David
http://richclientplatform.blogspot.com
Re: [CompositeTable] a question of size [message #589172 is a reply to message #51068] Fri, 28 March 2008 16:52 Go to previous message
Peter Centgraf is currently offline Peter CentgrafFriend
Messages: 52
Registered: July 2009
Member
Roland Tepp wrote:
> That is - if CompositeTable is used essentially as a larger, more
> scalable Table implementation, there should be some generic API to
> support that use case.

I think there may be a conceptual mismatch here. AFAIK, CompositeTable
will never be _more_ efficient than the native Table using TableItems
and the SWT.VIRTUAL style. CT uses more resources to provide a richer
UI than is possible with Table.

At least on Windows, TableItem is a very lightweight implementation, and
most of the drawing logic is based on a large virtual canvas. From what
I can tell, the scaling limits for Table are all based on the maximum
number of pixels that can be addressed by a 32 bit integer, which is a
pretty impressive number.

CompositeTable attempts to implement a similar UI concept using
heavyweight controls arranged on a smaller drawing surface. The scaling
limits for CT are based on the number of heavyweight controls that the
OS can efficiently manage, which is a much smaller number -- an OS-wide
global limit of 10,000 is the default for Windows XP. This is the total
number of controls allowed by the OS in all applications combined.

If you are using CT with a layout similar to a basic Table, each visible
row will consume at least 1 + C control handles, where C is the number
of columns. This assumes that you are using basic SWT controls like
Text that use only 1 handle each. More complex composite controls can
use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
therefore consume O(C*V) handles, where V is the number of visible rows.
The use case described by David Kyle would break the OS limits with a
single visible row.

The solution proposed by Dave Orme would use a Canvas to draw each row,
rather than heavyweight widgets. This would reduce the number of
handles to O(V), but would also forfeit the major benefits of using CT,
such as "always on" editing. With large monitors and narrow rows, even
O(V) controls might be too many for Windows.

Ultimately, I doubt that any custom table widget implementation will be
more efficient than the highly-optimized controls provided by the OS
vendors. If you need to display more data than the standard SWT Table
will support, you have two options: write a custom table that trades
more scale for less responsiveness by redrawing without the virtual
canvas or change the UI design to fit the working set within reasonable
constraints.

IMHO, the second option is much better. Human eyes and minds cannot
process that much visual information simultaneously, so it is not
effective to present it that way. You have entered the realm of
multi-dimensional visualization, which is still an active research area
for Human-Computer Interaction academics. Perhaps try a Context-Detail
approach using a 2D heat map for the context and a standard Table for
the detail. Even rendering a single pixel per cell would require a
large, high-res display for your data sets, and it might still have to
scroll.

P.S. David, if you are interested in exploring alternative UI design
ideas, please contact me directly by email.

--
Peter
Re: [CompositeTable] a question of size [message #589208 is a reply to message #51234] Mon, 07 April 2008 08:18 Go to previous message
Roland Tepp is currently offline Roland TeppFriend
Messages: 336
Registered: July 2009
Senior Member
Come to think of it, I tend to agree with you, although not in so many
words :)


As to Kyle's use case - I wander, why does he not use the native Table
implementation (with VIRTUAL support, of course...)

In his case, he also might want to implement his own viewer, as to my
experience, current implementations of Table viewers are somewhat
inefficient in that they do not release offscreen TableItems, once
initialized...

Peter Centgraf kirjutas mulle midagi seesugust:
> Roland Tepp wrote:
>> That is - if CompositeTable is used essentially as a larger, more
>> scalable Table implementation, there should be some generic API to
>> support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter

--
Roland Tepp
Re: [CompositeTable] a question of size [message #589215 is a reply to message #51234] Tue, 08 April 2008 02:41 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
My initial implementation of CompositeTable was based on the SWT Table.
However, SWT doesn't guarantee that the order of events will remain
consistent across platforms. I found this to be problematic when trying
to work across platforms while basing the control on SWT's Table.

This led to the idea of not using SWT's Table at all, but building
something out of real SWT controls that looks like a Table. After all,
computer graphics are just a big optical illusion anyway.

This was quite a few years ago (4, I think?), so maybe things have
improved since then.

I'd be interested to hear of peoples' cross-platform experiences trying
to build a rich editing experience on top of SWT's table.


Best regards,

Dave Orme

On Fri, 2008-03-28 at 12:52 -0400, Peter Centgraf wrote:
> Roland Tepp wrote:
> > That is - if CompositeTable is used essentially as a larger, more
> > scalable Table implementation, there should be some generic API to
> > support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter
Re: [CompositeTable] a question of size [message #589229 is a reply to message #51180] Tue, 08 April 2008 02:47 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
On Thu, 2008-03-27 at 10:50 -0600, David Kyle wrote:
> The size of a shot can vary up to 10,000 columns on some projects. In
> about
> three years we expect to see 20,000 columns per shot.

If CompositeTable were virtual in the horizontal as well as the vertical
direction, would that help you? In other words, how big is your
viewport?

The problem we have is that we may only have 50 rows visible, but some
of our clients might display a table across 7 or so monitors, so at over
200 columns visible in the viewport, the SWT handle usage is still
rather excessive.

BTW, I was in the UI design BOF too. Too bad we didn't meet. :-)


Regards,

Dave
Re: [CompositeTable] a question of size [message #589235 is a reply to message #51234] Tue, 08 April 2008 02:47 Go to previous message
Dave Orme is currently offline Dave OrmeFriend
Messages: 424
Registered: July 2009
Senior Member
Peter,

Do you have pointers to any good resources on alternate ways of
displaying large amounts of tabular data?


Regards,

Dave

On Fri, 2008-03-28 at 12:52 -0400, Peter Centgraf wrote:
> Roland Tepp wrote:
> > That is - if CompositeTable is used essentially as a larger, more
> > scalable Table implementation, there should be some generic API to
> > support that use case.
>
> I think there may be a conceptual mismatch here. AFAIK, CompositeTable
> will never be _more_ efficient than the native Table using TableItems
> and the SWT.VIRTUAL style. CT uses more resources to provide a richer
> UI than is possible with Table.
>
> At least on Windows, TableItem is a very lightweight implementation, and
> most of the drawing logic is based on a large virtual canvas. From what
> I can tell, the scaling limits for Table are all based on the maximum
> number of pixels that can be addressed by a 32 bit integer, which is a
> pretty impressive number.
>
> CompositeTable attempts to implement a similar UI concept using
> heavyweight controls arranged on a smaller drawing surface. The scaling
> limits for CT are based on the number of heavyweight controls that the
> OS can efficiently manage, which is a much smaller number -- an OS-wide
> global limit of 10,000 is the default for Windows XP. This is the total
> number of controls allowed by the OS in all applications combined.
>
> If you are using CT with a layout similar to a basic Table, each visible
> row will consume at least 1 + C control handles, where C is the number
> of columns. This assumes that you are using basic SWT controls like
> Text that use only 1 handle each. More complex composite controls can
> use many more -- CCombo uses 5 and CDateTime can use hundreds. CT will
> therefore consume O(C*V) handles, where V is the number of visible rows.
> The use case described by David Kyle would break the OS limits with a
> single visible row.
>
> The solution proposed by Dave Orme would use a Canvas to draw each row,
> rather than heavyweight widgets. This would reduce the number of
> handles to O(V), but would also forfeit the major benefits of using CT,
> such as "always on" editing. With large monitors and narrow rows, even
> O(V) controls might be too many for Windows.
>
> Ultimately, I doubt that any custom table widget implementation will be
> more efficient than the highly-optimized controls provided by the OS
> vendors. If you need to display more data than the standard SWT Table
> will support, you have two options: write a custom table that trades
> more scale for less responsiveness by redrawing without the virtual
> canvas or change the UI design to fit the working set within reasonable
> constraints.
>
> IMHO, the second option is much better. Human eyes and minds cannot
> process that much visual information simultaneously, so it is not
> effective to present it that way. You have entered the realm of
> multi-dimensional visualization, which is still an active research area
> for Human-Computer Interaction academics. Perhaps try a Context-Detail
> approach using a 2D heat map for the context and a standard Table for
> the detail. Even rendering a single pixel per cell would require a
> large, high-res display for your data sets, and it might still have to
> scroll.
>
> P.S. David, if you are interested in exploring alternative UI design
> ideas, please contact me directly by email.
>
> --
> Peter
Previous Topic:making column truly fixed, setMovable(false) on column can be worked around
Next Topic:Preserving cell selection and focus in GridTableViewer
Goto Forum:
  


Current Time: Thu Dec 18 15:29:03 GMT 2014

Powered by FUDForum. Page generated in 0.03267 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software