[eclipse.org-planning-council] the testing, usability and process of EPP

["Mik Kersten" <beatmik@xxxxxxx>]

Follow up to:
* Doug S's post on EPP as products:
 
http://dev.eclipse.org/mhonarc/lists/eclipse.org-planning-council/msg01043.h
tml 
* Martin A's post on a better download page
 
http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01586.html
* Markus' post on need for process
 
http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01715.html 
* Mike M's post on cross-projects testing
 
http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01493.html

It's good to see the discussions about improving quality.  I think that it's
very important to also make the EPP unit a first-class part of such
discussions and related process.  In my experience since June 29th, when
users download Europa via update site, they generally know that they're
getting a large collection of disparate projects.  When they download an EPP
package they're expecting a cohesive product on par with the quality and
usability of competing products like VisualStudio.  The fact that better
supported commercial offerings exist doesn't appear to change this
expectation much.  By the download numbers, the overall quality and
usability of the EPP packagings define the quality of what is "Eclipse",
just as the quality of the SDK used to do.  Independent of how EPP continues
to evolve, I think we need to recognize this and put some process or
practice in place around it.  Note that I am in no way trying to imply that
EPP projects are better than others, need higher priority on IP reviews, or
anything of that sort, just that bugs and inconsistencies in EPP projects
have a much broader impact. 

TESTING

We have seen two annoying cross-project related Mylyn bugs related to EPP
packagings:

1) An EPP-triggered change made mid-June resulted in Mylyn contributing
duplicate content assist rankings in an EPP packaging of a workspace in
which a user never used Mylyn.  This was discovered very late in the Europa
release and barely fixed in time.  (bug 194766)

2) In the JEE EPP packing of Mylyn, in relatively rare situations with the
Sun VM, if no Mylyn UI is loaded Mylyn will show class loading warnings to
show in the Error Log.  Some users interpret these warnings to be a broken
Eclipse install.  With the Fall Update we saw more complaints and these
warnings attributed to the MaxPermSize settings regression.  In terms of
blocking use this was a minor bug, but in terms of confusion it was major.
(bug 188524)

As with other projects, Mylyn relies on a large early adopter community to
identify these sorts of corner cases.  However, both of these bugs were
triggered by people *not* using Mylyn, so we are not covered in the same
way.  If we don't come up with a better way of testing the packagings bugs
and usability problems like will come up again.  

Assuming that user community testing is our best resource, I think that we
need a visible and continually integrated download of the latest Ganymede
EPP packagings using weekly I-builds from all the projects.  We might want
Europa ones too, since that could have caught the Platform MaxPermSize
setting regression.  One way of doing this is to link from downloads a page
that's called Early Adopter Downloads and have weekly-integrated Platform
and EPP builds available from it.

USABILITY

The community has a very high expectation for EPP packaging usability that
has been set by Platform's amazingly consistent and polished UI.  I think
that it's important for all projects to follow this example and the Eclipse
UI Guidelines, and I think that it's critical for EPP projects put at least
a some cycles towards following and evolving these guidelines.  Otherwise we
end up with understandable flames on blogs about why some button is always
visible on the toolbar, etc.  There is an open and difficult question of
enforcement of any UI guidelines.  But my position on Mylyn is that since
we're bundled with EPP we should follow the entire UI Guidelines checklist
items, help evolve them wherever not possible, and review our adherence to
the guidelines with the UI Best Practice Working group with enough time to
evolve based on input.  

PROCESS

I was at a loss around the 3.3.1.1 download issue in terms of whether Mylyn
should provide a fix for bug (2) above, so we didn't because it wasn't a
regression or blocker.  But I don't want us making such decisions alone, and
instead think we need some kind of coordinated discussion or review of what
goes into the update, just as Platform has their PMC review those changes.
Similarly, I don't know how we should proceed if we had found a bad Mylyn
regression or blocker that needed a 3.3.1.1 style update to go out.  We
don't necessarily need a heavyweight process for this, just some kind of
cross-project review or meeting where the great brains on this list put some
cycles to figuring out whether a bug that affects the overall quality should
result in an update. 

Once an update of this sort has been requested by an EPP project (e.g.
Platform), I think that other projects should be notified via cross-projects
asap and then given at least the +1 time to test around any updates *after*
an update I-build EPP packaging is available.   Due to (2) we had to do
considerable manual testing around 3.3.1.1 and had very little time to do
it.

-------

Some of the above points apply to all release train projects.  However, I
think that each of the testing, usability, and process items listed above
has a relatively straightforward way of being addressed, and that if we
don't address them soon for the EPP we risk lowering the overall perceived
quality of Eclipse.  As such, I think that we should not let difficulties in
implementing some of this for all Ganymede projects stop doing it for EPP
projects, learn from those experiences, and figure out how that overlaps
with the other projects on the train.

Mik

--
Mik Kersten
Project Lead, http://eclipse.org/mylyn