RE: [hyades-dev] black- and white-box, was: integration vs interfacing

[Michael.Norman@xxxxxxxxxxxxx]

James/Tom

One of the original thought processes behind the Hyades Native 
Behaviours was that these provide a resolution point for the various 
tools, in other words by providing mechanisms to map to these EMF-based 
formats and mandating their use as persistence formats we are able to 
force interoperability at the test asset level amongst competing tools.  
We drifted away from this type of enforcement as we allowed external 
behaviours in the early days and I don't want to reopen the discussion 
now, but there are still benefits to trying to move people in that 
direction.  

However, Hyades is and remains a broad set of interests.  For example my 
personal interest is in late-lifecycle testing and diagnosis by systems 
integrators of production or near-production systems where source code 
is typically owned by one or more ISVs and not available for contractual 
reasons.  If you like its the systems that have been through developer 
testing and QA testing (one tends to have to make charitable assumptions 
that this is the case) and are still broken, typically because of 
cross-project and cross-contractual communication difficulties. Most of 
this is black-box although when your production system is broken you 
tend to care less about the purity of approach.

On the quesion of whether co-opting Junit into the project would force 
all Junit uses to adopt Hyades, the answer is no not in the sense that 
you mean it, because elements of hyades can be adopted separately (for 
example Intel are thinking of using the RAC/HCE but not at this stage 
the data models or user interface, and perhaps not in all cases the 
Eclipse workbench itself).  What it might allow us to do is provide 
better integration between Junit in the Eclipse IDE and Junit in Hyades, 
which could have a benefit for test management of junit amongst the 
significant proportion of the Java developer community who use an 
Eclipse-derived platform.  If this idea is being seriously entertained, 
someone can talk to Erich about it.  Serge has spoken to him on and off 
about the resolution point beween Junit and Hyades, and I speak to him 
about other stuff on a regular basis, but I think we need our ducks in a 
row first.

Mike


-----Original Message-----
From: James.Saliba [mailto:James.Saliba@xxxxxx]
Sent: 22 September 2004 15:07
To: hyades-dev
Subject: RE: [hyades-dev] black- and white-box, was: integration vs
interfacing


<snip>
>At some point current terminology, which assumes all testing is
>manual, and that developers don't test, will need to change--if one
>develops automated stress tests for application(s) to which one lacks
>access to sources, is one a "tester" or a "developer"? But for now,
>when I say "developer" I mean someone whose primary responsibility is
>to deliver sources (no matter how much testing of those sources s/he
>might do) and by "tester" I mean someone whose primary responsibility
>is to verify the quality of an application or its deployment (no
>matter how much coding s/he might do) and who lacks access to the
>target's sources.

>1 Presently testers are far better served by current tooling than are
>  developers. There is already a lotta black-box tooling out there.
>  Certainly much more time/energy has been invested in developing
>  black-box tooling (for dynamic testing) than has been invested in
>  white-box tooling. (Static analysis is a separate matter, and not
>  what I mean by "white-box" or "developer-time" testing.)


I can not disagree that there already is "a lotta" black-box tools out
there.  From my experience in running QA/testing teams this is also a
problem.   None of the tools work together.  The tests scripts are saved
in multiple formats and in multiple repositories make analysis and
project status difficult.   The power of TPTP is to bring these tools
together, save the test assets in one place hopefully in the same or
parallel change control system branch as the product code.  Many JUNIT
extensions support white-box testing while other addresses Black-box.
With Eclipse/TPTP & JUNIT they can, (and I believe they should) live
together.  Combine this with some enhancements to test management,
automated execution and general reporting, TPTP can house an amazing set
of test tools support the whole life cycle from product design to
production.  (Sorry, I'm off my soapbox now :)


Regards,
Jim Saliba

-----Original Message-----
From: hyades-dev-admin@xxxxxxxxxxx [mailto:hyades-dev-admin@xxxxxxxxxxx]
On Behalf Of Thomas L Roche
Sent: Tuesday, September 21, 2004 9:59 PM
To: hyades-dev@xxxxxxxxxxx
Subject: [hyades-dev] black- and white-box, was: integration vs
interfacing

"Saliba, James" Fri, 17 Sep 2004 17:13:20 -0400
>>> From my point of view the larger portion our audience is not doing
>>> "green fields" development; people are extending and build on
>>> existing products/projects. Therefore we must make Eclipse/TPTP
>>> attractive enough for people to want to switch. I believe this
>>> would include items from three general areas:

<big snip>

>>> 3) The ease of writing new test cases with Eclipse/TPTP 

Tom Roche 19 September 2004 20:21
>> I would add one more

>> 4) The ease of maintaining test cases developed with Eclipse/TPTP

>> FWIW 3 and 4 are my primary concerns: specifically, reducing the
total
>> lifecycle cost of testing for *developers*, in order that they and
>> their management will see the (admittedly overhead) cost as
>> acceptable, and begin to consistently/continually do automated
>> testing, rather than pushing it off to "dedicated" or "3rd party"
>> acceptance-type/black-box testers. But that's a topic for another
>> thread ... suffice to say that, IMHO, while black-box tools have made
>> progress in lowering the cost of _creating_ tests, tests thus
>> created/run will never be as maintainable (or even, in many cases, as
>> easy to develop) as those developed using xUnit-based tools that

>> * allow developers to create tests as (or even before) they develop
>>   their sources

>> * leverage developers' {access to, knowledge of} their sources

>> * are more easily integrated into development tools and processes

>> <off soapbox/>

Michael.Norman Tue, 21 Sep 2004 20:36:08 +0100
> From the size of the soap box obviously this white-box vs. black-box
> thing exercises a lot of people, but personally I've always seen
> value in both approaches, and I think amongst Hyades launch goals
> was to be methodology-neutral.

Agreed. To express the above better, v0.1:

AFAICS most Hyades developers come from a black-box background, and
are mostly interested in doing more/better black-box tooling.
Black-box is not bad! but IMHO

0 The nature of defect-cost increase ensures that (ceteris paribus)
  there are more potential cost savings to be gained from increased/
  improved testing by developers (and architects, but that's another
  thread) than from increased/improved testing by traditional,
  acceptance-style testers.

At some point current terminology, which assumes all testing is
manual, and that developers don't test, will need to change--if one
develops automated stress tests for application(s) to which one lacks
access to sources, is one a "tester" or a "developer"? But for now,
when I say "developer" I mean someone whose primary responsibility is
to deliver sources (no matter how much testing of those sources s/he
might do) and by "tester" I mean someone whose primary responsibility
is to verify the quality of an application or its deployment (no
matter how much coding s/he might do) and who lacks access to the
target's sources.

1 Presently testers are far better served by current tooling than are
  developers. There is already a lotta black-box tooling out there.
  Certainly much more time/energy has been invested in developing
  black-box tooling (for dynamic testing) than has been invested in
  white-box tooling. (Static analysis is a separate matter, and not
  what I mean by "white-box" or "developer-time" testing.)

Therefore 

2 Presently the marginal benefit likely derivable from investment in
  tooling intended to assist developers to do more/better automated
  testing greatly exceeds the marginal benefit likely derivable from
  investment in tooling intended to assist testers to do more/better
  automated testing.

How to do that?

3 "White-box" or "source-enabling" tools (like JUnit and its
  extensions) that

* allow developers to create tests as (or even before) they develop
  their sources

* leverage developers' {access to, knowledge of} their sources

* are more easily integrated into development tools and processes

  are more likely to realize the potential savings of developer-time
  testing. IMHO, while black-box tools have made progress in lowering
  the cost of _creating_ tests, tests thus created/run will never be
  as maintainable (or even, in many cases, as easy to develop) as
  those developed using xUnit-based tools possessing the properties
  listed.

This does not mean that JUnit extensions cannot be used to
acceptance-test or drive deployments, only that black-box tools are
(IMHO empirically) suboptimal for developer-time testing.

Conclusion:

4 I would like to see more Hyades effort devoted to assisting
  *developers* to do more/better automated testing using JUnit
  and its extensions.

_______________________________________________
hyades-dev mailing list
hyades-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/hyades-dev



_______________________________________________
hyades-dev mailing list
hyades-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/hyades-dev