|
IMO, if the there is a bugzilla open
for a failing test and the failure cannot be fixed in less than a
week, then I think it's perfectly OK to comment out the test. At
least that's what we do in platform debug.
Cheers,
Pawel
On 08/21/2012 12:47 PM, Cortell John-RAT042 wrote:
I brought this up for discussion last week, before I pushed
the changes. One person had concerns; we began a debate and
the thread ended without retort to my points. Thus I went
ahead with my changes.
That said, I will respect a general consensus. I encourage a few
more folks to weigh in so we can get there. Honestly, I'm quire
surprised that such a small and innocuous set of changes to
test code is drawing such a reaction.
John
I agree, +1.
There are other ways to achieve test
suite that passes 100%. You can create separate test
suite that includes only tests you wish to test. You
can keep your changes on local branch as Sergey
suggested. You can create a special branch in CDT
repository even if there is really need to share.
But, please, do not keep it on master branch.
Andrew
On Tue, Aug 21, 2012 at
3:05 PM, Sergey Prigogin
<eclipse.sprigogin@xxxxxxxxx>
wrote:
I propose to revert the commit http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=1bd247ce3d8825b76623166a89b111884ae1b195.
John, you can keep the commit on a local
branch if you prefer.
-sergey
On Tue, Aug 21,
2012 at 11:55 AM, Cortell John-RAT042
<RAT042@xxxxxxxxxxxxx>
wrote:
Andrew,
If the filters I added did
not cover failures seen by
someone else, and that person
also wants to have a reliable
test suite, then he should add
additional filters. If I see
additional unreliable tests in
future runs, I will add
filters for them.
I wish creating a
bugzilla report would result
in these intermittent
failures being fixed. But
I'm not going to hold my
breath and I'm not going to
waste time creating bugzilla
reports for tests that
someone has written but has
not ensured they run
successfully and
consistently on Windows. As
you already know, there are
intermittent failures even
on Linux. I've spent a
considerable amount of time
debugging and fixing ones I
could tackle. But I have to
move on.
What I want is to be able
to run a CDT test suite
locally and have zero
failures, based on what's in
git. Sadly, with these
filters, I've reduced the
overall test suite from
about 12K tests to 3K[*].
I've had to filter entire
chunks because failures
happen not only
intermittently but in random
locations within the
particular chunk. Hopefully
someone who has the time and
motivation to run and
troubleshoot these tests on
Windows will try to address
this situation.
I said it before and I'll
say it again: a test suite
that produces dozens of
failures is all but useless
for regression detection. I
want a test suite that
passes 100%. What I've done
is rigged the tests so that
I and others can get that
subset with the flip of a
switch.
John
[*] Again, these filters
must be manually activated.
By default, all tests still
run.
Hi John,
This approach
does not look
particularly
scalable. Is it
intended that
every commiter
will add to the
CDT code filters
for individual
tests that fail
in his/her
environment?
//
Consistently
fails, and,
yes, I do have
Cygwin in my
PATH
if
(System.getProperty("cdt.skip.known.test.failures")
!= null) {
//$NON-NLS-1$
return;
}
Why don't
create a bug
in bugzilla
instead to fix
the problem
for good?
Thanks,
Andrew
On
Fri, Aug 17,
2012 at 12:49
PM, Cortell
John-RAT042
<RAT042@xxxxxxxxxxxxx>
wrote:
I
was able to
resolve
roughly a
dozen
intermittent
failures
caused by a
bug in
org.eclipse.cdt.ui.tests.BaseUITestCase.checkTreeNode(IViewPart,
int, String)
The
implementation
searched the
entire
workbench for
the SWT Tree
instead of
just the given
view. This
would
intermittently
(but
frequently
enough)
return, e.g.,
the Outline
view’s tree
because it has
a root element
with the same
label as that
in the Call
hierarchy
view. It was
just a matter
of the order
in which the
controls in
the workbench
were found,
which is
undefined and
inconsistent.
So,
here we have
an example of
a test being
unreliable,
unrelated to
OS or
environment.
The test had a
bug and you
and Hudson
were simply
getting lucky.
I’ve
pushed the fix
to master.
John
> Since
we don't have
Hudson on
Windows
There is a
Windows Hudson
slave, isn't
there? A job
could be set
up to run
either the
full CDT build
or just the
tests on it.
Andrew
_______________________________________________
cdt-dev
mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
|
