|Re: [wtp-dev] Performance tests|
Hi all, I've a few questions regarding performance tests. Are performance tests and non-performance tests going to be run on the same build/test machine?
Currently, yes. However, there are no performnace test committed yet.
Performance tests are more sensitive to the environment they run in. Their results are only accurate if the initial environment are reasonablely the same every time. Running performance tests right after a build or after non-performance tests can affect their results. Ideally, rebooting the system before running each test can give us more accurate numbers; however, this's probably not possible given time limits. Is this being taken into account?
No, but I would be interested in hearing more. The build machines is an IBM x-series xeon with 2 GB ram and red hat linux. It seems to do well with builds (i.e. the utilization is still very low). It sounds too radical to me to reboot a machine to have a "reset". Linux should do a good job managing these issues. However, the build machine is rebooted only a few times a month so it is likely that there are some issues for performance testing. Unfortunately, we (eteration) will not be able to dedicate a seperate machine for perf-tests only.
The standard to which performance tests have failed or passed is different from non-performance tests. Are there any documents that speak out what's consider a performance failure (ex. x% or x seconds regression) and what's the next course of action if this happens (ex. send an email, open a bug). Also, in some cases, performance regression can be justified, for example, a very compelling feature. Is there any process for justifying changes that cause performance regressions? Have these questions been answered or documented somewhere? If not, I can help out on sorting out these issues.
Currently, there are no documents beyond that of base eclipse. But, please do this. It will be very valuable.
Thanks, Jeff _______________________________________________ wtp-dev mailing list wtp-dev@xxxxxxxxxxx http://dev.eclipse.org/mailman/listinfo/wtp-dev
Naci Dai, Managing Director eteration a.s. Inonu cad. Sumer sok. Zitas D1-15 Kozyatagi, Istanbul 81090 +90 (532) 573 7783 (cell) +90 (216) 361 5434 (phone) +90 (216) 361 2034 (fax) http://www.eteration.com mailto:nacidai@xxxxxxx mailto:naci@xxxxxxxxxxxxx
Back to the top