Quality Assurance/Monitoring results
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
Some tips to productively make sense of Jenkins browser test failure
We run Continuous integration tests on the Beta cluster and on test2.wikipedia.org. The former can be unstable since it updates to latest code every few minutes, and the latter is not running the latest code.
If there is a failure in Jenkins, you need to figure out
- is it a transient failure on beta labs?
- is an intermittent failure?
- is the flaw in the test?
If the answers are "NO" then there's a bug in the master version of production code.
Once you've established the failure isn't transient or intermittent, you should manualy recreate the test scenario on beta labs or test2.wikipedia.org
(If any answer is "YES" it may be worth figuring out why there was a misleading test failure.)
Tips
[edit]Open lots of browser windows!
When a test fails, the failed test run's "Build Artifacts" will have a .png that is a screenshot of the browser at the point of failure. This can be an obvious indication of the problem.
The output under "Test Result" is less helpful than the full console log (click Console log in the left-hand navigation, then click Full Log), the latter has colored output so you can scroll to the red text, and it shows all failures.
Tests may go wrong several steps before the step detecting error. It can help to watch the Saucelabs screencast