Continuous integration/Workflow 2013

This page describes the typical job configuration for MediaWiki-related repositories, as drafted in 2012 and 2013.

It's important to understand what the tools we have are used for and in which order we assess and process their results.

• Linting (php-lint, jshint, ..)
  Static analysis for common mistakes, syntax errors, code quality etc. For example references to APIs not available in old IE, trailing commas that crash IE etc, loose comparison that can lead to side effects etc.
  • *.js: JSHint (with these settings).
  • *.php: php -l (should use lint.php instead for performance), we also run PHP CodeSniffer on some projects.
  • *.css: (TODO) I'm not sure if there is a good CSS linter out there, but we should have something for it. It would also have to support a way to allow used IE-hack workarounds that use invalid CSS.
• Server-side unit testing (PHPUnit)
  Assert server-side modules works as expected on the unit level.
• Client-side unit testing (QUnit)
  Assert client-side modules works as expected on the unit level.
  • ... webkit
    This catches things that would consistently fail in a browser. I.e. someone breaks a JavaScript unit test (regardless of whether it is Firefox, Chrome or IE), this catches it.
  • ... browser matrix (TODO)
    This will distribute the QUnit tests to the browser matrix (through TestSwarm and BrowserStack, or with Grunt and Saucelabs).
• Integration tests (Selenium) (TODO)
  Asserts high-level integration between different components and features that rely on OS interaction (creating an account and logging in, uploading files, drag and drop, editing/saving/reverting/recentchanges etc.). Our browser tests run regularly at https://wmf.ci.cloudbees.com/

Assessment should go from top to bottom. For example we assert that the database is operating properly before we try to request a web page and see if the JavaScript module for querying the API works properly. If tests show the database doesn't work, there is no point in running any of the API tests.

Example

This example is for mediawiki/core.git, configured zuul-config/layout.yaml

Whenever a change is pushed for review, or approved (and needs to be merged), Zuul initiates the "test" pipeline (or "gate-and-submit" pipeline, which is the same for most projects):

• Zuul gets notified and starts the pipeline for project "mediawiki/core"
• Build steps:
  • merge: Cherry picks change on to local copy of latest master
  • jshint: JSHint (with these settings)
  • phplint: Executes php -l on each changed file that is php-like
  • phpunit: Various different groups of phpunit executed (databaseless and on sqlite)
  • qunit: Running javascript unit tests in PhantomJS
• jenkins-bot leaves review on Gerrit.