Jump to content

Selenium/Ruby/Automation discussion

From mediawiki.org

Current setup

[edit]

README.md

--Zeljko.filipin(WMF) (talk) 14:07, 26 October 2012 (UTC)[reply]

Topics to discuss about browser test automation

[edit]

Discussing the possibilities for the first official WMF browser automation project. What we would like to do is to have the existing test for UploadWizard controlled by Jenkins and run headless on a Jenkins host machine. This is a simple framework that would prove out several features we will want for future tests as well.

To do that would require putting some software on the host machine: Xvfb and Firefox to run a headless browser, and RVM for Ruby (RVM at least would run from /home/jenkins/.rvm/bin/rvm, not from say /usr/bin). There is a Jenkins plugin for RVM that Zeljko has used successfully in the past.

Note that if we don't want to pollute the Jenkins host itself with extra software, it should be possible to do this on a Jenkins "slave", whether a VM or a real machine. In fact, hosting headless browser tests on a slave machine may have some distinct advantages. I am thinking for example that it might be possible to have a labs instance be a Jenkins slave, and store sensitive information like username/password for the test in a file on the slave machine itself rather than in source control.

--Cmcmahon(WMF)

  1. I would suggest that we try to run one simple test that drives a browser from Jenkins. Since you have already created some code, we could reuse it. For example, we could run upload wizard tests.
  2. I would make sure the tests run fine on my machine. It would involve adding a few files to the repository, so the test can use them for uploading.
  3. We also need to set up a dummy user at test site, since file upload works only for logged in users. You maybe already have created it
  4. We should ask operations guys how they store usernames and passwords. If we make the test code public, we have to make sure our real credentials are not in the repository.
  5. Maybe we could publish credentials for dummy user. In that case we have to make sure the user has almost no privileges on the site. We should also maybe automatically (every day?) delete all contributions made by the user.
  6. The simplest way to run the tests on Jenkins machine would be to run them using a headless browser. I have recently set it up for another client. We have to make sure Xvfb is installed on the server. We also need a browser, for example Firefox.
  7. The last thing we need on the Jenkins server is Ruby and a few Ruby gems. We can either use Ruby that is already there, or we could install Ruby via RVM.
  8. I would highly recommend using RVM. It makes managing Ruby and Ruby gems really easy. It installs Ruby and Ruby gems just for one user (jenkins). I think it requires build tools to be installed on the machine. (I do not have a Linux machine handy at the moment, so I can not check.)
  9. If we decide to use RVM, there is a Jenkins RVM plugin that makes is easy to install RVM, Ruby and configure gemsets. I have also recently used it on a project for another client and I was really pleasantly surprised how good it works and how simple it is to use.
  10. If RVM is a security risk, we could install it on a Jenkins slave machine that Jenkins master machine would send tests to. I was able to set up master-slave Jenkins machines for a client recently.
  11. When the number of tests grows, to speed up test runs we can use:
  1. a few Jenkins slave machines
  2. use Selenium grid
  1. self hosted
  2. hosted, for example Sauce Labs or TestingBot.
By definition cross-browser testing in e.g. IE creates a non-free software dependency. ;-) I'm very wary of trying to replicate a set of well-configured browser testing VMs like the ones cultivated by companies that make this their core business.--Eloquence (talk) 01:10, 23 October 2012 (UTC)[reply]
I agree with Mark. I suggest using TestSwarm instead. Wikimedia has had an installation in the past, and a new one is coming. It's currently disabled, but the new and improved version is being prepared at http://integration.wmflabs.org/testswarm/ by Krinkle, ^demon, etc. Basically, this is jQuery's distributed browser testing tool. Superm401 - Talk 05:46, 7 December 2012 (UTC)[reply]
TestSwarm is a unit testing tool. From it's readme file: "TestSwarm provides distributed continuous integration testing for JavaScript". We are not focused on unit testing, but integration testing. The tool that we decided to use is Selenium. From it's home page: "Selenium automates browsers. That's it." TestSwarm can not be used instead of Selenium, since they do not solve the same problem. If you have further questions, please let me know. --Zeljko.filipin(WMF) (talk) 11:22, 4 January 2013 (UTC)[reply]
  • would we always need shell access to the jenkins host? or just to install some basic Ruby/Xvfb/Firefox, etc. one time? --Cmcmahon(WMF)

--Zeljko.filipin(WMF) (talk)

Under what circumstances should we target the following test environments and features?

[edit]

Beta labs

nitial test targets:

  • Article Feedback
    • AFTv5 is scheduled to get backend changes soon, we want to make sure the front end remains stable
  • New Pages Feed/Page Curation toolbar
    • NPF/Curation is under active development right now in production. Toolbar could be challenging.
  • UploadWizard (commons)
    • UW has had recent improvements for Wiki Loves Monuments and Mobile

Production

  • post-deploy sanity check is harder than it would seem
  • other areas?

Commons

  • both beta labs and production?

Wikisource

  • ProofreadPage extension is subject to breakage

Architecture questions

Should we use Cucumber? or just RSpec only?

Create a client/server/grid environment

    • id server with Antoine
    • find clients. Sauce Labs? our own VMs?
      • Windows issues

Repositories

Right now I have in mind two separate repositories, possibly interlinked

  • A repo that members of the community can run locally
    • The community should be able to contribute new tests and enhancements to this repo
  • A repo that runs via WMF Jenkins on WMF hardware

Github and/or gerrit

  • Discuss integration, Mobile (and others) are using Github, but we should have gerrit integration

Mobile http://en.m.wikipedia.org/

  • Michelle and Tomasz
Retrieved from "https://www.mediawiki.org/w/index.php?title=Selenium/Ruby/Automation_discussion&oldid=4267374"