Compatibility/Web Testing

From MozillaWiki
Jump to: navigation, search

Automated Testing for Web applications and Web sites

This was started on July 15, 2013 - 9:00 AM EST on Etherpad

Keep adding stuff

SOME GUIDELINES:

  • Think about testing in a multi-browser context with different capabilities.
  • do not promote your own tools
  • Talk about real testing experience in a business context

Tools

When doing automated testing for Web applications/Web sites, what do you use? (selenium, PhantomJS, etc?) and why?

Your answers:

  • Qunit & Chai for TDD
  • CasperJS: fast execution of functional/integration tests on top of PhantomJS (Webkit) + now compatible with Gecko using SlimerJS
  • Mechanize + RSpec: We used to use Selenium, but reliability sucked.
  • Intern: Uses WebDriver, so works to test all browsers; async support using promises, not callbacks and test interfaces; interfaces for unit testing and integration/function or polling; easy one-command install with npm; built-in code coverage, built-in Grunt support, built-in SauceLabs support; enforces test case modularity; extensible reporteal testing; well-maintained
  • Watai: writing integration tests is one thing, writing them in a reliable & maintainable way is another. WebDriver is still unavoidable unless you want smoke tests (PhantomJS-based runtimes forget the web is more than 1 engine).
  • CasperJS : easy to use and to setup. (I agree; disclaimer: I'm n1k0)
  • PHPUnit: I'm awesome
  • CasperJs: the same, just a simple and powerfull tool
  • PHPUnit or Atoum http://docs.atoum.org/ for unit tests (PHP applications)
  • CasperJS (with PhantomJS or SlimerJS) for functional/UI tests
  • Selenium in some projects. One of its advantage is to allowing to run tests on real browsers. But it is slow and a bit bloatware.
  • Unit testing: Tytrle & Myrtle + Karma-runner / We want to unit test our code ;)
  • Acceptance testing: Selenium + WebDriver.js + Mocha / Because we want to write acceptance tests in JavaScript
  • Performance testing: Telemetry / It's an experiment right now + Jenkins for Continuous Integration
  • pa11y for accessibility testing https://github.com/nature/pa11y
  • Jasmine + Karma: Clean syntax, fast runner
  • Unit testing: Mocha
  • Functional/UI testing: CasperJS (with PhantomJS, and SlimerJS in a near future) + 1
  • Stack:
  • - integration tests: Watai
  • - components tests: Watai
  • - unit tests: Mocha on its own, Konacha when in a RoR app.
  • UI testing: Webdriver (watir-webdriver)
  • RSpec for unit testing for Ruby apps
  • Mocha for unit testing of node.js apps
  • Zombie.js for headless testing
  • Functional: Phantom.js and Zombie.js (disclosure: I maintain the HTML parser it uses), Robot Framework, testling
  • Unit: lime_test.php, tap.js and tape. PHP Makes me sad. PHP :(
  • Geb (gebish.org)
  • Automated testing in the cloud via BrowserStack / Saucelabs
  • Unit Testing: Doesn't really matter which tool you use. I prefer mocha (bdd) + bondjs (probably because I wrote it) + simple assert methods (node assert is fine)
  • Integration Testing: Zombie and CasperJS have too many issues to be usable. (now that's a bold statement. references? Blog post is coming. Keep an eye on the Groupon Engineering Blog.) PhantomJS is OK for faster dev (but also has too many issues to be comfortable), but you really have to run you tests in real browsers. Therefore, Selenium is mostly unavoidable. WebDriver is ok, but existing Node.js bindings aren't that great.
  • QUnit for unit testing, using Grunt to run the QUnit tests through PhantomJS for running tests on continuous integration servers (like Jenkins and CircleCI). Also run QUnit tests on Saucelabs to test every browser that should work with my app/library (this is done automatically through the grunt-saucelabs plugin). Then, I use selenium for UI based testing.
  • Automated testing of Web Application using SilkTest.
  • Mink: Abstract test code from which web driver is used; Supports both headless and non-headless drivers. http://mink.behat.org


Wish List

What are the things you would like to see in automated testing tools that do not exist yet?


Your answers:

  • PhantomJS no headless-IE. We need a way to run tests on headless versions of all browser platformsJs & SlimerJS offer a way of running tests on headless browsers, but the Webkit version of PhantomJS is obsolete, and there
  • Better interop with various IEs.
  • Easier setup for real device testing +1
  • CSS performance regression testing. yell at me when I commit something that regresses our render performance significantly. let me see a chart of key render perf statistics over project history. +2
  • More headless browsers, and those ones as close as possible to real browsers.+1
  • A standard Javascript API for driving multi-page tests.


References

Some anonymized comments from elsewhere

These are direct comments. It doesn't mean they represent a fact, but they are at least feelings, opinions, perceptions of what testing, tools are.

  • "Running integration tests on headless browsers is not acceptable. You have to make your tests on all latest versions of Firefox, Chrome, Safari, Opera et IE"
  • "From memory (a couple of years), selenium is very hard to install"
  • "Easy to install selenium with pip install selenium. no need to download the jar for the server mode."
  • "don't use Selenium™, thank me later".
  • Why not? Selenium Webdriver has been much improved over the last two years. This is a great project.
  • because http://ljouanneau.com/blog/post/2013/07/15/Tada-Here-is-SlimerJS#c7076 (you asked me):
  • «"for me Selenium appears to be complicated to use and too slow."
  • +1, that's pretty much why I've started casperjs actually.
  • Plus, there's something I don't like about the Webdriver spec and the JSON wire protocol: it just abstracts the DOM environment too much for my own taste; when you're a JavaScript developer, you really care about testing *within* a live DOM environment. Relying on remote operations sent through HTTP commands is pretty cool but can hardly handle complex stuff as you would do by accessing and manipulating the DOM directly. Would you prefer using a browser console or curl to debug your js code?  :)
  • But I hear concerns about multiplatform support and standardization, though I'd personally think there would be room for a more-phantomjs-like API standardization effort, or at least a more "close to the metal" one than what the WebDriver spec tries to provide.»
  • I don't see why a BDD setup using Jasmine or Mocha with a decent runner such as Testem or Karma isn't used much more for functional/UI testing of single-page applications in real browsers (see https://github.com/webpro/Automated-SPA-Testing#functional-testing or https://github.com/webpro/baseplate/blob/master/test/mocha-behavior/specs/moduleB.behavior.js for an example). The only real downside I can see so far is not being able to navigate across pages (which is usually not a big issue here).
Retrieved from "https://wiki.mozilla.org/index.php?title=Compatibility/Web_Testing&oldid=703219"