Automating Browser Testing the Easy Way

Automating browser tests is certainly possible, but isn’t always as easy as one might like. If you are just using Selenium, you can use Selenium RC to help automate your testing, or check out Sauce Labs for Selenium testing in the cloud. But if you’re using Selenium IDE or would like to automate other frameworks such as YUI Test, finding a general lightweight solution can be more challenging.

Don’t fear however, because if you or your testing framework can satisfy two simple requirements, you don’t need many tools at all. As long as your tests can:

  1. Run automatically when you visit their URL and then
  2. Post the results back to a listener URL

Then any browser on any OS can be your testing platform without modifications.

Posting Test Results

In the case of a Selenium test runner URL, starting the tests automatically is as simple as adding auto=true to the URL arguments. Adding a listener for Selenium results is just as easy with the resultsUrl=http://yourlistener argument, and YUI has the TestReporter object for the same purpose. Almost any testing framework will allow you to easily satisfy these two requirements, and once they’re satisfied all you have to do is start the desired browser from a command-line with the desired URL, and sit back and wait for the test results to POST.

Automating Test Running

Let’s flesh this out a little more. To run your tests, simply execute the browser from a command-line with the test URL as the first argument. All popular browsers will accept a URL as their first argument, so executing commands like firefox or iexplore will launch Google in Firefox and Internet Explorer respectively. Alternatively, if that browser (or browser profile) is set up only for testing, make that URL the home page. If your browser tests require some sort of human interaction to start them, you can write a tiny wrapper page with some javascript to click the right button or submit the appropriate form to get things started. You can now run browser tests from the command-line and have the results reported to any URL.

Capturing Test Results

So how do you capture these results that are posted to the URL you specify? In our case we use a few lines of Python and Twisted to throw up a temporary listener as part of the build process. Here is a simplified but functional version:

from twisted.web2 import server, resource, channel, http, http_headers
from twisted.internet import reactor
import sys, subprocess
def runBrowserTests(URL):"firefox", URL))
def didTestsFail(result):
    # A very simplified test for a YUI test failure.
    return "failure" in result["results"][0]
class HTTPListener(resource.PostableResource):
    def render(self, request):
        self.Result = request.args # Store the POST dictionary.
        # Queue shutting down the listener after we return a response.
        reactor.callLater(1, reactor.stop)
        headers = {'content-type': http_headers.MimeType('text', 'html')}
        return http.Response(200, headers, "Thanks!")
listener = HTTPListener()
site = server.Site(listener)
reactor.callWhenRunning( runBrowserTests("http://autostartingtests?listener=localhost") )
reactor.listenTCP(8080, channel.HTTPFactory(site)) # This blocks until the POST
sys.exit( didTestsFail(listener.Result) )

Now simply send runBrowserTests the desired URL, and implement didTestsFail(results) to appropriately parse the test results, and you’ll have a tiny script which will run browser tests from the command-line and have an exit code corresponding to whether or not your tests passed. The trivial runBrowserTests() function I have supplied here is easy to extend to accept the browser as an argument as well as handle locking so that only one test suite is running at a time per browser/profile. This is the basic approach that we use to automate our Javascript (YUI Test) and Selenium tests.

Improving Reporting Granularity

Like we did, however, you may want more granularity than an overall success or failure. There are a couple ways to accomplish this. If you are using a Continuous Integration system (Hudson, CruiseControl, Buildbot, etc.), then, instead of exiting with an error code, you can send the results to a function which generates a test report (probably XML) that your CI system understands (such as JUnit in the case of Hudson). Alternatively if you are running unit tests such as phpunit tests, you can dynamically generate a phpunit suite based on the results of the run, and include this suite in your test run. This way you can see your UI tests integrated with your traditional unit testing results, just as if they were actual phpunit tests.

In forthcoming posts, I’ll explain how to integrate individual test results from Selenium and YUI Test into your continuous integration system, as well as how to run these tests in browsers on remote machines so they can be started on headless servers.

  • Digg
  • StumbleUpon
  • Facebook
  • Twitter
  • Google Bookmarks
  • DZone
  • HackerNews
  • LinkedIn
  • Reddit