Categories Performance Monitoring

Headless Automated Regression Testing for Web Apps

with Selenium, PhantomJS and Jenkins.

Automated testing is a massive time saver, especially when it comes to regression testing. Regression testing is the retesting of the application when a new feature is introduced to verify the existing functionality is still working, as early as possible. To accomplish this, we need to execute a set of test cases, which can be repetitive and take a lot of time. By taking our existing test cases and automating them, we can save all that manual regression testing time. Here at AppNeta, we set up these tests to run on a regular basis ensuring the product remains in working order at all times and freeing up resources to focus on other tasks.

The challenge is, running web application tests generally requires a machine with a browser and graphical interface, something that is not necessarily present on the average Jenkins Node. However, by integrating Phantomjs, a headless WebKit with JavaScript API, we can solve this problem.


Project Setup

First create a Maven pom.xml file in a new project directory which includes all the necessary dependencies to get the project going. This includes Selenium, JUnit and Ghost Driver.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">


Next from a command-line, run:

mvn clean install

to download the dependencies.

Install PhantomJS

PhantomJS is a WebKit-based browser without a GUI, making it “headless”. It can do all the things a normal browser can do but by removing the heavy GUI it runs faster and does not require a graphical interface. PhantomJS is ideal for automated testing as part of a continuous integration system such as Jenkins since it can be installed directly on the Jenkins node and executed from the command-line.

Install PhantomJS from phantomjs.org/download.html and ensure phantomjs is accessible from the command line.

Create a Test

Now we will create a test to run and locate it in src/test/java. Here I have a sample test which navigates to the AppNeta Blog and verifies the search functionality. It searches for this term “Testing”, verifies that at least one result is returned and navigates to the result to verify that the post actually contains the search term. This is handy both for verifying that the search functionality has not regressed and for ensuring that expected content is still there.

import java.util.List;
import junit.framework.Assert;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.Dimension;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.remote.DesiredCapabilities;

public class AppNetaBlogSearchTest {
  public void testSearchReturnsResults() {
    //Create instance of PhantomJS driver
    DesiredCapabilities capabilities = DesiredCapabilities.phantomjs();
    PhantomJSDriver driver = new PhantomJSDriver(capabilities);

    //Navigate to the page
    driver.manage().window().setSize( new Dimension( 1124, 850 ) );

    //Click the Blog link

    //Input the search term into the search box
    String searchTerm = &quot;Testing&quot;;

    //Click the Search button
    driver.findElement(By.cssSelector(&quot;input[>Run the Test</h2>
To run the test, from the command-line type:

mvn test

The test will run using PhantomJS and output the results under target/surefire-reports in a JUnit xml format which is understood by Jenkins.

Integrate with Jenkins CI

We use Jenkins to manage our build process including scheduling builds of the application, automatically deploying the build and kicking off automated tests. Assuming you are already using Jenkins to build and deploy your web application you can extend it to run your automated tests. If you are not currently using Jenkins, you can still set it up and run the job manually or on a scheduled basis.

First, ensure PhantomJS is installed on all Jenkins Nodes that will run the tests. Next, we need to create a new job and set it to run after a build is deployed. To do this:

  1. In Jenkins click New Job
  2. Select Build a free-style software project
  3. Set the job to checkout your project
  4. Select build after other projects are built and choose the appropriate job
  5. Add a Build task which runs the Maven “test” target
  6. Add a Publish JUnit test result report task under Post-build Actions
  7. Set Text Report XMLs to “target/surefire-reports/*.xml”
  8. Save the job
  9. Run the job and view the Test Results


We now have a framework set up to run automated web application tests and report their results on a regular basis. With this we can catch regression issues right away so we will not have these bugs being discovered at the last minute. Already, in the short 1 month lifetime of our automated testing framework, it has managed to catch regression issues in our existing features. In one case, we found a table that look right on the surface, but the sort order was actual wrong!

This framework has already proven valuable under a monthly release system. As we move towards a more continuous delivery style, we expect to see even more benefits!

Jason Sherin: