Saltar al contenido

Visual Validation with Selenium

By Visual Validation we mean the comparison of the rendered user interface to a given baseline. In its most simplistic form it is a pixel-by-pixel comparison of screenshots. If any deviation is detected, the test will fail. One way to do this kind of automated testing is by utilizing a tool called Ocular:

Ocular is a simple utility which helps us to add the visual validation feature into the existing WebDriver test automation frameworks. Ocular uses Arquillian RushEye for the actual image comparison.

Project setup

We are going to create a test that checks Amazon's home page for differences. We will use:

  • Eclipse
  • Maven
  • jUnit
  • Selenium
  • Ocular

We create a new Maven project in Eclipse and define the pom.xml with the dependencies needed:

<project  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.orienteed.qa.visual</groupId>
  <artifactId>VisualValidation</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>VisualValidation</name>
  <url>https://www.orienteed.com</url>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>com.testautomationguru.ocular</groupId>
<artifactId>ocular</artifactId>
<version>1.0.0.Alpha</version>
</dependency>
  </dependencies>
</project>

The structure of the test code is simple and contained within a single class. We use jUnit annotations to execute methods that will configure Ocular and the WebDriver used to run the browser. In this example, we execute the test in Chrome, so we need a ChromeDriver. The ChromeDriver works as a bridge between Selenium and the Chrome browser and it can be found hereThe location of the ChromeDriver executable needs to be specified in a system variable. We also need to configure Ocular. We set the location of the baseline images and the results and set it to create a new baseline image, if a baseline does not exists beforehand. Furthermore, Ocular can be instructed to ignore certain parts of the web page. This can be useful where dynamic content is loaded. In this example we will mask out the logo of Amazon (top right corner) of the screen.

package com.orienteed.qa.visual.VisualValidation;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.Dimension;
import org.openqa.selenium.Point;
import org.openqa.selenium.UnexpectedAlertBehaviour;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.CapabilityType;
import com.testautomationguru.ocular.Ocular;
import com.testautomationguru.ocular.comparator.OcularResult;
import com.testautomationguru.ocular.sample.SampleBuilder;
public class VisualValidation {
private final static String OCULAR_SNAPSHOTS = "src/test/resources/blog/snapshots";
private final static String OCULAR_RESULTS = "src/test/resources/blog/results";
private final static String URL = "https://www.amazon.com";
private final static String FIRST_TEST = "FirstTest";
private final static String LOGO = "nav-logo";
private static WebDriver driver;

@BeforeClass
public static void setup() {
        Path ocularSnapshotsPath = Paths.get(".", OCULAR_SNAPSHOTS);
        File ocularSnapshotsDir = new File(ocularSnapshotsPath.toString());
        if(!ocularSnapshotsDir.exists()) {
        ocularSnapshotsDir.mkdirs();
        }
        Path ocularResultsPath = Paths.get(".", OCULAR_RESULTS);
        File ocularResultsDir = new File(ocularResultsPath.toString());
        if(!ocularResultsDir.exists()) {
        ocularResultsDir.mkdirs();
        }
Ocular.config()
.resultPath(ocularResultsPath)
.snapshotPath(ocularSnapshotsPath)
.globalSimilarity(50)
.saveSnapshot(true);
}
@Before
public void setupDriver() {
ChromeOptions options = new ChromeOptions();
options.addArguments("--incognito", "--disable-dev-shm-usage");
options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);
System.setProperty("webdriver.chrome.driver", "bin/chromedriver_win32/chromedriver.exe");
driver = new ChromeDriver(options);
driver.manage().window().setPosition(new Point(0, 0));
driver.manage().window().setSize(new Dimension(1400, 1000));
}
public OcularResult compare(String snapshotName, WebElement... elements) {
Path snapshotPath = Paths.get(snapshotName + ".png");
SampleBuilder builder = Ocular.snapshot().from(snapshotPath)
                     .sample().using(driver);
for (WebElement element : elements) {
if (null != element) {
builder = builder.excluding(element);
}
}
return builder.compare();
    }
@Test
public void runFirstTest() {
driver.get(URL);
String snapshotName = FIRST_TEST;
WebElement logo = driver.findElement(By.id(LOGO));
OcularResult result = compare(snapshotName, logo);
System.out.println(snapshotName + " " + result);
Assert.assertTrue(result.isEqualsImages());
} 
@After
public void cleanUp() {
driver.close();
}
@AfterClass
public static void tearDown() throws Exception {
driver.quit();
}
}

Results

In order to show the functionality of the test, we have created a baseline image by running the test once and having Ocular generate a screenshot. Then we have edited the screenshot to create artificial differences that will be highlighted in the second execution. The following is the edited baseline screenshot along with the results from the second execution:

FirstTest.pngSnapshot of the user interface of Amazon.com, used as a baseline. We have added certain elements to the page.FirstTest-1.png

Results of the comparison between the baseline and the current screenshot. Notice differences highlighted in red. Notice also that the Amazon logo has been masked out.

Ocular generates a result image that highlights the differences between the baseline image and the current screenshot. This result is in black and white with the differences highlighted in red. The blue rectangular encompasses all the differences that were discovered. Notice how the logo of Amazon is masked out by a black rectangular shape. An interesting side note is that there is a slight difference in the location of the categories listed at the top of the page. This is a real difference between the two loads of Amazon's front page.

Conclusion

We have seen how to incorporate Visual Validation into a Selenium QA testing environment by using the Ocular tool. This can add great value to your team's test suite and help to catch discrepancies in the user interface that are difficult to discover by using traditional value. We will conclude with a few tips and tricks:

  • How stable is the site to be tested? Highly volatile user interface will prove very difficult to test in this way.
  • When using Ocular ensure that zoom levels of the browser and the operating system are set to 100%. Otherwise you can experience discrepancies when trying to locate specific elements.
  • During automated testing with Selenium it is important to ensure that a site is fully loaded before executing tests. For obvious reasons this is absolutely crucial for Visual Validation.
  • Find each dynamic element (orderID, date, marketing spot, ...) and mask it, effectively ignoring it when comparing screenshots.
  • Ocular allows for a threshold to be set before failing a test. This can give some flexibility and stability to test execution.
Tags: