Appium Image Recognition using Java

We are going to identify the graphics content of a mobile app using Appium image recognition library.

We are using an image element (a base64-encoded image file) as a template to compare it with the image captured from a device screen.

Let’s see how this approach works.

Prerequisites

Appium, Java, Nodejs, Android studio, IDE

Get Started

Install the latest version of Appium

npm i -g appium

npm i -g opencv4nodejs

Install OpenCV image comparison library

npm i -g opencv4nodejs

Detailed installation guide for OpenCV library here

Methods Used:

Get Mukul Bhatnagar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

1. Locate images

public MobileElement getAllImage(AppiumDriver<MobileElement> driver, int imgCount) throws Exception {

By img = By.xpath(“//android.widget.Image”);

waitForElementVisibility(driver, img);

MobileElement currentImg = null;

List<MobileElement> imgs = driver.findElements(img);

for (int i = 0; i < imgs.size(); i++) {

if (imgCount == i) {

currentImg = imgs.get(i);

}

}

return currentImg;

}

2. Capture images from app to compare

public File doVisualCheckAllImages(AppiumDriver<MobileElement> driver, int imgCount) throws Exception {

String baselineFilename = VALIDATION_PATH + “/screenshots/app/img” + imgCount + “.png”;

File baselineImg = new File(baselineFilename);

File newBaseline = driver.getScreenshotAs(OutputType.FILE);

BufferedImage fullImg = ImageIO.read(newBaseline);

// Get the location of element on the page to capture screenshot

Point point = getAllImage(driver, imgCount).getLocation();

// Get width and height of the element to capture screenshot

int eleWidth = getAllImage(driver, imgCount).getSize().getWidth();

int eleHeight = getAllImage(driver, imgCount).getSize().getHeight();

// Crop the entire page screenshot to get only element screenshot

BufferedImage eleScreenshot = fullImg.getSubimage(point.getX(), point.getY(), eleWidth, eleHeight);

ImageIO.write(eleScreenshot, “png”, newBaseline);

FileUtils.copyFile(newBaseline, new File(baselineFilename));

return baselineImg;

}

3. Compare Result

public void compareResult(AppiumDriver<MobileElement> driver, String checkName, String checkName2) throws Exception {

SimilarityMatchingOptions opts = new SimilarityMatchingOptions();

opts.withEnabledVisualization();

File Img2 = new File(VALIDATION_PATH + “/screenshots/app/” + checkName2 + “.png”);

File Img1 = new File(VALIDATION_PATH + “/screenshots/app/” + checkName + “.png”);

try {

SimilarityMatchingResult res = driver.getImagesSimilarity(Img1, Img2, opts);

// If the similarity is not high enough, consider the check to be failed

if (res.getScore() < MATCH_THRESHOLD) {

File failViz = new File(VALIDATION_PATH + “/FAIL_APP_” + checkName + “.png”);

res.storeVisualization(failViz);

System.err.println(String.format(

“Visual check of ‘%s’ with ‘%s’ failed; similarity match is only %f, and below the threshold of %f. Visualization written to %s.”,

checkName, checkName2, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));

} else

// Otherwise, it passed!

System.out.println(String.format(“Visual check of ‘%s’ with ‘%s’ passed; similarity match is %f”,

checkName, checkName2, res.getScore()));

} catch (

Exception e) {

System.err.println(“Visual check of “ + checkName + “ with “ + checkName2 + “ failed; Both images are expected to have the same size in order to calculate the similarity score.”);

}

}

Test Run:

public void verifyPictures() throws Exception {

app.doVisualCheckAllImages(driver, 0);

app.doVisualCheckAllImages(driver, 1);

app.compareResult(driver, “img0”, “img1”);

}