Compatibility Screenshots

I’ve been trying to learn more about how screenshots can help us identify compatibility issues in Firefox. It started with the question:

How does Firefox compare to Chrome in the top 100 websites?

Pretty good it turns out, on the front pages at least, you can view them yourself [Some images are offensive and NSFW]. You can also check out the same list of sites but comparing Firefox to Firefox with tracking protection. I made some scripts to capture the screens in OSX. They make use of the screencapture utility and this other cool little utility called GetWindowID. GetWindowID determines which Window ID is associated to a program on the screen, Firefox or Chrome in this case.

Let’s look at how these utilities work together.

Running the GetWindowID command requires that we specify which program we are looking for and which tab is active as well. I’ve made sure that my version of Firefox starts up with the Mozilla Firefox Start Page. If we execute this command:

./GetWindowID "Firefox" "Mozilla Firefox Start Page";

It returns a numeric value like:
1072

This is great because the screencapture utility needs to know which window ID to look at.
So let’s take that same GetWindowID command from earlier and store the result into a variable called ‘gcwindow’.

gcwindow=$(./GetWindowID "Firefox" "Mozilla Firefox Start Page");

Now gcwindow has the value 1072 from before. Let’s feed that into the screencapture utility:

screencapture -t jpg -T 40 -l $gcwindow -x ~/Desktop/screens/firefoxtest/$site.jpg;

When this runs the program will wait 40 seconds from the "-T 40” parameter then take a screenshot of Window ID 1072, which is our Firefox instance. The JPG file will be stored in a folder on my desktop under screens/firefoxtest. The rest of the script is looping through each website name that we’ve entered, opening a new browser window, opening the website we want to capture, killing the browser process after each screenshot and some sleep commands in between that give the computer time to execute each step.

There are some browser preferences and considerations that you will want to be aware of before running these scripts.

Why do all this in OSX? Cause I like to work on a mac, I guess. OK I don’t have a good reason but if you want to make it work on a Linux docker or something cool that’d be super sweet. The other thing to keep in mind is I’m looking at viewport screenshots right now, full page would be nice, but we’ll get there.

So the side by side comparison of popular sites is pretty useful but looking at things is a lot of work. It would be cool if we could automate some or all of that looking, right? Luckily there are image comparison tools that can help with this. I decided to try out Yahoo’s blink-diff tool which is built using node.js.

First off only PNG’s are supported with this tool, but that’s easy to change using the screencapture command line tool.

So we use 'screencapture -t png’ instead of 'screencapture -t jpg’.

Let’s go through setting this up for a single test. You’ll need to have node.js installed first.
We need to create a new folder, the name isn’t important.

mkdir onetime-diff

Then download this javascript file from Github and put it in that folder. Now let’s initialize our project:

npm init

And just accept all the defaults. Next let’s install the dependancies:

npm install blink-diff
npm install pngjs-image

Great, it’s ready to run now. The index.js file we downloaded looks for two files in the same folder called firefox.png and chrome.png and will generate a file called output.png. If you need a couple files to test with:

firefox.png
chrome.png

Note that if you provide your own PNG files, you may need to adjust the cropping parameters. I’ve configured the script to work best for Firefox and Chrome screenshots captured on a retina display, if you aren’t using a retina display divide those numbers by 2. You can see here y:160 and y:144, this is cropping out the top portion of the screenshot where the browser's “chrome” is.

cropImageA: { x:0, y:160, width:0, height:0 }, // Firefox
cropImageB: { x:0, y:144, width:0, height:0 }, // Chrome

Once you’re ready to run the test, execute:

node index.js

After a minute, it should generate an output.png file that looks like this and the script will return a result to the command prompt:

Passed
Found 1116908 differences.

So this is a good start, we have an image comparison program and an automated screenshot utility. To make it more useful I created another script that combines these together. On a high level it works like this:

First site > Screenshot Firefox > Screenshot Chrome > Compare images in background process > Next Site...

It has the same dependancies as before, but now we run it like this:

./start.sh

After giving this is a few runs and playing with the settings, I started to see some issues.

  • Advertisements placed in different positions, sizes, style or even amount
  • Regional site redirects
  • Different home page, providing a ‘fresh look’ or they are A/B testing
  • Site surveys or other pop ups
  • Large image sliders
  • Random overlay pop up ads
  • Rotating background images
  • Very slow process when using one computer

We want each site to have a decent amount of time to load, I normally use between 30-40 seconds. But that adds up over 1000 or more sites. I decided to hack something basic together to allow multiple computers in my house to split the load. It helps but it would be much better to have this running on Linux virtual machines or dockers.

So what’s next?

  • More sample runs to find a decent set of parameters for the baseline
  • Identifying in the top 1000 sites, which ones will continue to fail
  • Can we set higher thresholds and still detect when something breaks?
  • Can the tool ignore areas that are constantly changing?
  • Get the results out in the open for others to look at

If any of this interests you and want to get involved, I’d love to hear from you. Or if you have advice on how to make this better, please reach out as well.

Published on