My website has a test suite with visual regression tests; screenshot different webpages and compare against a set of expected images:
https://github.com/twolfson/twolfson.com/tree/3.40.0/test/perceptual-tests/expected_screenshots
This weekend, I switched it from PhantomJS to node-webkit and am happy with the results:
Benefits
node_modules/ always work (e.g. no process is undefined issues)node_modules/ are automatically resolved via require- Rendering is more accurate (e.g. web fonts load)
Downsides
node-webkit has a slow startup time- Need to define
index.html and package.json to load JS in node-friendly context
- This can be worked around with a script wrapper (e.g. build a node module for it)
Gotchas
- There were concurrency issues, we had to throttle it to 2 screenshot runs at the same time
- Otherwise, we would see white screens sometimes
node-webkit@0.10.5 would occasionally draw an underline for a link when all others didn't have one
- Upon upgrading to
node-webkit@0.11.1, these issues went away
- Need to use
Xvfb to exceed maximum desktop screenshot size (e.g. 2000 height on 1080 tall resolution)
- Need to keep on using Vagrant to match Travis CI renders
Why not atom-shell?
node_modules/ always work (e.g. no process is undefined issues)node_modules/ are automatically resolved via requirenode-webkithas a slow startup time- Need to define
index.htmlandpackage.jsonto load JS innode-friendlycontext- This can be worked around with a script wrapper (e.g. build a node module for it)
Gotchas
- There were concurrency issues, we had to throttle it to 2 screenshot runs at the same time
- Otherwise, we would see white screens sometimes
node-webkit@0.10.5 would occasionally draw an underline for a link when all others didn't have one
- Upon upgrading to
node-webkit@0.11.1, these issues went away
- Need to use
Xvfb to exceed maximum desktop screenshot size (e.g. 2000 height on 1080 tall resolution)
- Need to keep on using Vagrant to match Travis CI renders
Why not atom-shell?
- Otherwise, we would see white screens sometimes
node-webkit@0.10.5 would occasionally draw an underline for a link when all others didn't have one- Upon upgrading to
node-webkit@0.11.1, these issues went away
Xvfb to exceed maximum desktop screenshot size (e.g. 2000 height on 1080 tall resolution)My first attempt was with atom-shell. However, there was an issue with screenshotting in the latest release (0.19.3):
https://github.com/atom/atom-shell/issues/847
The latest version that would capture screenshots (0.17.2) didn't have a mechanism for reading data from a website when node-integration was disabled. This was disabled to verify the website was accurate as possible.
Why not Selenium?
Selenium is would yield the most accurate set of screenshots. However setup, installation, and maintenance are not as easy as node-webkit.
Additionally, I wanted to verify that node-webkit was a viable replacement for future PhantomJS scripts for the benefits listed earlier.
Links
node-webkit screenshot script
- Perceptual diff test suite
node-webkit screenshot script