Crossbrowsing testing in the cloud

Crossbrowsing testing in the cloud

¡Hola de nuevo!, this is Oskar from the MERY backend team here, long time no read!

Some time ago we implemented a suite of automated tests using a cloud service called BrowserStack and the concept is interesting enough for writing some documentation about it, let's see how it goes:

The background

Besides the iOS and Android apps, we also have a PC and Smartphone versions of our MERY service, these are the target of the subject of this post. We try, and succeed most of the time, to maintain both site versions working smoothly with no errors: we review each other's pull requests and ensure to keep the tests code coverage as high as possible.

But even if we have every single server code line covered, the combination between platforms and browsers for Desktop and Smartphones is something that always has to be checked manually: get your iPhone, open Safari and go to the website to ensure the design is not displaced, the styles actually does their work and the javascript doesn't break the whole site.

At some point at some meeting someone said it would be great if we can have some way of making this automatic. Somehow.

Choosing a service for testing in the cloud

There are different services out there, some of them, like BrowserShots, allows to take screenshots of your site from different devices so you have a visual proof that your pages are looking as good as expected. We wanted to be able to perform also some operations inside the page (ideally use Capybara with Selenium) and the possibility of ensuring there were no javascript errors reported in the console, so we needed something a little more sophisticated.

After some research, two main services seemed good enough for our purposes:

• BrowserStack
• SauceLabs

Both are excellent tools with very similar functionalities at similar prices, they both also have their own Ruby gems for putting your tests at work with little setup time. We finally chose BrowserStack because it seems their response is faster and also they have a cool Screenshot API we can also use as an addition to our test reports (with SauceLabs you can only take screenshots from inside the tests, BrowserStack has it as a separate API which is pretty interesting for our purpose).

Selenium and Capybara under Rails

This is material for a whole new post itself, I just wanted to briefly explain both before we get our hands dirty:

Selenium

Pretty much is a layer, an interface for communicating with browsers exposing methods and helpers for managing the browser programatically. This mean you can do things like telling the browser to open a page, go to the login form, write the username in the text input, click the Login button and check what happens next.

Technically speaking is written in Java and it exposes a generic interface for us to use while having drivers implementing it for each browser like Chrome, Firefox, IE and even in iOS or Android devices.

Capybara

As Selenium is written in Java, we need a way to use them from Ruby. Luckily for us, Jonas Nicklas wrote a great gem that does exactly what we need: we write our browser instructions in Ruby and Capybara connects and interacts with Selenium.

This is the kind of code we will be able to use:

visit('page_url') # navigate to this URL
click_link('id_of_link') # click element found by DOM ID
click_link('link_text') # click link with certain text
click_button('button_name') # press a button by its name
fill_in('First Name', :with => 'Oskar') # write something in input field
choose('radio_button') # choose radio button
evaluate_script("alert('Hi there')") # execute some javascript

(More examples here)

As BrowserStack provides the interface which is compatible with the Selenium WebDriver, and you can use a gem called selenium-webdriver, you can imagine they will play together nicely. The whole idea is: we write Ruby code that Capybara will translate to Selenium commands which will guide the browsers we choose from Browserstack servers, and that code will test different pages from our MERY website.

Basic setup for local browser testing

Before using drivers from BrowserStack in the cloud, let's try to run a very simple example that will just open our webpage using a Google Chrome driver we will run in our local server, something like this:

Our tests will be written in Gherkin, the English-like language the Cucumber platform uses, as we will communicate via Capybara with BrowserStack by ourselves, we will just borrow their language support for Rspec using a gem called Turnip which is written, surprise, by Jonas Nicklas.

We will execute this standalone, we won't be using Rails, just a plain Ruby project.

Let's start with the basic gem setup:

gem 'rspec'
gem 'config'
gem 'turnip'
gem 'capybara', require: false
gem 'selenium-webdriver'

From Capybara we only need the Rspec module, and the config gem is an easy way of managing environment related settings, pretty much like Rails does.

Let's take a look also at the basic folder structure and files:

.
├── config
│   └──  config.yml
├── spec
│   ├── spec_helper.rb
│   ├── features
│   │   └── page_check.feature
│   └── steps
│       └── page_check.steps
├── .rspec
├── Gemfile
└── Gemfile.lock

This is a pretty standard Rspec setup with the addition of features/steps thanks to Turnip, let's go for some specifics:

- .rspec file:

Whatever you write here will be appended automatically to the rspec command, it is quite handy.

-r turnip/rspec
--color
--format documentation

The first line tells Rspec to load and use Turnip, so it is aware of the .feature test files. The other two settings are just for formatting Rspec output a little nicer.

- The Chrome driver

As we talked before, Selenium is an interface, by itself it does nothing without a driver that actually implements it. For our basic setup, let's download the Chromedriver from here. For the time being, let's move the executable to the root folder of our project. At this point, Selenium will tell to Chromedriver to do some operation, and the driver will launch and guide a Chrome browser with our commands.

- spec_helper.rb file:

Rspec will execute this file automatically, so this is the place where everything should be initialized. Let's start using bundler for requiring all the gems plus the Rspec module for Capybara and loading files from steps folder:

require 'rubygems'
require 'bundler/setup'
require 'capybara/rspec'
Bundler.require

SPEC_ROOT = File.expand_path(File.dirname(__FILE__))
Dir[File.expand_path("steps/**/*.rb",   SPEC_ROOT)].each {|f| load f, true}

These configures the Selenium Chrome driver and tells Capybara to use it:

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, :browser => :chrome)
end

Capybara.configure do |config|
  config.default_driver = :chrome
end

- spec/features/page_check.feature file:

Easy one:

Feature: Pages are shown without errors

  Scenario: Page check
    Given I access the page, there is no problem at all

- spec/steps/page_check_steps.rb file:

The step definition will first visit the page, and then check the page title includes the title is supposed to have so we know the page is loaded (this test is very naive but works for this example).

step "I access the page, there is no problem at all" do
  visit "http://mery.jp"
  expect(page.title.include?("MERY [メリー]")).to be_truthy
end

If we execute rspec from the command line, we will see how a new Chrome window appears and the page is loaded giving us a wonderful:

Pages are shown without errors
  Page check
    Given I access the page, there is no problem at all

Finished in 5.18 seconds (files took 1.02 seconds to load)
1 example, 0 failures

Let's move to the cloud

Our next goal is to execute the same test but in different platforms / browsers from the BrowserStack infrastructure. First thing we need is an account, they have a trial plan which gives you 30 minutes of testing for free. At this point, what we need is the username and the access key which can be found in the Account / Settings section once you are in. These should go into environment variables and never be commited to any repository, but let's just add them to our config.yml file for testing purposes.

Note we also added the server setting that can be found in the Ruby tab of the automate section of BrowserStack, we need that for telling Selenium where to go.

- config.yml file:

username: <some_username>
access_key: <some_acces_key>
server: hub-cloud.browserstack.com/wd/hub

As this is not rails, the config gem will help us to load this configuration easily, as we already have it in our Gemfile , let's edit the spec_helper.rb file and see how much it changed:

Config.load_and_set_settings("config/config.yml")

SPEC_ROOT = File.expand_path(File.dirname(__FILE__))
Dir[File.expand_path("steps/**/*.rb",   SPEC_ROOT)].each {|f| load f, true}

Capybara.register_driver :browserstack do |app|
  Capybara::Selenium::Driver.new(app,
    :browser => :remote,
    :url => "http://#{Settings.username}:#{Settings.access_key}@#{Settings.server}",
    :desired_capabilities => {
      build: "capybara-browserstack",
      browser: "chrome"
    }
  )
end

Capybara.default_driver = :browserstack
Capybara.run_server = false

Now if you execute the rspec command with the automate page open, and everything is working as expected, you will see how it connects to their servers and the new job is incoming:

And if you click on it, you will see a log of the steps and even a recorded video of the whole process!!

Pretty cool, huh? :)

Testing multiple pages at once

Thanks to Gherkin, this can be quite easily done, first let's add a root_url property to our config.yml file:

root_url: http://mery.jp

Then, edit the page_check.feature file adding what Gherkin calls a table:

Feature: Pages are shown without errors

  Scenario: Page check
    Given I access the page, there is no problem at all
      | url               | name                |
      | /                 | Top page            |
      | /?page=2          | Top page (page 2)   |
      | /search?q=adidas  | Search results      |
      | /cosme            | Category            |
      | /tag/8852         | Tag                 |

So, the page_check_steps.rb can read that information:

step "I access the page, there is no problem at all" do |table|
  pages  = table.hashes.map{|c| c["name"]}.join " | "
  table.hashes.each_with_index do |mery_page|
    url = mery_page["url"]
    visit "#{Settings.root_url}#{url}"
    expect(page.title.include?("MERY [メリー]")).to be_truthy
  end
end

(Ideally all these methods should go to some helper file to keep the steps as clean as possible with only Capybara commands)

At this time, all these pages will be visited and title-tested in a BrowserStack Windows/Chrome platform.

Testing in multiple browsers

It gets much better from now. BrowserStack has an impressive browser/platform combination support, all we need to do is pick up the ones and configure the driver properly. This is done by something called "capabilities"; each platform-browser combination needs its own defined and the easiest way is just adding them to our config.yml file:

support_devices: [pc, android, ios]
browsers:
  pc:
    browser: chrome
    browser_version: "49.0"
    os: OS X
    os_version: El Capitan
  android:
    browser: Android Browser
    os_version: 5.0
    os: android
    browserName: android
    platform: ANDROID
    device: Google Nexus 5
    acceptSslCerts: true
  ios:
    browser: Mobile Safari
    os_version: 8.3
    os: ios
    browserName: iPhone
    platform: MAC
    device: iPhone 6
    javascriptEnabled: true

As you can see, settings are specific to each browser, you can find the combinations and more extensive explanations in the documentation page. We just chose these for this example. Also notice we added a support_devices setting as well.

Next step is register these drivers before the tests in spec_helper.rb, right after the require list, this is how the file will look now:

SUPPORT_DEVICES = Settings.support_devices.map &:to_sym
SUPPORT_DEVICES.each do |device|
  caps = Selenium::WebDriver::Remote::Capabilities.new
  remote_caps = Settings.browsers.to_hash[device]
  remote_caps.each do |k,v|
    caps[k.to_s] = v.to_s
  end
  Capybara.register_driver(device) do |app|
    Capybara::Selenium::Driver.new(app, {
        desired_capabilities: caps,
        browser: :remote,
        url: "http://#{Settings.username}:#{Settings.access_key}@#{Settings.server}"
      }
    )
  end
end

Capybara.default_driver = SUPPORT_DEVICES.first
Capybara.run_server     = false

Basically, we are reading the capabilities from the config file and register the new drivers in Capybara, is the same we had before, but now with multiple browsers. As Capybara also needs a default driver, we just pick the first one defined.

The in the page_check_steps.rb file, we loop through the browsers, and then through the pages:

step "I access the page, there is no problem at all" do |table|
  SUPPORT_DEVICES.each do |device|
    Capybara.current_driver = device
    pages  = table.hashes.map{|c| c["name"]}.join " | "
    table.hashes.each_with_index do |mery_page|
      url       = mery_page["url"]
      page_name = mery_page["name"]
      puts "Testing #{device} -> #{page_name} (#{url})"
      visit "#{Settings.root_url}#{url}"
      expect(page.title.include?("MERY [メリー]")).to be_truthy
    end
  end
end

The Rspec log should look like this:

Pages are shown without errors
  Page check
Testing pc -> Top page (/)
Testing pc -> Top page (page 2) (/?page=2)
Testing pc -> Search results (/search?q=adidas)
Testing pc -> Category (/cosme)
Testing pc -> Tag (/tag/8852)
Testing android -> Top page (/)
Testing android -> Top page (page 2) (/?page=2)
Testing android -> Search results (/search?q=adidas)
Testing android -> Category (/cosme)
Testing android -> Tag (/tag/8852)
Testing ios -> Top page (/)
Testing ios -> Top page (page 2) (/?page=2)
Testing ios -> Search results (/search?q=adidas)
Testing ios -> Category (/cosme)
Testing ios -> Tag (/tag/8852)

Checking on BrowserStack we will see the different browsers opening our pages, it works as expected although the simple basic test was really, well, simple and somewhat useless. We should add more complex checkings, for example, we could add some script to our page and call it from our test for checking if Javascript is working properly:

<script>
      window.jsErrors = [];
      window.onerror=function(msg) {
        window.jsErrors.push(msg);
      }
</script>

As Selenium allows us to execute javascript programatically as well, we can do this to know if the page has any errors:

    js_errors = page.driver.evaluate_script("window.jsErrors && window.jsErrors.length > 0")

And of course, we should add very specific tests for the different key flows of our service: registration process, login / logout and such, we may need some time for coding every single step but then we will have the fundamental functionalities covered in the main platforms including OSX / Windows / iOS / Android browsers.

How we ended using it

Finally, allow me to briefly explain how we evolved the concept for making it useful for ourselves.

After writing some serious tests, the first thing we did is adding a connection to one Slack channel for the development team and the tests are executed several times a day, this is how it looks like:

Then, we did investigate how the BrowserStack Screenshots API works and added it to our tests so we get screenshots of each page for each device right after:

Clicking on them will get the full size site screenshots, the API even manages to scroll the page all the way down and send us a nice composited image, this is specially interesting for checking small but very long contents like the Smartphone version of our site (and most sites).

I won't go to implementation details in this post, but basically we send to the API the url of the page we want to test, the browser capabilities and a callback url. As the screenshot takes some time, they will post us the image to the callback url when is ready. In our case we just have a very simple Sinatra server that reads incoming images and sends them to Slack right away. It works smoothly!

Summary

The setup, the idea is good, but there is lot of space for improvement. With this post I tried to implement a basic example without having too many files involved so we don't get lost, but we do actually have helpers and support modules and classes that contributes to have the steps files with as few non-Capybara code as possible. We also handle connection errors or timeouts and group pages in sessions for optimizing the testing time left we have in our BrowserStack plan.

It makes sense also to integrate into deployment process like they explain here.

In summary: this is a pretty good way of having the most-close-to-real automated combination of integration and regression testing ensuring the site works and looks good in every combination of the targeted devices / browsers.

If you give it a try, please let me know your thoughts!