Heroku CI

Last updated 21 March 2017

Quick start for public beta users

1. Create or identify a Heroku Team that will own all of the apps in your Pipeline, and contain all the Heroku users you want to use the Pipeline’s features (including Heroku CI).
2. Set up a Heroku Pipeline that includes only apps owned by your Heroku Team.
3. Connect the Pipeline to your private GitHub repository.

4. Go to the new Pipeline tests page, and turn on automatic test runs

5. We recommend that you choose your Heroku Team or Enterprise Organization for Heroku CI billing.

6. Add a test environment to your app.json file:

   1. Add a test-setup script
   2. Add a test script
   3. Specify add-ons that you may wish to use in CI test runs.

7. Push to GitHub (to master, or by creating a Pull Request).

8. Find your tests running in the Pipeline.

You will notice that the GitHub interface also displays the CI results next to a Heroku icon.

Test run behavior

After you enable it, Heroku CI automatically runs tests for every subsequent push to your GitHub repository. Any push to any branch triggers a test run, including a push to master. This means that all GitHub pull requests are automatically tested, along with any merges to master.

Test runs are executed inside an ephemeral Heroku app that is provisioned for the test run. The app is destroyed when the run completes.

Configuration using app.json

Heroku CI is configured by adding a test environment to the app.json file in your repository. A few keys require additional consideration:

Environment variables

If your application or add-ons require environment variables during CI runs, you can specify them in app.json. For example:

{
  "environments": {
    "test": {
      "env": {
        "PHOTON_TORPEDOES": "online"
      }
    }
  }
}

Unlike Heroku Review Apps, environment variables are not inherited from a parent app. For any sensitive or volatile environment variables (like access tokens) that you do not want in your app.json manifest, you can add them to your pipeline’s Heroku CI settings instead. Any environment variables specified in the Config variables will be available to each test run:

Additionally, the following environment variables will be available for each test run:

HEROKU_TEST_RUN_BRANCH: A string representing the branch of the commit under test HEROKU_TEST_RUN_ID: A string uuid representing the unique ID of the test run

Scripts

The scripts section of your app.json manifest identifies test setup tasks and the test command. For typical usage with our officially supported languages, you do not need to specify these. Our buildpacks are able to detect, set up, and run your tests without configuration (please see additional documentation below specific to your language). If you are using a less common or custom test framework or language, you need to tell Heroku CI how to set up and run your tests.

For example, if you didn’t have any tests, but you wanted to at least verify the code meets your standards, you could install and run rubocop like this:

{
  "environments": {
    "test": {
      "scripts": {
        "test-setup": "gem install rubocop",
        "test": "rubocop ."
      }
    }
  }
}

test-setup:

The test-setup script can optionally be used to perform one-time setup tasks, for example:

• Installing test dependencies like code linters, compilers, test runners etc.
• Setting up and seeding databases

test

The test script should be the command you use to execute your tests. It’s likely the same thing you use in your local environment, e.g.: bundle exec rspec, or npm test.

Add-ons

CI apps will use your project’s app.json to determine which add-on services to provision using the ephemeral default plan specified by each partner. These defaults provide configurations that fit well for temporary apps and help mitigate the impact of churn on partners.

{
  "environments": {
    "test": {
      "addons":[
        "heroku-redis:hobby-dev"
      ]
    }
  }
}

Release Phase and Heroku CI

The release phase is ignored during Heroku CI test runs. We suggest placing any scripts that need to run before your tests in your test-setup script.

Migrating applications still using the now-deprecated app-ci.json

We previously recommended that all Heroku CI limited public beta users add an app-ci.json file alongside their app.json file. This file will be retired when Heroku CI is released to General Availability, and its functionality is available through the test environment in app.json.

The app-ci.json file:

• Allowed to users to specify add-ons, scripts, and buildpacks for CI test runs.
• Addressed Review Apps and Heroku Button incompatibilities

These settings are now stored in the test environment, as described above.

If your application has an app-ci.json file, you can migrate to a test environment by moving its contents directly into app.json, under the environments:test key, then deleting app-ci.json. No changes to the contents of app-ci.json are required.

Keep in mind that settings in app-ci.json are not merged with settings in app.json. The settings in app-ci.json are used only if app.json does not define a test environment.

Migration tool

The heroku-ci CLI plugin includes a migration command that you can use to automatically make the change.

$ cd my-app-directory
$ heroku plugins:install heroku-ci
$ heroku ci:migrate-manifest
Updating app.json file... done
Deleting app-ci.json file... done
Please check the contents of your app.json before committing to your repo.
You're all set! 🎉

$ git diff # check you're happy with the changes
$ git commit -am 'Migrate from app-ci.json to app.json'

Headless Chrome for Heroku CI test runs

The (headless) Chrome buildpack is used to enable tests that require Chrome (e.g. Selenium and other UAT technologies).

Instructions

1. Add the headless chrome buildpack to the “test” environments section of app.json "environments": { "test": { "scripts": { "test": "bundle exec rake test" }, "buildpacks": [ { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }, { "url": "heroku/ruby" } ] } }
2. Include the Chrome-requiring technology in your codebase (e.g. Selenium, Testem)
3. Done. (You can now include tests that require Chrome).

Temporary workaround for “connection refused” error with certain add-ons.

See our gist here for details.

Running browser tests using Phantom.js

This third-party Phantom.js buildpack can be used to run tests that are referenced in the scripts section of your app.json file.

The buildpack should be included in the buildpacks section of your app.json file’s test environment.

Currently supported languages

Go

Go on Heroku has full support for Heroku CI.

Compiles with detected tool (e.g. govendor, godep); Runs tests with go test ./….

Example app.json:

{}

Elixir

Elixir on Heroku has full support for Heroku CI.

Compiles with MIX_ENV=test. Note that as Elixir is not an officially supported language you must still specify the buildpack in both your app.json.

Example:

{
  "buildpacks": [
    {"url": "https://github.com/HashNuke/heroku-buildpack-elixir"}
  ]
}

Node

The Node buildpack has full support for Heroku CI. It compiles your tests with NPM_CONFIG_PRODUCTION=false and YARN_PRODUCTION=false (which installs your test and dev dependencies), then run your tests with npm test or yarn test as appropriate.

For typical Node apps, this should mean you do not need to define test scripts in app.json.

Example:

{}

Apex, Lightning (Salesforce.com development)

Supported only if you’re already in the Salesforce DX Developer preview.

PHP

PHP on Heroku has full support for CI.

Dependencies from require-dev in composer.json will be installed for test runs.

Example app.json:

{
  "environments": {
    "test": {
      "scripts": {
        "test": "phpunit"
      }
    }
  }
}

Ruby

Heroku CI supports Ruby with one caveat: we do not yet support capybara-webkit, but we will before the Heroku CI GA (General availability) release. While we do currently support alternative technologies for UAT (user acceptance testing), such as Phantom.js, we recognize that Capybara is the de-facto standard for the Rails community and are working to add support as soon as possible.

The Ruby buildpack will compile for testing by installing your development and test dependencies, then run your tests with bundle exec rake. For typical Ruby apps, this should mean you do not need to define scripts in app.json.

Example (for a Rails app):

{
  "environments": {
    "test": {
      "addons":[
         "heroku-redis",
         "heroku-postgresql"
      ]
    }
  }
}

Python

Python applications support explicit test setup, as shown here. All dependencies specified within requirements.txt will be installed, and if a requirements-test.txt file is present, those dependencies will be installed as well.

In your app.json file you’ll need to specify tests and setups, e.g.:

{
  "environments": {
    "test": {
      "scripts": {
        "test": "nose test"
      }
    }
  }
}

The test script in app.json can be any shell command, including the path to an executable that exists within your application repo (e.g. make tests).

Languages to be supported soon

Java/Scala

Will be supported when we have buildpack support, probably in early December.

Test run lifecycle

The following section contains detail on the internal and user-facing sequence of events executed to accomplish CI test runs. This section is for use in understanding the internal function of Heroku CI, and will be of help to those who are integrating Heroku CI with their devtools, or building Heroku CI support into third-party buildpacks.

Sequence of events in a Heroku CI test run:

1. A new CI app is provisioned

• The CI app uses Performance-M dyno by default
• The CI has no user-facing UI (it does not appear in the Heroku Pipelines page)
• future: The CI app will be of stage:test

2. Add-ons are provisioned to the CI app

• Add-ons are specified in your app.json
• (future: Add-on vendors can block provisioning for “temporary deployments” like CI apps)

3. Buildpacks are invoked

• If buildpacks are specified in app-json, they are invoked in that order.
• If buildpacks are not specified, we attempt to detect the buildpack among our officially supported buildpacks.
• If a buildpack supports bin/test-compile , it will be run, otherwise, bin/compile
• If the buildpack exits with a non-zero exit code, the test run will be flagged as errored.

4. test-setup is run

• If app.json contains a scripts.test-setup in either the base manifest or the test environment, it is run in the $HOME directory
• Use this to install any dependencies not installed by buildpacks that are needed for your test.
• If this script exits with a non-zero exit code, the test run will be flagged as errored

5. Tests are run

• If app.json contains a scripts.test in either the base manifest or the test environment, it is run in the $HOME directory.
• Otherwise, Heroku CI will attempt to run each buildpack’s bin/test script
• If any of these scripts exit with a non-zero status, the run is flagged as a failure
• If all these scripts exit with a zero status, the run is flagged as a success
• If the buildpack or test scripts emit TAP formatted output to STDOUT, individual passes and failures will be counted and displayed in the test run UI.

6. Results are reported

• Buildpack streams compilation, test-setup, test output, presented formatted in Pipelines UI (Tests tab)
• If buildpack or test script does not support TAP output, CI UI present red/green for entire test suite (script)
• Results are stored at a permanent (Dashboard) URL
• (future: truly streaming output)

7. Cleanup

• Any test runs that have run over 1 hour are terminated
• The CI app (along with any add-ons) is deleted

Debugging

The heroku-ci CLI plugin includes a debug command which can be used to start a debug test run. This allows for inspection of the Heroku CI environment and execution of tests inside a test dyno. This is handy for solving issues where tests pass locally but not on CI or for debugging test setup problems.

This command should be run from within your code repository. It will build a new test run and execute the test setup phase.

$ cd my-repository
$ heroku plugins:install heroku-ci
$ heroku ci:debug
Preparing source... done
Creating test run... done
Running setup and attaching to test dyno...

~ $ npm test # or whatever test command your application uses

To skip the test setup phase (for debugging setup issues), run with the --no-setup flag:

$ heroku ci:debug --no-setup
Preparing source... done
Creating test run... done
Attaching to test dyno...
 ▸    Skipping test setup phase.
 ▸    Run `sprettur setup && for f in .profile.d/*; do source $f; done`
 ▸    to execute a build and configure the environment
~ $

Costs

Costs for Heroku CI follow the same cost model as outlined on our pricing page, with a few important exceptions:

Dynos

All tests run on Performance-M dynos and will be billed at that rate, prorated to the second. Note that this run time will include both test-setup and test phases. For example, a 5 minute test run would incur $0.03 in dyno costs.

$250/month * 5 minutes / 43200 minutes/month = $0.03

Add-ons

When a test run is created, any add-ons listed in app.json’s test environment (or in the base manifest, if the test environment is either absent or omits the addons key) will be provisioned with the specified plan. Once the test run has been reported as failed, erred, or succeeded, the test run’s add-ons will be de-provisioned and destroyed. This means that any paid add-ons will be billed at the plan’s rate, prorated to the second for the duration of the test run. Free add-on plans remain free on Heroku CI.

Billing

On the settings page in your Pipelines interface, under “Configure Heroku CI” you can select which party will be billed for CI runs.

If your Pipeline is comprised only of resources owned by a Heroku Teams or Heroku Enterprise Organization, you will have only the option to bill that Team or Organization. If you have Personal apps in your Pipeline, you will be offered the option to bill CI runs to your personal account.

Dynos used for CI test runs will be billed at the performance-M dyno rate, only for the actual duration of the test run. When released to General Availability (GA), each CI-enabled Pipeline will incur an additional nominal monthly charge.

Feedback

We’re eager to hear about your experience (and help) with Heroku CI, so please send comments and questions to heroku-ci-feedback@heroku.com .