Heroku CI
Last updated 21 March 2017
Table of Contents
- Quick start for public beta users
- Test run behavior
- Configuration using app.json
- Release Phase and Heroku CI
- Migrating applications still using the now-deprecated app-ci.json
- Headless Chrome for Heroku CI test runs
- Temporary workaround for “connection refused” error with certain add-ons.
- Running browser tests using Phantom.js
- Currently supported languages
- Languages to be supported soon
- Test run lifecycle
- Debugging
- Costs
- Billing
- Feedback
Heroku Continuous Integration (CI) is currently available as a public beta. Until release to General Availability (GA), this feature is provided for evaluation purposes without guarantee of functionality or uptime.
Test runs that use Heroku Postgres, Heroku Redis, and/or the Salesforce add-on are limited to the Hobby plan for those add-ons.
Quick start for public beta users
- Create or identify a Heroku Team that will own all of the apps in your Pipeline, and contain all the Heroku users you want to use the Pipeline’s features (including Heroku CI).
- Set up a Heroku Pipeline that includes only apps owned by your Heroku Team.
- Connect the Pipeline to your private GitHub repository.
Go to the new Pipeline tests page, and turn on automatic test runs
We recommend that you choose your Heroku Team or Enterprise Organization for Heroku CI billing.
-
Add a
test
environment to yourapp.json
file:- Add a
test-setup
script - Add a
test
script - Specify add-ons that you may wish to use in CI test runs.
- Add a
Push to GitHub (to master, or by creating a Pull Request).
Find your tests running in the Pipeline.
You will notice that the GitHub interface also displays the CI results next to a Heroku icon.
Test run behavior
After you enable it, Heroku CI automatically runs tests for every subsequent push to your GitHub repository. Any push to any branch triggers a test run, including a push to master. This means that all GitHub pull requests are automatically tested, along with any merges to master.
Test runs are executed inside an ephemeral Heroku app that is provisioned for the test run. The app is destroyed when the run completes.
CI does not run on Pipeline promotions, because no new code is being built.
Configuration using app.json
Heroku CI is configured by adding a test
environment to the app.json
file in your repository. A few keys require additional consideration:
Environment variables
If your application or add-ons require environment variables during CI runs, you can specify them in app.json
. For example:
{ "environments": { "test": { "env": { "PHOTON_TORPEDOES": "online" } } } }
Unlike Heroku Review Apps, environment variables are not inherited from a parent app. For any sensitive or volatile environment variables (like access tokens) that you do not want in your app.json
manifest, you can add them to your pipeline’s Heroku CI settings instead. Any environment variables specified in the Config variables will be available to each test run:
Additionally, the following environment variables will be available for each test run:
HEROKU_TEST_RUN_BRANCH
: A string representing the branch of the commit under test
HEROKU_TEST_RUN_ID
: A string uuid representing the unique ID of the test run
Scripts
The scripts
section of your app.json
manifest identifies test setup tasks and the test command. For typical usage with our officially supported languages, you do not need to specify these. Our buildpacks are able to detect, set up, and run your tests without configuration (please see additional documentation below specific to your language). If you are using a less common or custom test framework or language, you need to tell Heroku CI how to set up and run your tests.
For example, if you didn’t have any tests, but you wanted to at least verify the code meets your standards, you could install and run rubocop like this:
{ "environments": { "test": { "scripts": { "test-setup": "gem install rubocop", "test": "rubocop ." } } } }
test-setup
:
The test-setup script
can optionally be used to perform one-time setup tasks, for example:
- Installing test dependencies like code linters, compilers, test runners etc.
- Setting up and seeding databases
test
The test
script should be the command you use to execute your tests. It’s likely the same thing you use in your local environment, e.g.: bundle exec rspec
, or npm test
.
Add-ons
CI apps will use your project’s app.json to determine which add-on services to provision using the ephemeral default plan specified by each partner. These defaults provide configurations that fit well for temporary apps and help mitigate the impact of churn on partners.
{ "environments": { "test": { "addons":[ "heroku-redis:hobby-dev" ] } } }
Release Phase and Heroku CI
The release phase is ignored during Heroku CI test runs. We suggest placing any scripts that need to run before your tests in your test-setup
script.
Migrating applications still using the now-deprecated app-ci.json
We previously recommended that all Heroku CI limited public beta users add an app-ci.json
file alongside their app.json
file. This file will be retired when Heroku CI is released to General Availability, and its functionality is available through the test
environment in app.json
.
The app-ci.json
file:
- Allowed to users to specify add-ons, scripts, and buildpacks for CI test runs.
- Addressed Review Apps and Heroku Button incompatibilities
These settings are now stored in the test
environment, as described above.
If your application has an app-ci.json
file, you can migrate to a test environment by moving its contents directly into app.json
, under the environments:test
key, then deleting app-ci.json
. No changes to the contents of app-ci.json
are required.
Keep in mind that settings in app-ci.json
are not merged with settings in app.json
. The settings in app-ci.json
are used only if app.json
does not define a test
environment.
Migration tool
The heroku-ci
CLI plugin includes a migration command that you can use to automatically make the change.
$ cd my-app-directory $ heroku plugins:install heroku-ci $ heroku ci:migrate-manifest Updating app.json file... done Deleting app-ci.json file... done Please check the contents of your app.json before committing to your repo. You're all set! 🎉 $ git diff # check you're happy with the changes $ git commit -am 'Migrate from app-ci.json to app.json'
Headless Chrome for Heroku CI test runs
The (headless) Chrome buildpack is used to enable tests that require Chrome (e.g. Selenium and other UAT technologies).
Instructions
- Add the headless chrome buildpack to the “test” environments section of app.json
"environments": { "test": { "scripts": { "test": "bundle exec rake test" }, "buildpacks": [ { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }, { "url": "heroku/ruby" } ] } }
- Include the Chrome-requiring technology in your codebase (e.g. Selenium, Testem)
- Done. (You can now include tests that require Chrome).
The (headless) Chrome buildpack uses the Chrome Canary release channel. You may want to set --remote-debugging-port=9222
to keep Chrome running if you observe unexpected exits after the first page load. Different test frameworks have different ways of specifying browser startup flags, and different interactions with Chrome. We will update this documentation as issues become known.
Temporary workaround for “connection refused” error with certain add-ons.
See our gist here for details.
Running browser tests using Phantom.js
This third-party Phantom.js buildpack can be used to run tests that are referenced in the scripts
section of your app.json
file.
The buildpack should be included in the buildpacks
section of your app.json file’s test
environment.
Currently supported languages
Go
Go on Heroku has full support for Heroku CI.
Compiles with detected tool (e.g. govendor, godep); Runs tests with go test ./…
.
Example app.json:
{}
Elixir
Elixir on Heroku has full support for Heroku CI.
Compiles with MIX_ENV=test
.
Note that as Elixir is not an officially supported language you must still specify the buildpack in both your app.json.
Example:
{ "buildpacks": [ {"url": "https://github.com/HashNuke/heroku-buildpack-elixir"} ] }
Node
The Node buildpack has full support for Heroku CI. It compiles your tests with NPM_CONFIG_PRODUCTION=false
and YARN_PRODUCTION=false
(which installs your test and dev dependencies), then run your tests with npm test
or yarn test
as appropriate.
For typical Node apps, this should mean you do not need to define test scripts in app.json.
Example:
{}
Apex, Lightning (Salesforce.com development)
Supported only if you’re already in the Salesforce DX Developer preview.
PHP
PHP on Heroku has full support for CI.
Dependencies from require-dev
in composer.json
will be installed for test runs.
Example app.json
:
{ "environments": { "test": { "scripts": { "test": "phpunit" } } } }
Make sure your test dependencies, e.g. phpunit/phpunit
, are in your composer.json
require-dev
section, and that composer.lock
is up to date.
Ruby
Heroku CI supports Ruby with one caveat: we do not yet support capybara-webkit, but we will before the Heroku CI GA (General availability) release. While we do currently support alternative technologies for UAT (user acceptance testing), such as Phantom.js, we recognize that Capybara is the de-facto standard for the Rails community and are working to add support as soon as possible.
The Ruby buildpack will compile for testing by installing your development and test dependencies, then run your tests with bundle exec rake
. For typical Ruby apps, this should mean you do not need to define scripts in app.json.
Example (for a Rails app):
{ "environments": { "test": { "addons":[ "heroku-redis", "heroku-postgresql" ] } } }
Python
Python applications support explicit test setup, as shown here. All dependencies specified within requirements.txt
will be installed, and if a requirements-test.txt
file is present, those dependencies will be installed as well.
In your app.json
file you’ll need to specify tests and setups, e.g.:
{ "environments": { "test": { "scripts": { "test": "nose test" } } } }
The test
script in app.json
can be any shell command, including the path to an executable that exists within your application repo (e.g. make tests
).
Languages to be supported soon
Java/Scala
Will be supported when we have buildpack support, probably in early December.
Test run lifecycle
The following section contains detail on the internal and user-facing sequence of events executed to accomplish CI test runs. This section is for use in understanding the internal function of Heroku CI, and will be of help to those who are integrating Heroku CI with their devtools, or building Heroku CI support into third-party buildpacks.
Sequence of events in a Heroku CI test run:
1. A new CI app is provisioned
- The CI app uses Performance-M dyno by default
- The CI has no user-facing UI (it does not appear in the Heroku Pipelines page)
- future: The CI app will be of
stage:
test
2. Add-ons are provisioned to the CI app
- Add-ons are specified in your
app.json
- (future: Add-on vendors can block provisioning for “temporary deployments” like CI apps)
3. Buildpacks are invoked
- If buildpacks are specified in app-json, they are invoked in that order.
- If buildpacks are not specified, we attempt to detect the buildpack among our officially supported buildpacks.
- If a buildpack supports
bin/test-compile
, it will be run, otherwise,bin/compile
- If the buildpack exits with a non-zero exit code, the test run will be flagged as errored.
4. test-setup
is run
- If app.json contains a
scripts.test-setup
in either the base manifest or thetest
environment, it is run in the$HOME
directory - Use this to install any dependencies not installed by buildpacks that are needed for your test.
- If this script exits with a non-zero exit code, the test run will be flagged as errored
5. Tests are run
- If app.json contains a
scripts.test
in either the base manifest or thetest
environment, it is run in the$HOME
directory. - Otherwise, Heroku CI will attempt to run each buildpack’s
bin/test
script - If any of these scripts exit with a non-zero status, the run is flagged as a failure
- If all these scripts exit with a zero status, the run is flagged as a success
- If the buildpack or test scripts emit TAP formatted output to STDOUT, individual passes and failures will be counted and displayed in the test run UI.
6. Results are reported
- Buildpack streams compilation, test-setup, test output, presented formatted in Pipelines UI (Tests tab)
- If buildpack or test script does not support TAP output, CI UI present red/green for entire test suite (script)
- Results are stored at a permanent (Dashboard) URL
- (future: truly streaming output)
7. Cleanup
- Any test runs that have run over 1 hour are terminated
- The CI app (along with any add-ons) is deleted
Debugging
The heroku-ci
CLI plugin includes a debug command which can be used to start a debug test run. This allows for inspection of the Heroku CI environment and execution of tests inside a test dyno. This is handy for solving issues where tests pass locally but not on CI or for debugging test setup problems.
This command should be run from within your code repository. It will build a new test run and execute the test setup phase.
$ cd my-repository $ heroku plugins:install heroku-ci $ heroku ci:debug Preparing source... done Creating test run... done Running setup and attaching to test dyno... ~ $ npm test # or whatever test command your application uses
To skip the test setup phase (for debugging setup issues), run with the --no-setup
flag:
$ heroku ci:debug --no-setup Preparing source... done Creating test run... done Attaching to test dyno... ▸ Skipping test setup phase. ▸ Run `sprettur setup && for f in .profile.d/*; do source $f; done` ▸ to execute a build and configure the environment ~ $
Costs
Costs for Heroku CI follow the same cost model as outlined on our pricing page, with a few important exceptions:
Dynos
All tests run on Performance-M dynos and will be billed at that rate, prorated to the second. Note that this run time will include both test-setup
and test
phases. For example, a 5 minute test run would incur $0.03 in dyno costs.
$250/month * 5 minutes / 43200 minutes/month = $0.03
Add-ons
When a test run is created, any add-ons listed in app.json
’s test
environment (or in the base manifest, if the test
environment is either absent or omits the addons
key) will be provisioned with the specified plan. Once the test run has been reported as failed
, erred
, or succeeded
, the test run’s add-ons will be de-provisioned and destroyed. This means that any paid add-ons will be billed at the plan’s rate, prorated to the second for the duration of the test run. Free add-on plans remain free on Heroku CI.
Billing
On the settings page in your Pipelines interface, under “Configure Heroku CI” you can select which party will be billed for CI runs.
If your Pipeline is comprised only of resources owned by a Heroku Teams or Heroku Enterprise Organization, you will have only the option to bill that Team or Organization. If you have Personal apps in your Pipeline, you will be offered the option to bill CI runs to your personal account.
Dynos used for CI test runs will be billed at the performance-M dyno rate, only for the actual duration of the test run. When released to General Availability (GA), each CI-enabled Pipeline will incur an additional nominal monthly charge.
Feedback
We’re eager to hear about your experience (and help) with Heroku CI, so please send comments and questions to heroku-ci-feedback@heroku.com .