Description of Incident

Various GitHub OAuth tokens and secure environment variables that users included in their builds were accidentally exposed via inclusion in build logs on Travis CI. The vulnerability was responsibly disclosed by security researcher Ivan Vyshnevskyi, first to the Google Security team, who in turn disclosed it to Travis CI in a private channel. To our knowledge, no exposed OAuth tokens have been published. Neither Travis CI nor GitHub have been compromised.

Why do people use tokens in their CI Builds?

Many people over the years have included additional GitHub OAuth tokens in their builds for the purpose of interacting with GitHub during a CI build. For example, deploying via GitHub releases has been a popular option. For tasks such as this, GitHub supports scoping OAuth tokens, so that these tokens can be created with the correct level of access. These additional tokens included in builds by the creator for a specific purpose were the ones potentially impacted by this incident - not OAuth tokens used by Travis CI directly.

Whether a build includes secrets (such as OAuth tokens) or not, builds on Travis CI feature a streaming log that enable customers to watch tests, deployments, and other tasks run live in the browser. For our open-source service (travis-ci.org) these build logs are public. Our closed-source service (travis-ci.com) was also impacted, though these build logs are private.

While we take a number of steps to ensure secure environment variables and tokens are obfuscated when displayed in the UI, and have documentation outlining the build configuration we require to ensure this, once a VM is booted and tests are running we have less control over what information utilities or add-ons are able to print to the VM’s standard output.

In this case it was reported that there were issues with the git push command, which was exposing full remote URLs to the stderr on a failed push. These consequently appeared within the build log in the Travis CI UI.

Normally one would expect the utility itself, in this case git, to redact secure information before sending it to stderr or stdout. We learnt however that git versions earlier than 2.9.3 expose full remote URLs on a successful push, and that all versions of git can expose secure credentials when a push fails, regardless and/or depending on different options or flags specified.

In any case, regardless of the utility, the highly-customizable nature of our builds means that any number of them can be installed on VMs as part of the build process. So it was clear the potential for exposure of this nature was very large.

Upon investigating, it became apparent that indeed a significant number of credentials and tokens had been compromised in this way, including, but not limited to, GitHub tokens.

Scope of the problem

Our service has historically processed hundreds of millions of build jobs, of which over 304 million jobs created since April 2011 across both our open-source and closed-source platforms were available. Every build log is archived indefinitely unless deleted by a user. In order to ascertain the full extent of the problem, it was necessary to search through each build config, identify when tokens and/or secure env vars were configured, and then search through each corresponding log archive for any possible disclosures. Due to the sheer number of builds, this process took several days.

We determined that unique GitHub tokens had been exposed as well as a number of other types of tokens. Based on our analysis, it appears that 2833 valid GitHub tokens and 158 GitHub usernames were exposed in public or private build logs. While the total percentage of user tokens affected was relatively low, it was nevertheless a significant number, and had occurred over a long period of time.

How We Fixed the Problem

There were two issues addressed: Deleting exposed tokens from archived build logs. Preventing exposure of this kind from appearing in future build logs.

The first issue was handled by identifying affected logs, then scrubbing any disclosed credentials from them. This took several days to complete across both our .org and .com domains. Travis CI Enterprise Customers are able to remove their logs via the “Remove Log” button in our web UI.

With the second issue, we needed to implement a fix that would operate at runtime within the build, regardless of its components. This was solved initially by updating travis-build to automatically filter secure environment variables from the logs for any new build. This is currently deployed in production on both .org and .com. If you are an Enterprise customer you’ll receive this fix by May 12th. We will notify you with information about how to update your installation.

As this fix runs in Ruby, it results in larger memory overhead. To address this, we are working on an executable binary that will be installed as part of the build-image. Work on this is expected to be complete by the end of the week.

Current Status

Both new builds and archived build logs with this issue will see the string [secure] instead of potentially exposed secure environment variables. Logs on both our open-source .org, and closed-source .com platforms have been updated.

We have been in close communications with GitHub since we began working on this incident, and have shared impacted tokens with GitHub for their records. GitHub will revoke token access for the affected tokens and will contact token owners to notify them. Both Travis CI and GitHub recommend affected owners revoke their own access tokens and create new tokens immediately. To our knowledge, no OAuth tokens have been published by anyone external to Travis CI. Neither Travis CI nor GitHub have been compromised.

We have also been in contact with security teams at many major third-party services that may have been exposed by disclosed identifiable secure environment variables. We will continue working with them throughout the week on consolidating information and addressing the exposure for each service.

In the meantime, if you suspect credentials for a third-party service may have been compromised in your builds, please cycle the credentials immediately. If you are a third-party service and you suspect your users’ tokens may have been exposed through secure environment variables, please contact security@travis-ci.com with questions.

Finally, we have also published a comprehensive guide to security best practices with Travis CI documentation on best practices as a future reference.

Going Forward

We care about build privacy, and will continue to ensure we proactively protect data wherever possible. Our preventive measures to hide potentially disclosed tokens will remain in effect, and will be added to our build-images permanently to help avoid similar issues in the future. It’s important to understand what we are able to control (log streaming, runtime changes), what we cannot control (user scripts, the output of utilities and addons), and the part you can play in securing your own data -- see our documentation on best practices

Internally, we are conducting a more detailed post-mortem/retrospective across our company. This event has acted as a catalyst in our efforts to review our security posture throughout our platform, build environment, and other processes. The positive takeaway is that this has helped us identify and consider various other attack surfaces system-wide, and to come together to discuss, collaborate, and build solutions. We are also including learning documentation with security in mind.

We’d like to thank Ivan Vyshnevskyi for uncovering this issue and working with us to investigate the problem, the security team at Google for reporting it to us, and to GitHub for their support.

Thanks also to the many members of the Travis Team who came together to co-author this post.

For any questions please contact support@travis-ci.com.