Secure Pipeline Secret Management Practices

In our coding journeys, we’ve all made mistakes such as uploading sensitive information to the cloud. Whether it’s GitHub, Azure DevOps, or another platform, many of us have encountered or might face this issue. Below you find a breakdown of the options for both Azure DevOps as well as GitHub to minimize the risk that something will go wrong.

Prevention

Preventing the exposure of sensitive information is essential for maintaining your environment’s integrity. While not everyone has malicious intent, a continually leaked secret could be misused, posing substantial risks. Each cloud provider and CICD tool has its own best practices for handling secrets during infrastructure deployment via pipelines. This section focuses on Infrastructure as Code rather than CLI and environment variables. For more on environment variables and other methods, stay tuned!

Azure DevOps

In Azure DevOps, variable groups are used to manage variables centrally for use in multiple pipelines whereas “pipeline-based” variables are singular and updated on a single pipeline level. Both come with one big benefit:

They both allow the values of variables to be concealed or masked, these are so-called “secret” or “hidden” variables. This allows for parsing the value through without having it visible to your team(s) in plain text within the variable group or pipeline’s variable section respectively. Since it is not shown, if you would have to rotate it, you should replace/update the value and hit “save”.

You can also use Secure Files in Azure DevOps to store sensitive files securely, authorizing their use in selected pipelines. In practice, this is often replaced by using a blob storage solution and securing this with a central solution like Azure Key Vault, but it’s nice to have the option to choose from.

Finally, for enhanced security, for some use cases, you might want to consider the integration of Azure Key Vault to store and fetch secrets securely within your pipeline but this just adds value if you use Azure Resouces. Otherwise (in terms of AWS) you could consider AWS Secrets Manager which could offer the same… Examples could include default VM configuration user details or a default SQL db user credential…

Feature	What It Does	Key Details	When You’d Use It
Variable Groups	Lets you manage variables that multiple pipelines use, all in one place.	Set up at the project level so all your pipelines can use the same variables. You can keep values secret, meaning they’re hidden from team members. Easy to update for just all pipelines at once, when you need to change settings or rotate secrets.	When you want to ensure that all your pipelines are working with the same settings, like API keys, and you want to keep sensitive information secure.
Pipeline Variables	Gives each pipeline, a set of variables (values) to work with.	Handled at the level of each pipeline. You can keep values secret, meaning they’re hidden from team members. Easy to update for just 1 pipeline when you need to change settings or rotate secrets.	Perfect for when you need a pipeline to work with specific values, or when you need to keep things like passwords or tokens private and securely managed.
Secure Files	Securely keeps important files safe. Alike certificates, so that only certain pipelines can use them.	The files are stored securely and you decide which pipelines can access them. Alternatives might include more complex setups like blob storage with KV integration.	When you have critical files that only certain parts of your pipeline should use, like when you need to sign something or use a special configuration.
Key Vault Integration	Helps keep your secrets, like passwords or API credentials, safe and easy to update—especially useful with Azure services.	Pulls in secrets from Azure Key Vault to your pipeline, supporting regular updates and secure access. More valuable when working with Azure resources.	Ideal when your work involves Azure and you need to ensure everything from VM details to database credentials are safely managed and regularly updated.

GitHub Actions

In GitHub Actions, security is a top priority when managing sensitive information. Much like Azure DevOps, GitHub allows the use of encrypted secrets, but they’re defined at different levels such as the repository or organization. These secrets are crucial for safely storing sensitive data, like API keys and passwords, and are only accessed within workflows, ensuring controlled and secure usage.

For enhanced security, GitHub offers the OpenID Connect (OIDC) provider for authentication. This allows workflows to interact with cloud services using temporary, verified tokens, which significantly reduces the reliance on long-lived static credentials, thereby enhancing security.

Moreover, GitHub provides a ::add-mask:: command that is used to prevent sensitive outputs from appearing in logs. This ensures that even if something goes wrong, sensitive data is protected and not displayed in plain text for team members or anyone else to see.

In practice, these features collectively ensure that your team can run builds, tests, and deployments confidently, knowing that sensitive information remains shielded from unauthorized access or accidental exposure. With the careful use of encrypted secrets, token-based authentication, and masking techniques, GitHub Actions allows developers to maintain the integrity and confidentiality of their projects.

Feature	Purpose and Description	Key Characteristics	Typical Use Cases
GitHub Encrypted Secrets	A safe place to store sensitive data like API keys and passwords, keeping them secure during workflows.	Automatically encrypted and hidden after creation. Accessible securely by workflows. Stored at a repository level.	Securely managing sensitive configuration data for workflows.
GitHub Action Environments	Special zones in your project with rules for deployment, like requiring approval to ensure everything runs smoothly and securely.	Set up rules and approvals for workflows. Scoped to specific repositories or branches. Monitor deployments easily.	Managing deployment checks and approvals before workflow execution in critical areas.
GitHub OIDC for Authentication	Allows secure communication with cloud services by using temporary, verified access tokens instead of storing long-lived credentials.	Uses OpenID Connect tokens for secure access. No need for manual credential management. Trust-based automation.	Federated authentication without managing credentials manually.
GitHub Workflow Masking	Protects sensitive information from appearing in workflow logs, keeping data safe during executions.	Automatically hides secrets in logs. Ensures sensitive data remains private. Only defined information is masked.	Preventing sensitive data exposure in logs during operations.

Mitigation

When a secret is uploaded to a repository on a feature branch, even if there is a PR present (But not yet completed). There is a set of Git commands that allow the removal of commits from visibility when executed in a specific order…

To start with this, select the branch you want and run commands tailored to your needs. For example, git reset --hard HEAD~2 to revert to two commits ago. Push changes using git push --force .

Be aware that branch policies might restrict changes. In such cases, consider reversing changes via a PR and getting assistance from Azure DevOps Support or GitHub Support respectively to remove both PRs if needed.

Finally, if a PR has been completed, you can remove the commit from the “main” branch, however, they might still show up in the PRs.
If you are using GitHub, you’re in luck! You can reach out to GitHub Support to remove the PRs in question
When using Azure DevOps, it is currently not possible to get this done. The easiest way to get rid of your exposed secrets there would be rotating your secrets at that point. You can do this by creating a new commit (via PR) on the “main” branch. This ensures alignment with their best practices.

CloudTechWorks