Keeping your applications and infrastructure secure is a significant concern for most organizations. But you need to balance the inertia of ongoing development, maintenance, and deployment against the friction of making sure those tasks are secure and your deployed apps and infrastructure remain secure.
As a streamlined DevOps practice, automated Continuous Integration/Continuous Delivery (CI/CD) pipelines use tools like Jenkins, Concourse, or CircleCI to create a highly configurable and accelerated build, test, and deployment system. Taking DevOps a step further into operations and Infrastructure as Code (IaC) practices, many teams are now using CI/CD pipelines to manage configuration files for building out the infrastructure their applications run on. This means CI/CD tools are also handling change management, testing, and deployment for Docker files, Kubernetes manifests, Helm charts, and more.
While most discussion of CI/CD focuses on application code — avoiding bugs and crashes — the shift to managing IaC configuration with CI/CD tools highlights another potential problem: managing the secrets (Machine Identity or Application account) needed to run your applications and systems, secrets that can end up being exposed, putting your systems at risk.
Automating build, test, and deployment tasks takes much of the need for storing and keeping track of secrets off the shoulders of your developers and administrators — which is a valuable first step — but moves responsibility for those secrets into your CI/CD pipeline and associated tools.
In this blog, we’ll look at some background concepts and best practices for making sure authentication and authorization secrets are used effectively and securely in CI/CD pipelines.
Identity, Authentication, and Authorization
What do we mean by secrets? Secrets are digital credentials that are used to provide identity authentication and authorize access to privileged accounts, applications, and services. They prove the identity of a user or system. Authenticated identities then have authorization to access or use systems based on user or group privileges defined within systems.
Developers need to understand what authentication secrets they need to control, where they are necessary in the CI/CD pipeline, and how they are configured, stored, and managed securely within the pipeline and deployed infrastructure.
For example, users authenticate to have access to a source control system like GitHub to commit changes to a codebase, which kicks off test, build, and deployment tasks in a CI/CD pipeline.
Additional examples of secrets include:
- User or auto-generated passwords
- API, GitHub tokens, and other application keys/credentials
- Hard-coded credentials in containerized applications
- SSH Keys
- Private certificates for secure communication, transmitting and receiving of data (TLS, SSL, and so on)
- Private encryption keys for systems like PGP
- System-to-system passwords
- One-time password devices
The Challenge of Secrets in a CI/CD Pipeline
Automated processes are a critical component of DevOps infrastructure. CI/CD orchestration and configuration tools such as Jenkins, Puppet, Ansible, and Chef are increasingly deployed in DevOps processes to improve processes, facilitate faster deployment of software and product delivery, and provide continuous cost reduction. >
However, CI/CD tools are the biggest consumers of secrets and have access to a lot of sensitive resources such as other apps and services and information like codebases and databases. As the number of secrets grows, it becomes harder to store, transmit, and audit secrets securely.
Furthermore, secrets aren’t just for authentication between tools. Often, secrets need to be provided as part of the build and deployment process so that deployed resources have access. This is particularly important in hybrid cloud and microservices deployments, and with the automated scaling capabilities of tools like Kubernetes.
Best Practices for CI/CD Security
The first steps for securing your team’s CI/CD pipeline include locking down configuration managers, systems that host repositories, and the build servers. The pipeline should be monitored from end to end across the entire toolchain.
The security of secrets needs to apply both during transit and at rest. Best practices include the following:
- Remove hardcoded secrets from source code , and related CI/CD config files.
- Have rigorous security parameters, such as one-time passwords, for secrets regarding more sensitive tools and systems.
- Distribute secrets during runtime
- Use password managers, and rotate passwords after each use.
- Know who has access to what. Whether it’s role-based, time-based, or task-based, there should be a clear repository of access management. Another option to consider is segmenting secrets based on levels of access.
- Machine identity is critical to secure non-human access in containers. Typically, an authenticator certifies that the client runtime container attributes of the requesting container match the native characteristics of the valid container. Once authenticated, the container can access multiple resources based on predefined role-based access control policies. You should destroy containers and VMs after use.
- Ensure secrets are not inadvertently passed on during builds for pull requests via your CI/CD pipelines.
- Deploy the practice of least privilege: Give access only to secrets that are requisite. Besides employee access, least privilege also applies to applications, systems, or connected devices that require privileges or permissions to perform tasks. You should regularly audit levels of access to maintain the level of least privilege.
Securing Your CI/CD Pipeline Next Steps
Security needs to be a top priority in any developer team, especially when considering the key uses of secrets and the specific challenges faced in securing the CI/CD pipeline. We explored some best security practices to ensure that digital authentication credentials are resistant to attacks by malicious agents, and that the privileged accounts and identities remain secure.
About the Author: The DevSecOps Technical Working Group subcommittee was formed in July 2020. The team, led by Saravanan Thiyagarajan, includes Ramnath Krishnamurthi, Rohini Rani-Barik, Carlos Garcia, Eric Lordahl, Max Bareither, Stefan Lesaru, Thani Anandan and Vikram Chandrasekaran.