Building an infrastructure-as-code pipeline in the cloud

What is a DevOps pipeline?

A CI/CD pipeline, also called a DevOps pipeline, is the set of processes and tools that teams use to develop, test, deploy and manage software. Integrating each of these processes into a unified pipeline makes the software development and delivery process smoother, more reliable and faster in most cases than if each process took place separately.

What is infrastructure as code?

IaC is the use of programming languages to automate processes beyond the application.

Originally, IaC meant to provision and manage infrastructure, hence the terminology. Today it's common to write code to manage processes as well, such as those inside a DevOps pipeline.

DevOps pipelines and IaC: Better together

When you adopt an IaC approach to manage a DevOps pipeline, you write code that describes how the tools and processes inside a DevOps pipeline should work. Then, you automatically enforce the desired state of those tools and processes. Desired state is a description of the correct setup for the pipeline to work.

You don't need to use IaC to set up a DevOps pipeline. You can manage DevOps tools and processes without code. But a code-based approach delivers benefits such as the following:

• Visibility into the DevOps pipeline. Configuration code serves as a single source of truth that tracks the state of processes and configurations.
• Greater consistency. Configurations are enforced using code instead of manual processes.
• Repeatability. Teams can reuse configurations in other DevOps pipelines.

The term GitOps is sometimes used to describe CI/CD managed primarily through code. It refers to Git version control, widely used for repositories of application and infrastructure code.

Why use an IaC pipeline in the cloud?

An IaC approach to DevOps pipeline management is especially useful in the cloud. Cloud-based software development environments can be complex and change easily.

Cloud environments are well-suited to run DevOps pipelines for several reasons. Virtual resources are flexible and mutable, meaning they can be changed as needed. Managed, serverless pipeline services don't require client-operated infrastructure or initial installation. Additionally, cloud services support APIs to connect steps, provide automation features and support third-party automation tools. This setup facilitates unattended triggers and pipeline step execution.

If you host the DevOps pipeline on cloud services, using infrastructure as code to manage that pipeline might help you operate it more efficiently and with less risk. Code-based control of the pipeline offers visibility, consistency and repeatability, as described previously.

Major cloud services provide native tools as well as documentation, interfaces and samples for integration with other IaC and CI/CD tools. Thus, an organization can build modern IT environments with IaC. For example, a DevOps engineer can use HashiCorp Terraform to develop cloud-agnostic resource deployment scripts and apply them to any of the cloud environments. Figure 1 lists DevOps IaC tools in each category across the major cloud services as well as tools used in private cloud, on-premises and cloud-agnostic builds.

What does an IaC pipeline look like?

IaC pipelines can take many forms. As one example, consider an event-driven cloud-native pipeline that uses serverless functions to initiate steps along the pipelines. The key components in this setup include the following:

• A code repository. For applications, microservices, serverless functions and IaC code. IaC descriptions are usually in YAML or Jinja templating or a domain-specific language.
• A build and test cycle. To compile code into an executable, VM or container image, for IaC deployment and configuration steps.
• Release artifact staging and beta testing. To stage and test compiled code, images and configurations without affecting production infrastructure. Staging provides an environment where developers can rapidly iterate to fix bugs and roll back unsuccessful versions.
• A release-to-production strategy. Artifacts that pass beta testing go live. The pipeline should support multiple deployment strategies such as blue/green, canary or rolling releases. Follow deployment best practices to manage live releases.

These components of a DevOps pipeline are familiar in application development. Code is the operative word in infrastructure as code. The goal is reusable, repeatable configurations.

As with application software, IaC means developers compile, integrate and test configurations on preproduction environments. Each step in an IaC process can be triggered by events, such as a code check-in to the repository or successful completion of syntax checks or beta test suites. A tool then automatically performs the next task.

Create an IaC pipeline in the cloud

All the tools required in an infrastructure-as-code pipeline are available as cloud services, which makes it easy to set up wholly in the cloud. Cloud-based tools offer tight integration with cloud infrastructure, database and application services. Basing the IaC pipeline in the cloud has the potential to reduce networking complexity and interfacing with multiple systems.

The process to set up an IaC DevOps pipeline in the cloud varies depending on which DevOps tools you're using and which cloud vendor provides them, or provides the virtual infrastructure on which you host them. But in most cases, they boil down to the following steps.

1. Deploy DevOps tools

Deploy DevOps tools to the cloud environment. You could use a DevOps pipeline service that is incorporated into the cloud platform, such as AWS CodePipeline. Another option is third-party CI/CD tools that work on cloud services, such as Jenkins or CircleCI.

2. Configure a pipeline using IaC

Define desired configurations for the pipeline tools and processes using code. In some cases, you might find that you can't manage every aspect of operations using code. But most modern tooling and processes lend themselves to code-driven configuration and management.

You might wish to write scripts or configure cloud services to automate parts of the pipeline or integrate tools. For example, you can write serverless functions to trigger steps within the pipeline.

3. Store the code

After describing the pipeline's desired state and process automations using code, store the code in a central repository using a platform like Git.

4. Enforce configurations

With the IaC files in place, you're ready to automate operations within a cloud pipeline. In most cases, enforcing the desired configurations is a matter of connecting the tools' configurations and scripts to the actual DevOps environment.

5. Implement changes as code

To change a configuration or process within the pipeline, modify the relevant code within the repository. Enable versioning in the repository to track changes over time and revert automatically to an earlier state if needed.

IaC pipelines: Adaptable to many situations

A chief benefit of DevOps pipelines is flexibility to suit almost any situation. IaC pipelines are no different.

This example from the AWS DevOps blog illustrates the concepts and workflow, which includes the following steps:

1. A developer changes IaC code and commits it to a repository -- AWS CodeCommit in this case, but often Git or a Git service.
2. A workflow manager, such as AWS CodePipeline, or event-driven function detects the commission and triggers a code build. Event-driven functions can be AWS Lambda, Microsoft Azure Functions or another serverless cloud service.
3. The build engine, whether AWS CodeBuild, Terraform from HashiCorp or another IaC tool, instantiates a compute environment and runs the specified build commands.
4. The IaC deployment engine provisions the build environment resources and runs a validation check on the configuration. In the AWS ecosystem, it's CloudFormation, but other options include Terraform, Ansible from Red Hat, Deployment Manager on Google Cloud or a similar product.
5. If the infrastructure configuration is validated, the IaC processes the configuration file and builds the environment.

Editor's note: This article, originally written by Kurt Marko in 2020, has been updated by TechTarget contributor Chris Tozzi.

Kurt Marko was a longtime TechTarget contributor who passed away in January 2022. He was an experienced IT analyst and consultant, a role in which he applied his broad and deep knowledge of enterprise IT architectures.

Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.