Comparing CloudFormation, Terraform and Ansible Part #2 • Technodrone

The feedback I received from the first comparison was great – thank you all.

Obviously the example I used was not really something that you would use in the real world – because no-one actually creates a only a VPC – and does not create anything inside it, that is pretty futile.

So let’s go to the next example.

The scenario is to create a VPC, with a public presences and a private presence. This will be deployed across two availability zones. Public subnets should be able to route to the internet through an Internet Gateway, private subnets should be able to access the internet through a NAT Gateway.

This is slightly more complicated than just creating a simple VPC with a one-liner

aws ec2 create-vpc --cidr-block 192.168.90.0/24

So to summarize - the end state I expect to have is:

1x VPC (192.168.90.0/24)
4x Subnets
- 2x Public
  - 192.168.90.0/26 (AZ1)
  - 192.168.90.64/26 (AZ2)
- 2x Private
  - 192.168.90.128/26 (AZ1)
  - 192.168.90.192/26 (AZ2)
1x Internet Gateway
2x NAT Gateway (I really could do with one – but since the subnets and resources are supposed to be deployed in more than a single AZ – there will be two – and here I minimize the risk impact of loss of service if a single AZ fails)
1x Public Route Table
2x Private Route Table (1 for each AZ)

And all of these should have simple tags to identify them.

(The code for all of these scenarios is located here

First lets have a look at CloudFormation

Description:
  This template deploys a VPC, with a pair of public and private subnets spread
  across two Availability Zones. It deploys an Internet Gateway, with a default
  route on the public subnets. It deploys a pair of NAT Gateways (one in each AZ),
  and default routes for them in the private subnets.

  The Availability zone information is hard-coded (on purpose)

Parameters:
  EnvironmentName:
    Description: An environment name that will be prefixed to resource names
    Type: String
    Default: "testvpc"

  VpcCIDR:
    Description: Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 192.168.90.0/24

  PublicSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
    Type: String
    Default: 192.168.90.0/26

  PublicSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone
    Type: String
    Default: 192.168.90.64/26

  PrivateSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
    Type: String
    Default: 192.168.90.128/26

  PrivateSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
    Type: String
    Default: 192.168.90.192/26

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: us-east-2a
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Public Subnet (AZ1)

  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: us-east-2b
      CidrBlock: !Ref PublicSubnet2CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Public Subnet (AZ2)

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: us-east-2a
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: us-east-2b
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

  NatGateway1EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway2EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway1:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway1EIP.AllocationId
      SubnetId: !Ref PublicSubnet1

  NatGateway2:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway2EIP.AllocationId
      SubnetId: !Ref PublicSubnet2

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Public Routes

  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet1

  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet2


  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Routes (AZ1)

  DefaultPrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway1

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1

  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Routes (AZ2)

  DefaultPrivateRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway2

  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      SubnetId: !Ref PrivateSubnet2

So this is a bit more complicated than the previous example. I still used the native resources in CloudFormation, and set defaults for the my parameters. You will see some built in functions that are available in CloudFormation – namely !Ref which is a reference function to lookup a value that has previously been created/defined in the template and !Sub that will substitute a value in the template with an environment variable.

So there are a few nifty things that are going here.

You do not have remember resource names – CloudFormation keeps all the references in check and allows you to address them by name in other places in the template.
CloudFormation manages the order in which the resources are created and takes of care of all of that for – and it will take care of the order what resources are created. For example – the route table for the private subnets will only be created after the NAT gateways have been created.
More importantly – when you tear everything down – then CloudFormation takes care of the ordering for you, i.e. you cannot tear down a VPC – while the NAT gateways and Internet gateway are still there – so you need to delete those first and then you can go ahead and rip the everything else up.

Lets look at Ansible. There are built-in modules for this ec2_vpc_net, ec2_vpc_subnet, ec2_vpc_igw, ec2_vpc_nat_gateway and ec2_vpc_route_table.

---
- name: VPC creation playbook
  hosts: localhost
  connection: local
  gather_facts: no

  vars_files:
    - vars/vpc_vars.yml

  tasks:
  - name: Create a VPC
    ec2_vpc_net:
      region: "{{ region }}"
      name: "{{ project_name }}"
      cidr_block: "{{ cidr_block }}"
      state: present
    register: vpc

  - name: Create subnets for Public network (AZ1)
    ec2_vpc_subnet:
      state: present
      az: "{{ az1 }}"
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      cidr: "{{ pub1_cidr }}"
      resource_tags:
        Name: "{{ pub1_name }}"
    register: pub1_subnet

  - name: Create subnets for Public network (AZ2)
    ec2_vpc_subnet:
      state: present
      az: "{{ az2 }}"
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      cidr: "{{ pub2_cidr }}"
      resource_tags:
        Name: "{{ pub2_name }}"
    register: pub2_subnet

  - name: Create subnets for Private network (AZ1)
    ec2_vpc_subnet:
      state: present
      az: "{{ az1 }}"
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      cidr: "{{ private1_cidr }}"
      resource_tags:
        Name: "{{ priv1_name }}"
    register: priv1_subnet

  - name: Create subnets for Private network (AZ2)
    ec2_vpc_subnet:
      state: present
      az: "{{ az2 }}"
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      cidr: "{{ private2_cidr }}"
      resource_tags:
        Name: "{{ priv2_name }}"
    register: priv2_subnet

  - name: Create a new Internet Gateway
    ec2_vpc_igw:
      state: present
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      tags:
        Name: "{{ project_name }}"
    register: igw

  - name: Create NAT Gateway AZ1
    ec2_vpc_nat_gateway:
      wait: yes
      state: present
      subnet_id: "{{ pub1_subnet.subnet.id }}"
      region: "{{ region }}"
      if_exist_do_not_create: true
    register: ngw_az1

  - name: Create NAT Gateway AZ2
    ec2_vpc_nat_gateway:
      wait: yes
      state: present
      subnet_id: "{{ pub2_subnet.subnet.id }}"
      region: "{{ region }}"
      if_exist_do_not_create: true
    register: ngw_az2

  - name: Create the public routing table and associate to Public subnets
    ec2_vpc_route_table:
      state: present
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      tags:
        Name: Public Route Table
      subnets:
        - "{{ pub1_subnet.subnet.id }}"
        - "{{ pub2_subnet.subnet.id }}"
      routes:
        - dest: 0.0.0.0/0
          gateway_id: "{{ igw.gateway_id }}"
    register: public_route_table

  - name:  Create the routing table for each of the Private Subnet (AZ1)
    ec2_vpc_route_table:
      state: present
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      tags:
        Name: Private Route Table (AZ1)
      subnets: "{{ priv1_subnet.subnet.id }}"
      routes:
        - dest: 0.0.0.0/0
          gateway_id: "{{ ngw_az1.nat_gateway_id }}"
    register: private_route_table_az1

  - name:  Create the routing table for each of the Private Subnet (AZ2)
    ec2_vpc_route_table:
      state: present
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{ region }}"
      tags:
        Name: Private Route Table (AZ2)
      subnets: "{{ priv2_subnet.subnet.id }}"
      routes:
        - dest: 0.0.0.0/0
          gateway_id: "{{ ngw_az2.nat_gateway_id }}"
    register: private_route_table_az2

As you can see this is bit more complicated than the previous example – because the subnets have to be assigned to the correct availability zones.

There are are a few extra variables that needed to be defined in order for this to work.

pub1_cidr: 192.168.90.0/26
pub2_cidr: 192.168.90.64/26
private1_cidr: 192.168.90.128/26
private2_cidr: 192.168.90.192/26
pub1_name: "Public Subnet (AZ1)"
pub2_name: "Public Subnet (AZ2)"
priv1_name: "Private Subnet (AZ1)"
priv2_name: "Private Subnet (AZ2)"
az1: us-east-2a
az2: us-east-2b

Last but not least – Terraform.

resource "aws_vpc" "testvpc" {
    cidr_block = "${var.vpc_cidr}"
    enable_dns_hostnames = true
    tags {
        Name = "${var.project_name}"
    }
}

resource "aws_subnet" "pub1_subnet" {
    vpc_id = "${aws_vpc.testvpc.id}"
    cidr_block = "${var.pub1_cidr}"
    availability_zone = "${var.az1}"
    tags {
        Name = "${var.pub1_name}"
    }
}

resource "aws_subnet" "pub2_subnet" {
    vpc_id = "${aws_vpc.testvpc.id}"
    cidr_block = "${var.pub2_cidr}"
    availability_zone = "${var.az2}"
    tags {
        Name = "${var.pub2_name}"
    }
}

resource "aws_subnet" "priv1_subnet" {
    vpc_id = "${aws_vpc.testvpc.id}"
    cidr_block = "${var.private1_cidr}"
    availability_zone = "${var.az1}"
    tags {
        Name = "${var.priv1_name}"
    }
}

resource "aws_subnet" "priv2_subnet" {
    vpc_id = "${aws_vpc.testvpc.id}"
    cidr_block = "${var.private2_cidr}"
    availability_zone = "${var.az2}"
    tags {
        Name = "${var.priv2_name}"
    }
}

resource "aws_internet_gateway" "igw" {
    vpc_id = "${aws_vpc.testvpc.id}"
    tags {
        Name = "${var.project_name}"
    }
}

resource "aws_eip" "eip1" {
    vpc = true
}

resource "aws_eip" "eip2" {
    vpc = true
}

resource "aws_nat_gateway" "nat_az1" {
    allocation_id = "${aws_eip.eip1.id}"
    subnet_id = "${aws_subnet.pub1_subnet.id}"
}

resource "aws_nat_gateway" "nat_az2" {
    allocation_id = "${aws_eip.eip2.id}"
    subnet_id = "${aws_subnet.pub2_subnet.id}"
}

resource "aws_route_table" "pub_rt" {
    vpc_id = "${aws_vpc.testvpc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.igw.id}"
    }
    tags {
        Name = "Public Route Table"
    }
}

resource "aws_route_table_association" "pub_rt_association1" {
    subnet_id = "${aws_subnet.pub1_subnet.id}"
    route_table_id = "${aws_route_table.pub_rt.id}"
}

resource "aws_route_table_association" "pub_rt_association2" {
    subnet_id = "${aws_subnet.pub2_subnet.id}"
    route_table_id = "${aws_route_table.pub_rt.id}"
}

resource "aws_route_table" "priv1_rt" {
    vpc_id = "${aws_vpc.testvpc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_nat_gateway.nat_az1.id}"
    }
    tags {
        Name = "Private Route Table (AZ1)"
    }
}

resource "aws_route_table" "priv2_rt" {
    vpc_id = "${aws_vpc.testvpc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_nat_gateway.nat_az2.id}"
    }
    tags {
        Name = "Private Route Table (AZ2)"
    }
}

resource "aws_route_table_association" "priv1_rt_association" {
    subnet_id = "${aws_subnet.priv1_subnet.id}"
    route_table_id = "${aws_route_table.priv1_rt.id}"
}

resource "aws_route_table_association" "priv2_rt_association" {
    subnet_id = "${aws_subnet.priv2_subnet.id}"
    route_table_id = "${aws_route_table.priv2_rt.id}"
}

And a new set of variables

variable "pub1_cidr" {
    description = "The CIDR block for the public subnet in us-east-2a"
    default     = "192.168.90.0/26"
}
variable "pub2_cidr" {
    description = "The CIDR block for the public subnet in us-east-2b"
    default     = "192.168.90.64/26"
}
variable "private1_cidr" {
    description = "The CIDR block for the private subnet in us-east-2a"
    default     = "192.168.90.128/26"
}
variable "private2_cidr" {
    description = "The CIDR block for the private subnet in us-east-2b"
    default     = "192.168.90.192/26"
}
variable "pub1_name" {
    description = "The name tag for the public subnet in us-east-2a"
    default     = "Public Subnet (AZ1)"
}
variable "pub2_name" {
    description = "The name tag for the public subnet in us-east-2b"
    default     = "Public Subnet (AZ2)"
}
variable "priv1_name" {
    description = "The name tag for the private subnet in us-east-2a"
    default     = "Private Subnet (AZ1)"
}
variable "priv2_name" {
    description = "The name tag for the private subnet in us-east-2b"
    default     = "Private Subnet (AZ2)"
}
variable "az1" {
    description = "The Availability zone for AZ1"
    default     = "us-east-2a"
}

variable "az2" {
    description = "The Availability zone for AZ2"
    default     = "us-east-2b"
}

First Score - # lines of Code (Including all nested files)

Terraform – 164

CloudFormation - 172

Ansible – 204

(Interesting to see here how the order has changed)

Second Score - Easy of deployment / teardown.

I will not give a numerical score here - just to mention a basic difference between the three options.

Each of the tools use a simple command line syntax to deploy

CloudFormation

aws cloudformation create-stack --stack-name testvpc --template-body file://vpc_cloudformation_template.yml
Ansible

ansible-playbook create-vpc.yml
Terraform

terraform apply -auto-approve

The teardown is a bit different

CloudFormation stores the information as a stack - and all you need to do to remove the stack and all of its resources is to run a simple command of:

aws cloudformation delete-stack --stack-name <STACKNAME>
Ansible - you will need to create an additional playbook for tearing down the environment - it does not store the state locally. This is a significant drawback – you have to make sure that you have the order correct – otherwise the teardown will fail. this means you need to understand as well how exactly the resources are created.

ansible-playbook remove-vpc.yml
Terraform - stores the state of the deployment - so a simple run will destroy all the resources

terraform destroy -auto-approve

You will see below that the duration of the runs are much longer than the previous example – the main reason being that the amount of time it takes to create a NAT gateway is long – really long (at least 1 minute per NAT GW) because AWS does a lot of grunt work in the background to provision this “magical” resource for you.

You can find the full output here of the runs below:

Ansible
CloudFormation
Terraform

Results

Terraform: create: 2m33s
destroy: 1m24s

Ansible: create: 3m56s
destroy: 2m12s

CloudFormation: create: 3m26s
destroy: 2m14s

Some interesting observations. It seems that terraform was the fastest one of the three – at least in this case.

The times are all over the place – and I cannot say one of the tools is faster than the other because the process is something that happens in the background and you have to wait for it complete. SO I am not sure how reliable the timings are.
The code for the Ansible playbook is by far the largest – mainly because in order to tear everything down – it requires going through the deployed pieces and ripping them out – which requires a complete set of code.
I decided to compare how much more code (you could compare increase in the amount of code to increased complexity) was added from the previous create step to this one

Ansible: 14 –> 117 (~8x increase)
CloudFormation: 24 –> 172 (~x7 Increase)
Terraform: 7 –> 105 (~x15 increase)
It is clear to me that allowing the provisioning tool to manage the dependencies on its own – is a lot simpler to handle – especially for large and complex environments.

This is by no means a recommendation to use one tool or the other - or to say that one tool is better than the other - just a simple side by side comparison between the three options that I have used in the past.

Thoughts and comments are always welcome, please feel free to leave them below.