Replacing the AWS ELB - The Network Deep Dive

This is Part 4 in the Replacing the AWS ELB series.

  1. Replacing the AWS ELB - The Problem
  2. Replacing the AWS ELB - The Challenges
  3. Replacing the AWS ELB - The Design
  4. Replacing the AWS ELB - The Network Deep Dive (this post)
  5. Replacing the AWS ELB - Automation
  6. Replacing the AWS ELB - Final Thoughts

Why does this whole thing with the network actually work? Networking in AWS is not that complicated - (sometimes it can be - but it is usually pretty simple) so why do you need to add in an additional IP address into the loop - and one that is not even really part of he VPC?

To answer that question - we need to understand the basic construct of the route table in an AWS VPC. Think of the route table as a road sign - which tells you where you should go .

directions
Maybe not such a clear sign after all
(Source: https://www.flickr.com/photos/nnecapa)
Here is what a standard route table (RT) would look like

route

The first line says that all traffic that is part of your VPC - stays local - i.e. it is routed in your VPC, and the second line says that all other traffic that does not belong in the VPC - will be sent another device (in this case a NAT Gateway).

You are the master of your RT - which means you can route traffic destined for any address you would like - to any destination you would like. Of course - you cannot have duplicate entries in the RT or you will receive an error.

route_error1

And you cannot have a smaller subset the traffic routed to a different location - if a larger route already exists.

route_error2

But otherwise you can really do what you would like.
So defining a additional interface on an instance is something that is straight forward.

For example on a Centos/RHEL instance you create a new file in /etc/sysconfig/network-scripts/

DEVICE="eth0:1"  
BOOTPROTO="none"  
MTU="1500"  
ONBOOT="yes"  
TYPE="Ethernet"  
NETMASK=255.255.255.0  
IPADDR=172.16.1.100  
USERCTL=no  

This will create a second interface on your instance.

ip

Now of course the only entity in the subnet that knows that the IP exists on the network - except the instance itself.
That is why you can assign the same IP address to more than a single instance.

network_4

Transferring the VIP to another instance

In the previous post the last graphic showed in the case of a failure - haproxyb would send an API request that would transfer the route to the new instance.

keepalived has the option to run a script that execute when the it's pair fails - it is called a notify

vrrp_instance haproxy {  
  [...]  
  notify /etc/keepalived/master.sh  
}  

That is a regular bash script - and that bash script - can do whatever you would like, luckily that allows you to manipulate the routes through the AWS cli.

aws ec2 replace-route --route-table-id <ROUTE_TABLE> --destination-cidr-block <CIDR_BLOCK> --instance-id <INSTANCE_ID>

The parameters you would need to know are:

  • The ID of the route-table entry you need to change
  • The network of that you want to change
  • The ID of the instance that it should be

Now of course there are a lot of moving parts that need to come into place for all of this to work - and doing it manually would be a nightmare - especially at scale - that is why automation is crucial.

In the next post - I will explain how you can achieve this with a set of Ansible playbooks.