Footgun Prevention with AWS VPC Subnetting and Addressing

How many online tutorials and examples instruct Dear Reader to create VPCs with the network address of 192.168.1.0/24? Or 10.0.0.0/24? Legions of AWS users are set up to learn about VPCs the hard way, because this practice not only limits the number of instances available (251 in a /24 on AWS), but it also limits your ability to subnet to cover availability zones, environments and things like private vs. public networks.

If you run out of room, you have to move to a new VPC and that is painful and best avoided.

This post is for folks who are using AWS but might not realise they are now Cloud Network Engineers :) VPCs are simple when you know them, but we’re all Donald Rumsfeld at some point: we don’t know what we don’t know (until it bites us in the proverbial).

Learning by doing sounds great, but hold on a minute. When you’re using AWS it’s not just for the sake of it, you’ve got some end result to deliver. Right? So doing the minimum with VPCs (and unknowingly being productive today by creating technical debt for tomorrow) is fine. Right?

After all, why would you need to know about the best practices for VPCs if all you want to do is launch Wordpress on EC2? Isn’t that the Network Engineer’s job?

No way! That’t not me! (It is now. Cloud.) Everyone knows that Network Engineers are weird, right? I mean, who starts a fist fight about creating trunk ports to servers? They’re nearly as bad as developers, talking to rubber ducks! And anyway, if they’re so awesome, why is everything their fault?! Or “their” network’s fault. Isn’t it all just bits of string? Who cares?

Fact is, in the cloud, you are the Network Engineer. Being weird is optional, but knowing VPCs is mandatory.

This post is about stepping away from the footgun and doing VPC subnetting in a non-lethal manner.

VPC Subnetting and Sizing

There are lots of changeable, programmable things in the cloud. Things you can change your mind about later. Change the instance size. Change the database size. Change security groups. Change the IAM role on an instance.

One thing you can’t do, though, is modify a VPC network address after it has been created.

Imagine you decide on a basket size before you go shopping: the size of the basket decides how much shopping you can do. Once your shopping is in the basket and you need “more basket” then you either add a basket and try to carry two, or take all your shopping out of the current basket and put it in a bigger basket. So it is with VPCs. Just swap fruit for EC2 instances.

Non-cloud networks are different to cloud networks. Old school, non-cloud networks (ie. the majority out there in your home and office) worried about things like broadcast domains, spanning trees and all kinds of weird gobbledygook that only Network Engineers knew / cared about / used to protect their job.

In AWS, where you don’t have to worry about broadcast domains and all that crap, you do networking differently. For a start, when you create your VPC, you don’t carefully size it for what you need right now (your Wordpress on EC2 tutorial, for example): you size it so that you avoid problems in the future. That means: going big! Cloud networks are, in this way, totally different to non-cloud networks.

From the AWS doc on VPCs and Subnets:

VPC and Subnet Sizing for IPv4

You can assign a single CIDR block to a VPC. The allowed block size is between a /16 netmask and /28netmask. In other words, the VPC can contain from 16 to 65,536 IP addresses.

When you create a VPC, we recommend that you specify a CIDR block (of /16 or smaller) from the private IPv4 address ranges as specified in RFC 1918:

10.0.0.0 - 10.255.255.255 (10/8 prefix)

172.16.0.0 - 172.31.255.255 (172.16/12 prefix)

192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Are you familiar with CIDR? Scrumpy jokes aside, this is a tricky bit for even seasoned IT folks. I used to use variable-length subnet masking as an interview question until I saw someone cry, it’s that “fun”.

Does this shock you: you know that familiar IP address, 192.168.1.1? Did you know there’s actually TWO addresses represented there?

“Surely one is having the bantz?”

No, I’m serious. 192.168.1.1 is two addresses — a NETWORK address and a HOST address. Sorry for shouting, but it’s important, and if we English know anything it’s that shouting gets results.

https://www.youtube.com/watch?v=Daa8ZnxC-0Y

Enter the Netmask

The way you find out the network and host address from a single IP address (no bantz) is by applying something called a netmask. It looks like this, the netmask being the /24 at the end of 192.168.1.1/24.

The /24 netmask at the end means that, if you wrote out 192.168.1.1 in binary there would be 32 bits, and the left-most 24 bits represent the network address. 24 is the count of bits from the left. That’s all it is.

The right-most bits (the 8 left of the whole 32) are the host address. In binary, eight bits adds up to 256 (128+64+32+16+8+4+2+1). In AWS this means you can have 251 instances in that network (broadcast is .255, and AWS reserve the first .1, .2, .3 leaving 251 possible instance addresses).

So, in the case of 192.168.1.1/24 the network address is the 192.168.1 octets (fancy talk for eight binary bits) and the host address (of your laptop on your home wifi) is the remaining .1.

The way you tend to describe the network address is to ignore the host bits on the right (cos, hey, we’re Network Engineers now and who cares about computers?) and just report the network address and the netmask, a la: The VPC network address is 192.168.1.0 / 24.

Don’t believe me? Useful site alert.

“Pack it in, Steve, what’s CIDR got to do with VPCs?”

You’ve been in the AWS console and created a VPC with the address of 192.168.1.0/24 and you know that lets you have 254 hosts, less a few for AWS reserved use. Let’s call is 251 for ca$h.

Quick point here, just checking we’re all awake: you do know that a VPC isn’t a network you can put instances directly into, right? Even though we’ve given it a network address, it’s an AWS cloud construct that aligns with a region and spans availability zones. You can attach special things to a VPC like Internet Gateways, but you don’t stick instances directly into a VPC: these every day non-networky things go in subnets.

Really, all you’ve done so far is create your shopping basket (VPC). Now you need compartments inside the basket to put your actual shopping (cloud objects like EC2 instances) inside. These compartments are called subnets.

One subnet per VPC is called a footgun

You can create one or more subnets inside your VPC. You could create one subnet with the same network address as the VPC itself, that’s totally possible but not fine at all. One subnet in your VPC means you can only ever have one usable network in your VPC, for your EC2s and everything else.

Why is this a problem? Well, what if you want two kinds of networks — a public one connected to the internet for your web server, and a private network that the internet can’t get to for your database of customer data? Can’t do it with one subnet, it’s either on or off the internet. To do both in this case you’d need two VPCs. Madness.

A public network in AWS is one that is has a route to the VPC’s Internet Gateway (the gateway is attached to your VPC, not your subnet). If your house (VPC) didn’t have a door (internet gateway), and even if it did have a door your couldn’t find it (route), then nobody couldn’t get in and out of your house. Same thing. Sometimes you want a door, sometimes you don’t.

If you have created just one subnet in your VPC then you have to go all in on “public or private”, there’s no “public and private” in the same subnet.

Then there’s the other problem with one subnet in your VPC: a subnet is aligned to one availability zone, whereas your VPC is across all AZ’s in the Region. Some AWS features like RDS MultiAZ need more than one AZ. And if the AZ that houses your subnet spontaneously combusts, that’s you done.

So, for both reasons (private and public, and multi-AZ) you probably want more than one subnet in your VPC. Without multiple subnets, there’s no autoscaling for you. No need for a load balancer. No multi-AZ RDS database. Etc. Not a good direction. One subnet per VPC is bad for these reasons. But totally possible. Footgun.

How many subnets should I start with?

You probably want six subnets as a minimum in your VPC, but it’s wise to allow for more (spares, if you like) because you know how the future likes to spring surprises. But let’s do six for illustration:

All those orange boxes are subnets that we actually create in AWS (we don’t create one big subnet for the AZ and we don’t create one for the VPC).

How do we Make It So if our VPC has the network address of 192.168.1.0/24? Well, with subnets, the clue is in the name: we need to subdivide our network.

We get more networks by increasing the size of the netmask, but it costs us by gobbling up our hosts bits and reducing the number of instances we can shove into our shiny, new, half-size subnets. The netmask, per AWS, can go up to /28, but what should it be?

At this point, you can cheat, if you want to be like a glassy eyed fat seal catching fish in the zoo, and not knowing why you’re unhappy.

Subnet Masking for Muggles

How it works is this: for every extra bit you increase the netmask by, you are creating two more subnets that are are half the (hosts) size of the original network.

So, in our VPC’s case:

• 192.168.1.0 / 24 = 254 hosts theoretical, 251 on AWS.

we increase the netmask by one bit and it becomes two new networks with half as many hosts:

• 192.168.1.0 / 25 = 127 hosts theoretical, 123 on AWS.
• 192.168.1.128 / 25

Note that the second network address is .128, that’s because our netmask of /25 is now using the left-most bit of the last octet. That left-most bit is 128 — remember our binary: 128–64–32–16–8–4–2–1

But two subnets is not enough for us! We need six! So let’s increase the netmask again by one bit to /26. We are now using the two left-most bits of the last octet, so now our network address “boundary” has moved from 128 to 64… so, if you’re awake, we have four possible subnets and they are called .0, .64, .128 and .192:

• 192.168.1.0 / 26 = 63 hosts theoretical, 60 on AWS
• 192.168.1.64 / 26
• 192.168.1.128 / 26
• 192.168.1.192 / 26

Notice something? We started with a possible 251 hosts in our 192.168.1.0/24 network, now we have a possible 240 as each network gobbles up a few special addresses.

“What! What? Why 240?” Because we have four networks, and each network “loses” four addresses to the AWS network gods, so with a network of 192.168.1.0 / 26 that’s 60 host addresses, and 4 x 60 = 240.

However, we need six subnets, not four… increasing the netmask again to /27 creates 8 subnets. There is no netmask boundary for six subnets (but you’ll see later there’s a way to kinda do this).

The /27 is on the boundary of the 3rd left-most bit in the octet, which is 32, so you guessed it again, our network addresses go up in lots of 32.

• 192.168.1.0 / 27 = 28 hosts on AWS
• 192.168.1.32 / 27
• 192.168.1.64 / 27
• 192.168.1.96 / 27
• 192.168.1.128 / 27
• 192.168.1.160 / 27
• 192.168.1.192 / 27
• 192.168.1.224 / 27

We have achieved our goal of six subnets across three AZs in our region-wide VPC, but at a cost. Our original single network could host 251 EC2 instances or other AWS gizmos, but now we are down to (8 x 28 =) 224.

Is there a better way? Yes!

VPCs and Subnetting for Chilled People

“Go big” is my advice for my fellow VPCing Subnetters!

Going big means creating a VPC with a small netmask. The AWS block size limits go from /16 to /28, so we’ll go for /16.

For our illustration, we’ll pick 172.16.0.0/16. That /16 means the left-most 16 bits, ie the first two octets of 172.16, are our network address and the right-most 16 bits give us room for 65,531 hosts. Before you say “Don’t be ridiculous, Steve, we don’t need 65k hosts!” — have you forgotten our previous lesson? This is about creating subnets just as much as deploying instances. We need more than one subnet in a VPC and each time we increase the netmask and create two more networks, we are halving the amounts of hosts in those subnets.

Yeah, yeah, yeah. But how?

For our illustration we’re going to state the following for our VPC design:

1. We don’t expect the VPC to connect/peer with other VPCs or VPN to on-premises so we aren’t concerned with IP address clashes (seriously, this is an Important Point. If you have to connect your VPC to some non-AWS network then you need to work out the correct addressing for your VPC. Usually people use a VPC with a smaller, less-likely-to-overlap network space specifically for this purpose).
2. We are using the eu-west-1 region with has three AZs.
3. We want subnets in all AZs in the region.
4. We want a spare “AZ-sized” subnet.
5. In each AZ we need two subnets (private, public) and spare.

With that said, we walk over to the whiteboard and scribble:

The thinking that emerges from this whiteboarding is:

1. We need four “AZ-sized” subnets, so that means we need to increase the netmask by two bits to from /16 to /18.
2. We will then subdivide each “AZ subnet” again to give us private and public, so that’s increasing the netmask again to /19.
3. But we want a spare subnet too, so we’re going to sub-divide the public network into two — public and spare, which means this one goes from /19 to /20. We expect fewer public instances, and lots of private ones.

But, and this is important, this is design theory at this stage, and here is something that people get tripped up on:

We don’t actually create all of these subnets in AWS.

This seemingly obvious fact is not obvious to people new to networking. If you tried to create all these subnets in AWS you would get overlap errors. You only create the “leaf node” subnets. It’s clearer (I think!) when you read this then see the picture, then my list of subnets in AWS.

• VPC eu-west-1 address is 172.16.0.0/16
• eu-west-1a address is 172.16.0.0/18 (but we don’t create this, it’s just in our diagram, we create the “leaf” subnets marked below)
• eu-west-1a-private address is 172.16.0.0/19 (we _DO_ create this in AWS. There’s no overlap with the VPC address because the VPC is not a subnet of ours)
• eu-west-1a-* is the partner subnet of “private” above, and it’s “theoretical” address would be 172.16.32.0/19 but we are subnetting this again to create public and leave a spare
• eu-west-1a-public address is 172.16.32.0/20. (see, if we’d created the same address but with a netmask of /19 in the previous step, we’d get an overlap issue here).
• eu-west-1a-spare address is 172.16.48.0/20 (but we don’t create this, we just leave it spare for future use).

The full picture looks like this:

And in the VPC page of the AWS Console is looks like this:

Summary

The bigger the netmask the smaller the room to maneouvre. Not only do you limit yourself to small number of hosts, when you need to subnet it you have less room. Be careful, though, if you will need to peer that VPC as addresses can’t overlap with your peers (routing, innit?) and the larger YOUR network the bigger chance of overlap. Start with a subnet closer to the size of /16 and not /24. You can’t change a VPC after it has been created, and if it’s full of stuff like hosts and gateways then you don’t want to go through the pain of migrating them to a new VPC. That way, lies madness.