VPN, security groups, & more on NATs: Part 2 of our Hybrid Cloud Tips & Tricks Series

// 11.19.2014 // Data

Today’s post picks up where we left off last week. As a refresher, here at MediaMath, we run a hybrid data center-cloud environment. In this two-part blog series, I will be highlighting a couple of the trickier aspects of this integration with a focus on maintaining data integrity between your in-house data centers and the cloud – specifically, an Amazon Web Services cloud.

Go to part 1: VPCs, Jump Boxes, and NATs

Using a NAT on a Public Subnet

Some of our services need endpoints accessible from our in-house data centers. These services must go in a public subnet with either Elastic IPs or, if your services are flexible enough, with somewhat ephemeral auto-assigned public IPs.

However if we have any outbound communication to our in-house data centers (for example reporting stats to Graphite and Qasino), these machines just appear as random machines on the internet unless we add their IPs to our firewall rules. But as we add or change machines, their public IPs will shift. Maintaining these in firewall rules is a non-starter.

So to solve this problem we can route outbound traffic in this public subnet through our NAT instance whose Elastic IP is already in our firewall rules. There are two ways of doing this. One way is to create static routes on each node that send traffic destined for our internal destinations through the NAT IP (instead of the internet gateway it will use the NAT as the gateway).

For example, if is an internal address of a service in our existing data center and is the subnet address of the NAT (must be in the same subnet):

/sbin/route add gw dev eth0

In order for this route to persist be sure to add the corresponding line in the network configuration file for your OS (on Debian systems this is /etc/network/interfaces).

These per-node settings are best maintained with your installation process. We use Ansible, which can very easily ensure that one or more static routes get setup. Here are some example plays:

name: “for aws only: resolve static routes to ip addrs”
tags: [ static_routes ]
sudo: False
local_action: command bash -c “host {{item}} | awk ‘{print $NF}'”
register: static_route_ips
when: static_route_hosts is defined
with_items: static_route_hosts

name: “for aws only: setup static routes”
tags: [ static_routes ]
action: command /sbin/route add “{{item.stdout}}” gw “{{static_route_gateway}}” dev eth0
when: static_route_ips is defined and static_route_gateway is defined
with_items: static_route_ips.results
ignore_errors: true

name: “for aws only: setup static routes in /etc/network/interfaces”
tags: [ static_routes ]
action: lineinfile insertafter=’^iface eth0′ regexp=”up route add -host {{item.stdout}} gw {{static_route_gateway}}” line=”up route add -host {{item.stdout}} gw {{static_route_gateway}}” dest=/etc/network/interfaces backup=yes
when: static_route_ips is defined and static_route_gateway is defined
with_items: static_route_ips.results

The second option is to change the routing table for the subnet. In this case it will affect all nodes in the subnet. The route should have specific destinations (i.e. with the Target being the NAT instance in the subnet.

Security Groups

Anytime you create an instance in a public subnet it is, by definition, directly connected to the internet (through an internet gateway). Make sure you setup a security group with proper restrictions and add it to the instance. More information about security groups here.

In our case, we’re building infrastructure that we only access from our data centers. To maintain security, we’ve added the CIDR blocks of those data centers to our security group and deny all other traffic. This gives us a little more peace of mind about limiting access to this service. This should be just a part of your overall security process and policies.

There are other things to consider as well such as secure protocols (e.g. TLS), authentication, ACLs, etc., but I won’t be covering those here.


Finally, it is also an option to set up a secure VPN between your data centers and Amazon.  Keep in mind this is quite a bit more complicated to setup (so out of the scope of this article) and will cost additional money.

There is also a performance penalty going through a VPN.  The performance degradation can vary based on many factors.  But as you are adding extra hops and hardware between your systems and Amazon instances you could see as much as a 10% decrease in throughput depending on the application.  Our service needs as much throughput and as low of a latency as possible, so we are not planning to use a VPN for this particular service. A VPN however is one of the most reliable and secure ways to connect your existing data centers with Amazon cloud VPCs.

Other Suggestions or Techniques?

Clearly there are a number of techniques to use in Amazon’s cloud services that maintain integrity between your existing data centers and your instances in the cloud. I’ve just hit on a couple of options, such as taking advantage of a VPC for customizing subnets for public and private use, setting up a jump box, utilizing NATs in a public subnet with custom routes, and finally setting up security groups properly.

If you have found success with other techniques, or you have any questions about what I’ve covered here, please feel free to hit me up in the comments.

A Picture of Brad Wasson


Principle Software Engineer, Data Services Brad Wasson is a Principle Software Engineer in the Data Services group at MediaMath. He leads the design and build of the real time decisioning systems in MediaMath's data management platform. Prior to MediaMath he worked on the distributed systems that powered Akamai Technologies' Advertising Decision Solutions division. Brad has a BS in Computer Science from Rensselaer Polytechnic Institute.

Leave a Reply

Your email address will not be published. Required fields are marked *