VPCs, jump boxes, & NATs: Part 1 of our Hybrid Cloud Tips & Tricks Series

// 11.12.2014 // Data

Here at MediaMath, we are building out new services for our hundreds of terabytes of data at a lightning fast pace. This isn’t your grandma’s software development shop. Using cloud services like Amazon’s EC2 service allows us to scale up our infrastructure to match this pace.

However, MediaMath doesn’t run a purely cloud-hosted environment. Instead, we run a mixed data center environment, with a number of high performance components running in a variety of data centers throughout the world.

That means that the new AWS-hosted pipelines and data stores we are building must integrate with our in-house data centers, which poses some interesting challenges. I’m going to highlight a couple of the trickier aspects of this integration with a focus on maintaining data integrity between your in-house data centers and the cloud.

Use a VPC

As part of our mixed data center infrastructure, we of course require all the supporting hardware (routers, switches, racks, etc.) as well as “software” support of this infrastructure including Zenoss, StatsD, and Graphite. As we start to utilize cloud services like Amazon EC2, we want to meld these two worlds as seamlessly as possible.

The Amazon Virtual Private Cloud (VPC) gives you much better control over your internal cloud infrastructure then classic EC2 because it allows you to have separate routing rules, security groups, public and private subnets.

For example we’ve deployed Cassandra in some private subnets (in multiple availability zones), keeping our data protected and only accessible from our systems within the VPC. Our data enters the cloud through Amazon Kinesis Service or Amazon S3 Service and a service in the same availability zone (and private subnet) can pull from the Kinesis feeds or S3 and write to the Cassandra data store. This results in an efficient and secure flow of data from our data centers to the Amazon cloud service and then to our own custom cloud services.

Use A Jump Box

When using a VPC you’ll need to be able to access instances in private subnets. This can be accomplished using a jump box. A jump box is an instance you set up in a public subnet (with an Elastic IP) that has a security group that allows incoming ssh connections. We also recommend running sshd on a different port than the standard port 22 for added obscurity.

Once setup, you can ssh to your jump box and then ssh on to any instances in different subnets. Furthermore we recommend disabling (through security groups) external access via ssh to all other instances, especially those instances in public subnets. This makes public machines more secure by limiting access to come only through one or more jump boxes, which can be tightly controlled and audited.

To ease connecting with instances through a jump box, you can setup an ssh config that automatically proxies your ssh sessions through the jump box. This will give you the impression and convenience of directly connecting to instances, even in private subnets. Here is an example config to give you the idea:

Host aws-jump
HostName 54.x.x.x
Port 22
User awsuser
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes
ServerAliveInterval 30

Host 10.7.*
User awsuser
Port 22
ServerAliveInterval 30
IdentityFile ~/.ssh/mykey.pem
StrictHostKeyChecking no
ProxyCommand ssh -A aws-jump -W %h:%p

NATs Are Your Friend

We have services that access Cassandra from both private and public subnets. Private subnets use NAT instances to control access to external resources (external to Amazon). In a mixed environment those external resources are actually just other parts of our MediaMath infrastructure such as Graphite, Zenoss, Qasino, or internal services.

Using a VPC and NAT instances allows our in-house data center firewalls to more easily allow traffic back in from Amazon. The traffic will always be from a finite and manageable set of NAT instances (with stable public Elastic IPs).

More information on setting up a NAT can be found here.

That’s all for this week. Check back next week for more tips and tricks, where I’ll cover security groups, using NATs on public subnets, and when to use VPN.

A Picture of Brad Wasson


Principle Software Engineer, Data Services Brad Wasson is a Principle Software Engineer in the Data Services group at MediaMath. He leads the design and build of the real time decisioning systems in MediaMath's data management platform. Prior to MediaMath he worked on the distributed systems that powered Akamai Technologies' Advertising Decision Solutions division. Brad has a BS in Computer Science from Rensselaer Polytechnic Institute.

Leave a Reply

Your email address will not be published. Required fields are marked *