When you’re moving at the speed of a real-time exchange, you need to continually find ways to improve your internal processes. Like many fledgling tech startups with agile dev processes, the MarkITx team wanted to remove barriers to our deployment process so we can move faster and get more done.
As we grow the dev team and add more contributors, it's important to us that everyone is able to participate in the building of the exchange. That means, learning by watching others. And what better way to watch others than via chat? Amid the daily dose of kittehs, discussion about LunchMetrics and frequent requests to adjust the room temp or lighting, chat is a place that already has our daily attention.
So once we had our tools in mind, we took some inspiration – both in terms of process and philosophy – from GitHub and Netflix and adapted GitHub’s open source tool, Hubot to suit our unique deployment process.
Similar to Netflix, we rely heavily on AWS for our infrastructure. One of the great things about relying on AWS is that we can roll out zero downtime deployments and really, really fast rollbacks.
We do this by using CloudFormation to define our Virtual Private Cloud (VPC) and our elastic load balancers. We also have CloudFormation templates to define each application we want to deploy in our environment. Here are links to our VPC, load balancers, and an example application CloudFormation templates. Our VPC template is very generic, but you may need to customize the load balancer and application to meet your needs.
Once we have our VPC, load balancers, and applications up and running we can start to release new versions. Here’s how we do zero-downtime deployments:
* First we use CloudFormation to create a new stack of the application with the latest version.
* Once that stack is up and the instances are ready, we add them to our load balancer and remove the old version from the load balancer.
* Since we wait to direct traffic to the new version until the machines are fully provisioned and the app is running and stable, we have no downtime when we switch to the new version. Unless we screwed up, of course. * If we did screw up, we have the ability to rollback really quickly. We keep the old instances running for a brief period of time after our deployment. Then, if we do have to rollback, we just add the instances running the old version to the load balancer and remove the new version. Within just a few seconds we’ve rolled back.
Those of you who have used CloudFormation and AWS know that it is a real pain to create new stacks or switch instances by hand. So, we wrote a small Node.js script to automate the process. In addition to managing all of the AWS infrastructure, this script also copies our code from Github to S3 to make deployments a bit easier.
We are considering open sourcing our code that does the deployment. Let us know if you’d find it useful (you can hit me up on Twitter).
Because we don’t want a mysterious script used for deploys we hooked our deployment infrastructure up with our Hubot chat bot. Hubot is an open source project from GitHub which, conveniently for us, is written in Node.js. For example, here is a snippet of our chat log showing a deploy I did yesterday to our QA environment that uses production data:
Before I get any further I should explain who Adam Smith is. We call our instance of Hubot Adam, named after Adam Smith, author of The Wealth of Nations. Our CTO, Ben, thought this a fitting free-market moniker given our goal to build an online exchange for trading enterprise IT hardware.
Because we don’t want Adam to be a boring robot that only does deploys we’ve enabled a few scripts that give Adam more of a personality. Our Adam is fond of pugs, sloths and - of course - kittens. can chat back images. Adam kitten me. Adam pug me.
One of the things Adam does really well is give our CEO, Frank, a little more style. Here are two examples:
We don’t know that we’ve created anything new necessarily or how widespread this setup is for AWS deployments, but our customization certainly has improved our deployment process by an order of magnitude. If you have questions about how to adapt, hit us up on Twitter or email us for tips.
p.s. Be sure to check out this GitHub talk to learn how they use Hubot.
p.p.s. Stay tuned for a follow-on post about the Hubot plugin we open sourced.