Indraneel's blog

Service deployment script - part 1: Establishing context

July 19, 2014

Whether you run a single person service or a massive operation, being able to deploy repeatedly with out needing manual steps should be the top priority. The same script (or set of scripts) should preferably be able to deploy to any target starting from a developer's box all the way upto the datacenter or your choice of public cloud. To understand what should be in your deployment script, it helps to agree on some deployment jargon.

Establishing some deployment context

Phases: Deployment phases get split up into the broad categories listed below

1. Resource allocation phase
2. Deployment of service phase
3. Deployment of service monitoring phase
4. Validation phase

Environments: Deployment targets the following environment types

1. Developer Onebox - A single VM which has all the services colocated on it, easily created (often for a single project) and then disposed.
2. Test environments - A multi-VM environment, all services logically segregated in their own VMs, but with out the scale of production. Often created for a massive integration testing, and kept around for a couple of weeks
3. Dogfood environments - A long running environment built with enough scale to support your entire team, which is where your periodic code pushes happen and get validated.
4. Production environments - This is where your customers are

Canonical Environment: Consider a 3 tier service production environment, with 2 front end boxes, 2 middle tier boxes, and 1 database machine. Also assume there exists a load balancer infront of the web boxes. This is likely the simplest setup you can build, that can be reliably updated without your customers noticing service degradation (Note: This commnent ignores the fact that your data needs to be backed up elsewhere to account for outages, more on that at a later point).
To be safe, you need to ensure a couple more things. Only web services should run on the front end boxes (by that I mean, you should run almost nothing that is not hosted in your preferred web server). Your middle tier boxes can have all forms of cron jobs, background services, caching tools as you desire. Basically middle tier boxes are where you implement anything custom. You may need to implement some form of load balancing in your middle tier boxes. The database roles should have zero services running on it that aren't hosted by your preferred SQL provider.

Canonical service monitoring environment: There are numerous strategies for service monitoring, but in it simplest form, it is often like the 3 tier architecture described above. The middle tier boxes run monitoring tests on the actual service and store results in the database layer. The front end boxes are used to show the status and historical data based on the monitoring results.

Scenarios: Deployments tend to target the following scenarios

1. Fresh deployment - The type of deployment on a freshly allocated resource
2. Patch deployment - Deployments on an existing environment
3. Scale out - Deployments that merely modify the number of nodes they run on, instead of changing the code
4. Machine replacement - Deployments that allow you to replace a failing machine with another
5. Disaster recovery - Often involves restoring data from backup and then deploying the services on top

Once you have selected the environment you are deploying to, and the scenario you are expecting to achieve, it becomes obvious which deployment phases you are interested in. Some environments excercise all the phases, (e.g. the developer Onebox), while others excercise only some of the phases (e.g. the production environment may not include resource allocation in case of patching).

Miscellaneous: Once you have identified the environment, the scenario of interest and the phases of deployment, a few more things have to be selected.

1. Builds: Full builds or Partial builds
2. Configuration: Describes all variables (and their values) that the service feels is necessary for functioning
3. Secrets: Passwords or certificates

All our following discussions on deployment scripts will refer to this context.