Using Chef to automate Octopus Deployments

If you are using Octopus Deploy to deploy your .Net code to your Windows servers and also using AWS Auto Scaling Groups, you may have come across some of the limitations of Octopus. Primarily, Octopus has no built-in process to deploy the current project release to a newly registered tentacle immediately upon the tentacle registration. Tentacles can either be registered as 'polling' or 'listening' tentacles.  Polling tentacles will periodically check with the Octopus server to determine if a deployment is required, while a listening tentacle will wait until a deployment is pushed from the Octopus server.  Neither of these approaches will deploy the release immediately.  This post will provide a process of using a Chef recipe to configure the Octopus tentacle in listening mode and then initiate a deployment of the latest project release from the Octopus Server using an API call.

I must give credit to CodeKing and his article of how to use Octopus Deploy with AWS.  His script provides the steps necessary to query the Octopus server for the latest project release and to initiate the deployment. Here is his post:
http://www.codeproject.com/Articles/719801/AWS-Deployment-With-Octopus-Deploy

The process I will demonstrate will rely on Chef to register the Octopus Tentacle as well as initiating the deployment of the latest project release to the server.  I'm not going to go into details about using Chef, if you need help using Chef please refer to their documentation at: https://docs.chef.io/

The recipe assumes the following:

• Amazon's AWS CLI is installed on the server (this can be done using another chef recipe)
• The server has access to S3 to download files to install Octopus Tentacle
• The Octopus project is configured to increment version numbers
• An Octopus account with an API key that has proper permissions to deploy the release.

The recipe will perform the following tasks:

1. Download the Octopus Tentacle installer.
2. Install Octopus Tentacle
3. Register the Octopus Tentacle to the Octopus Server
4. Deploy the latest project release from Octopus Server

1. Download the Octopus Tentacle installation from S3

It's not necessary to put the installer file on S3, but I prefer this approach as I know the file will always be available, vs relying on a web link that could potentially change.

Modify this script to use your S3 bucket
powershell_script 'Download Octopus Deploy' do code <<-EOH set-location c:/temp read-s3object -bucketname YourS3Bucket -key Octopus.Tentacle.3.1.2-x64.msi -file OctopusTentacle.msi EOH not_if do ::File.exists?('C:\temp\OctopusTentacle.msi') end end



2. Install Octopus Tentacle
With the installer downloaded, we must now install Octopus on the server.

windows_package "Install Octopus Deploy" do source "C:/temp/OctopusTentacle.msi" options "/qn" installer_type :msi action :install not_if do ::File.exists?('C:/Program Files/Octopus Deploy/Tentacle/Tentacle.exe') end end


3. Register the Octopus Tentacle to the Octopus Server
The next step is to configure and register the Octopus tentacle to the Octopus server. This will register the tentacle using the server hostname, which is later used for the deployment process.

Modify this script to use your Octopus server, API key, and role
batch 'configure octopus' do code <<-EOH cd "c:/Program Files/Octopus Deploy/Tentacle" Tentacle.exe create-instance --instance "Tentacle" --config "C:/Octopus/Tentacle.config" --console Tentacle.exe new-certificate --instance "Tentacle" --if-blank --console Tentacle.exe configure --instance "Tentacle" --reset-trust --console Tentacle.exe configure --instance "Tentacle" --home "C:/Octopus" --app "C:/Octopus/Applications" --port "10933" --noListen "True" --console Tentacle.exe register-with --instance "Tentacle" --server "http://octopus.domain.local" --name #{node.hostname} --apiKey "abcdefg12345678" --comms-style "TentacleActive" --server-comms-port "10943" --force --environment #{node.chef_environment} --role "webserver" --console Tentacle.exe proxy --instance "Tentacle" --proxyEnable "False" --console Tentacle.exe service --instance "Tentacle" --install --start --console EOH not_if do ::File.exists?('C:/Octopus/Tentacle.config') end end


4. Deploy the latest release
The final step is to query the Octopus server for the latest release of the specified project and then make an API call to the Octopus server to initiate a deployment to the server.

Modify this script to use your Octopus server, Project, and API key,
powershell_script 'Octopus Deploy Latest Release' do code <<-EOH #Sets an Alias for Octo.exe Set-Alias Octo "c:/chef/cookbooks/octopus/files/octopus/Octo.exe" -scope Script #Get the latest release $release = (Octo list-latestdeployments --project="OctopusProject" --server="http://octopus.domain.local" --apikey="abcdefg12345678" --environment="#{node.chef_environment}" | ?{$_.contains(" Version:") } | select-object -first 1) #Trim the $release variable to just the relase number $release = $release -replace "\\s+Version:\\s+", "" #Deploy the latest release for the project to this server $deploy = (Octo deploy-release --project="OctopusProject" --server="http://octopus.domain.local" --apiKey="abcdefg12345678" --releaseNumber="$release" --specificmachines="#{node.hostname}" --deployto="#{node.chef_environment}" --waitfordeployment --deploymenttimeout="01:00:00") #If the deploy fails, return the error if ($deploy -match 'Exit code:'){ Write-Host "Error occurred: " $deploy } EOH end


Running Chef - putting it all together
The above steps can be placed into a single Chef recipe or kept into separate recipes and run individually.  I prefer to put these in a single recipe and call it via the Chef run-list.

I currently use Chef-Solo and therefore make a call like this, specifiying the runlist as well as the environment.
chef-solo -c c:/chef/solo.rb -j c:/chef/runlist.json -E development -L c:/chef/log.log -l info

AWS - Bootstrap Windows EC2 instance with Chef-Solo

If you have an environment like we do you, fairly small and few service layers, it may not make sense to provision a Chef Server.  Luckily, we can still get the benefits of using Chef to configure our servers by using the included Chef-Solo.  Chef-Solo will run entirely locally on the instance, therefore we must include all required dependencies on the server, ie. cookbooks, run-lists, environments, etc.

The following process I will demonstrate how to launch an AWS EC2 instance and have Chef-Solo configure the instance. I will not go into details about how Chef works with recipes and cookbooks.

Here are the steps we will need to follow to start this process.

1. Create S3 bucket to store Chef files, this includes your cookbooks
2. Create IAM Role and define access to the S3 bucket
3. Configure UserData to run PowerShell on initial launch
• The UserData will do the following
• Download the Chef-Client from an S3 Bucket
• Download the Chef Cookbooks, Recipes, etc
• Install the Chef-Client
• Run Chef-solo

Let's get started!

1. Create an S3 bucket and upload the Chef MSI, cookbooks, run scripts, etc

Within the AWS console, create a new S3 bucket to store the Chef installer as well as all the required Chef cookbooks, environment files, run scripts, etc.  For this example we will use,  'examplebucket' for the S3 bucket name, You will need to use your own unique bucket name.

2. Create an IAM role with a policy to allow Read only access to the S3 bucket

By creating an IAM role and assigning the role to the instance we can eliminate the need to use an IAM user account with access keys.  IAM roles utilize temporary credentials to grant access to AWS resources.

Within the AWS console create a new IAM role and Select Role Type: AWS Service Roles > Amazon EC2

Follow the prompts clicking through until the Role is finally created. With the role created, we must now create a new Inline policy which will grant access to the S3 bucket.

Select the newly created Role and expand the 'Inline Policies' to create a new policy:

Choose the option to create a Custom Policy:

For the policy, we will grant ListBucket and GetObject restricted to the S3 bucket we created earlier.

Here is the policy, you must modify the bucket name :

{ "Version": "2012-10-17", "Statement":[{ "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": ["arn:aws:s3:::examplebucket/*"] }] }

3. Launch a new instance and configure UserData

When launching a new EC2 instance assign the previously created role to the instance.  Also, expand the Advanced details to provide 'UserData'.  This UserData allows you to run scripts when the instance is first launched.  Our instance will need to download and install Chef as well as execute Chef-Solo.

The userdata will utilize PowerShell to execute downloading and installing Chef.  Here is the actual userdata to include in the instance launch.  This assumes the instance AMI you are launching with has the AWS CLI available (the AWS provided AMI's for Windows include this already)

I've provided comments to the code for clarifications.

<powershell> #Load the compression assembly used to unzip files [System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem') Set-ExecutionPolicy unrestricted -Force New-Item c:/temp -ItemType Directory -Force set-location c:/temp #Download the chef client msi installer read-s3object -bucketname examplebucket/chef -key chef-client-12.4.1-1.msi -file chef-client-12.4.1-1.msi #Run the MSI installer in silent mode and wait for completion Start-Process C:/temp/chef-client-12.4.1-1.msi /qn -Wait New-Item c:/chef -ItemType Directory -Force set-location c:/chef #Download the chef files read-s3object -bucketname examplebucket/chef -key solo.rb -file solo.rb read-s3object -bucketname examplebucket/chef -key run-list.json -file run-list.json New-Item c:/chef/cookbooks -ItemType Directory -Force set-location c:/chef/cookbooks read-s3object -bucketname examplebucket/chef/cookbooks -key example-cookbook.zip -file example-cookbook.zip #Unzip the Chef cookbook $cookbooks_dir = "c:/chef/cookbooks" $example_zip = "c:/chef/cookbooks/example.zip" [System.IO.Compression.ZipFile]::ExtractToDirectory($example_zip, $cookbooks_dir) New-Item c:/chef/environments -ItemType Directory -Force set-location c:/chef/environments read-s3object -bucketname examplebucket/chef/environments -key dev.rb -file dev.rb #Update the environment variables for the newly installed chef #Since we are executing 'chef-solo' in the same PowerShell session we must #update the environment variables, otherwise chef-solo will be an unrecognized command $env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") #Run chef-solo with appropriate parameters chef-solo -c c:/chef/solo.rb -j c:/chef/run-list.json -E dev -L c:/chef/log.log -l info </powershell>

Troubleshooting

The UserData is only run on instance launch and not on restarts.  This is initiated by the Ec2ConfigService which records log information at:

C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog.txt

Chef-solo will record log information as defined by the -L option, in this case we are creating the logs at:   C:\chef\log.log

I hope this helps you progress to using Chef and moving more towards Infrastructure as Code in your environment!