Using Chef to automate Octopus Deployments

If you are using Octopus Deploy to deploy your .Net code to your Windows servers and also using AWS Auto Scaling Groups, you may have come across some of the limitations of Octopus. Primarily, Octopus has no built-in process to deploy the current project release to a newly registered tentacle immediately upon the tentacle registration. Tentacles can either be registered as 'polling' or 'listening' tentacles.  Polling tentacles will periodically check with the Octopus server to determine if a deployment is required, while a listening tentacle will wait until a deployment is pushed from the Octopus server.  Neither of these approaches will deploy the release immediately.  This post will provide a process of using a Chef recipe to configure the Octopus tentacle in listening mode and then initiate a deployment of the latest project release from the Octopus Server using an API call.

I must give credit to CodeKing and his article of how to use Octopus Deploy with AWS.  His script provides the steps necessary to query the Octopus server for the latest project release and to initiate the deployment. Here is his post:
http://www.codeproject.com/Articles/719801/AWS-Deployment-With-Octopus-Deploy

The process I will demonstrate will rely on Chef to register the Octopus Tentacle as well as initiating the deployment of the latest project release to the server.  I'm not going to go into details about using Chef, if you need help using Chef please refer to their documentation at: https://docs.chef.io/

The recipe assumes the following:

• Amazon's AWS CLI is installed on the server (this can be done using another chef recipe)
• The server has access to S3 to download files to install Octopus Tentacle
• The Octopus project is configured to increment version numbers
• An Octopus account with an API key that has proper permissions to deploy the release.

The recipe will perform the following tasks:

1. Download the Octopus Tentacle installer.
2. Install Octopus Tentacle
3. Register the Octopus Tentacle to the Octopus Server
4. Deploy the latest project release from Octopus Server

1. Download the Octopus Tentacle installation from S3

It's not necessary to put the installer file on S3, but I prefer this approach as I know the file will always be available, vs relying on a web link that could potentially change.

Modify this script to use your S3 bucket
powershell_script 'Download Octopus Deploy' do code <<-EOH set-location c:/temp read-s3object -bucketname YourS3Bucket -key Octopus.Tentacle.3.1.2-x64.msi -file OctopusTentacle.msi EOH not_if do ::File.exists?('C:\temp\OctopusTentacle.msi') end end



2. Install Octopus Tentacle
With the installer downloaded, we must now install Octopus on the server.

windows_package "Install Octopus Deploy" do source "C:/temp/OctopusTentacle.msi" options "/qn" installer_type :msi action :install not_if do ::File.exists?('C:/Program Files/Octopus Deploy/Tentacle/Tentacle.exe') end end


3. Register the Octopus Tentacle to the Octopus Server
The next step is to configure and register the Octopus tentacle to the Octopus server. This will register the tentacle using the server hostname, which is later used for the deployment process.

Modify this script to use your Octopus server, API key, and role
batch 'configure octopus' do code <<-EOH cd "c:/Program Files/Octopus Deploy/Tentacle" Tentacle.exe create-instance --instance "Tentacle" --config "C:/Octopus/Tentacle.config" --console Tentacle.exe new-certificate --instance "Tentacle" --if-blank --console Tentacle.exe configure --instance "Tentacle" --reset-trust --console Tentacle.exe configure --instance "Tentacle" --home "C:/Octopus" --app "C:/Octopus/Applications" --port "10933" --noListen "True" --console Tentacle.exe register-with --instance "Tentacle" --server "http://octopus.domain.local" --name #{node.hostname} --apiKey "abcdefg12345678" --comms-style "TentacleActive" --server-comms-port "10943" --force --environment #{node.chef_environment} --role "webserver" --console Tentacle.exe proxy --instance "Tentacle" --proxyEnable "False" --console Tentacle.exe service --instance "Tentacle" --install --start --console EOH not_if do ::File.exists?('C:/Octopus/Tentacle.config') end end


4. Deploy the latest release
The final step is to query the Octopus server for the latest release of the specified project and then make an API call to the Octopus server to initiate a deployment to the server.

Modify this script to use your Octopus server, Project, and API key,
powershell_script 'Octopus Deploy Latest Release' do code <<-EOH #Sets an Alias for Octo.exe Set-Alias Octo "c:/chef/cookbooks/octopus/files/octopus/Octo.exe" -scope Script #Get the latest release $release = (Octo list-latestdeployments --project="OctopusProject" --server="http://octopus.domain.local" --apikey="abcdefg12345678" --environment="#{node.chef_environment}" | ?{$_.contains(" Version:") } | select-object -first 1) #Trim the $release variable to just the relase number $release = $release -replace "\\s+Version:\\s+", "" #Deploy the latest release for the project to this server $deploy = (Octo deploy-release --project="OctopusProject" --server="http://octopus.domain.local" --apiKey="abcdefg12345678" --releaseNumber="$release" --specificmachines="#{node.hostname}" --deployto="#{node.chef_environment}" --waitfordeployment --deploymenttimeout="01:00:00") #If the deploy fails, return the error if ($deploy -match 'Exit code:'){ Write-Host "Error occurred: " $deploy } EOH end


Running Chef - putting it all together
The above steps can be placed into a single Chef recipe or kept into separate recipes and run individually.  I prefer to put these in a single recipe and call it via the Chef run-list.

I currently use Chef-Solo and therefore make a call like this, specifiying the runlist as well as the environment.
chef-solo -c c:/chef/solo.rb -j c:/chef/runlist.json -E development -L c:/chef/log.log -l info

AWS - using SQS to cleanup Active Directory of terminated instances

If your Amazon EC2 instances are part of an Auto Scaling Group and are required to be joined to a Windows Domain, then maintaining a clean Active Directory environment may be an after thought. EC2 instances can be terminated for a variety of reasons, and since they may terminate abruptly their Active Directory objects may not be removed from Active Directory.  The following steps will help you create a process utilizing Amazon's Simple Queue Service (SQS) to remove terminated instances from Active Directory and to help keep a cleaner Active Directory structure.

The following assumes:

• Your servers are launched using an Auto Scaling Group and are auto joined to an Active Directory domain using the AWS Instance Id as its hostname.  Please see this post for details on how to accomplish this -  Auto Join EC2 instances to domain
• An EC2 windows instance that has
• An IAM role assigned to the instance
• The AWS CLI installed on the instance
• Access to the Active Directory domain

1. Create the SQS queue

Within AWS, create a new SQS queue.  Be sure to set the message retention period to a value greater than how often you plan to run the scheduled powershell script. We will set the permissions in a later step, after we've created the SNS topic.

2. Create the SNS topic

Create a new SNS topic in AWS and add a subscription to the SNS topic selecting 'Amazon SQS' as the endpoint, ie: arn:aws:sqs:us-east-1:123456789012:SQS-InstanceTerminations

3. Configure the SQS queue permissions

Return to the SQS created in the prior step and select the Permissions tab.  Add/Modify the permissions to allow SQS:SendMessage from the SNS topic you just created.  Modify the below policy to use your SNS ARN and the SQS ARN resource.

{ "Version": "2012-10-17", "Id": "SQSInstanceTerminationsPolicy", "Statement": [ { "Sid": "SQS-Access", "Effect": "Allow", "Principal": "*", "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:us-east-1:123456789012:SQS-InstanceTerminations", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:SNS-InstanceTerminations" } } } ] }

4. Configure the notification for the Auto Scaling Group

Select your Auto Scaling Group and choose the 'Notifications' tab and then 'Create notification'.
For the notification choose the option 'terminate' and select the SNS topic created earlier.

5. Configure the IAM role

The EC2 instance that will be running our Powershell cleanup script  requires permissions to access the SQS queue.  To allow this, configure a security policy for the IAM role that is attached to the instance.  Modify the policy below for the Resource ARN to match your SQS ARN.

"SQS-Access": { "Version": "2012-10-17", "Statement": [ { "Action": [ sqs:GetQueueAttributes, sqs:GetQueueUrl, sqs:ReceiveMessage, sqs:DeleteMessage ], "Resource": "arn:aws:sqs:us-east-1:123456789012:SQS-InstanceTerminations", "Effect": "Allow" } ] }

6. Create the Powershell script to retrieve the SQS messages

Powershell is used to obtain the SQS messages of the terminated instances and then removes the terminated servers from Active Directory.  Save the script on the server that will run the scheduled task.

Here is the script for the complete process. Modify this script to use your SQS queue name.

#Get the SQS queue URL $queueurl = Get-SQSQueueUrl -queuename SQSQueueName -region us-east-1 #Get the number of SQS messages in the queue $messages = Get-SQSQueueAttribute -QueueUrl $queueurl -AttributeName ApproximateNumberOfMessages -Region us-east-1 $messageCount = $messages.ApproximateNumberOfMessages #Loop through each message to remove the terminated server from Active Directory While ($messageCount -gt 0) { $messageCount-=1 $message = Receive-SQSMessage -QueueUrl $queueurl -Region us-east-1 $messagebody = $message.Body.Split(",") $id = $messagebody[19].TrimStart('\"EC2InstanceId\":\"') $id = $id.TrimEnd('\"') Get-ADComputer -Identity $id | Remove-ADObject -Recursive -Confirm:$False Remove-SQSMessage -QueueUrl $queueurl -ReceiptHandle $message.ReceiptHandle -region us-east-1 -Force }

7. Create a scheduled task to run the Powershell script

To schedule the script,  configure a scheduled task on the Windows EC2 instance to run "Powershell" with an optional argument. 

The program path for PowerShell is: 'C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe'

The optional arguments is the path to your script:  'C:\Scripts\ActiveDirectory-CleanUp.ps1'

NOTE: You must run the scheduled task using a Windows User account that has the appropriate user permissions to remove objects from Active Directory.

AWS - Auto join EC2 Windows instance to Active Directory Domain

Some environments will require you to join your Windows servers to a domain.  The following will show the steps taken to automatically join a server to a Windows domain.  This assumes the following:
   An existing AWS VPC with access to S3 bucket
   New instances are able to communicate to a domain controller.

NOTE:  Amazon does offer its Directory Service with AD Connector that will connect your VPC to your ActiveDirectory, but this will show how you can do so without the AD Connector.

The steps:

1. Create a PowerShell script to join a server to the domain
2. Secure the credentials by converting the PowerShell script to an Exe executable using PS2exe
3. Create an S3 bucket and upload the exe file
4. Create an IAM role with a policy to allow Read access to the S3 bucket
5. Launch a new instance, assigning the IAM role and providing User Data which will run the required scripts at first launch

1. Create the PowerShell script

The PowerShell script will join the server to the domain.   We will use the Add-Computer function, and a user account that has permissions to join computers to the domain. Here is the full script, modify the username, password, and DomainName for your environment.  

#Retrieve the AWS instance ID, keep trying until the metadata is available $instanceID = "null" while ($instanceID -NotLike "i-*") { Start-Sleep -s 3 $instanceID = invoke-restmethod -uri http://169.254.169.254/latest/meta-data/instance-id } $username = "domain\username" $password = "password12345" | ConvertTo-SecureString -AsPlainText -Force $cred = New-Object -typename System.Management.Automation.PSCredential($username, $password) Try { Rename-Computer -NewName $instanceID -Force Start-Sleep -s 5 Add-Computer -DomainName domain.local -OUPath "OU=YourOU,DC=domain,DC=local" -Options JoinWithNewName,AccountCreate -Credential $cred -Force -Restart -erroraction 'stop' } Catch{ echo $_.Exception | Out-File c:\temp\error-joindomain.txt -Append }
Save the file as JoinDomain.ps1

2. Convert the PowerShell script to an executable file

To help secure the credentials we will convert the PowerShell script using PS2exe to an executable file. Download PS2exe from: PS2exe download

Extract the zip file to a folder and then run PS2exe.ps1 on the JoinDomain.ps1 script to convert it to an exe file. From a command prompt run the following:

c:\> .\ps2exe.ps1 -inputfile JoinDomain.ps1 JoinDomain.exe

This will create the JoinDomain.exe file.

3. Create an S3 bucket and upload the exe file

Within the AWS console, create a new S3 bucket to store the JoinDomain.exe file.

For this example we will use,  examplebucket  for the bucket name, You will need to use your own unique bucket name.

With the bucket created, we can upload the JoinDomain.exe file to the bucket.

4. Create an IAM role with a policy to allow Read only access to the S3 bucket

By creating an IAM role and assigning the role to the instance we can eliminate the need to use an IAM user account with access keys.  IAM roles utilize temporary credentials to grant access.

Create an IAM role in the AWS console, and Select Role Type: AWS Service Roles > Amazon EC2

Follow the prompts through, clicking next until the Role is finally created. With the role created, we must now create a new Inline policy which will grant access to the S3 bucket.

Select the newly created Role and expand the 'Inline Policies' to create a new policy:

Choose the option to create a Custom Policy:

For the policy, we grant ListBucket and GetObject restricted to the S3 bucket.  Here is the policy, you must modify the bucket name :

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": ["arn:aws:s3:::examplebucket/*"]
}
]
}

5.  Launch a new instace

Launch a new instance into the VPC.  We need to attach the IAM Role to the instance as well as configure the Advanced > User Data

The User Data is used to run scripts when the instance is first launched.  For our example, we will be downloading the JoinDomain.exe file from S3 and finally executing it.

First assign the IAM Role to the instance.

Next, expand the Advanced Details to show the User Data field.  Here we can provide some PowerShell commands to download the exe file and execute it.  Here is the UserData to include, modifying the S3 bucket location to your environment:

To join newly launched instances to a domain you need to make use of UserData, which allows you to run scripts during the initial startup of the launch.
By using the UserData you can run commands. For our case, we will be executinig an EXE to join to the domain.

<powershell>
Set-ExecutionPolicy unrestricted -Force
New-Item c:/temp -ItemType Directory -Force
set-location c:/temp
read-s3object -bucketname examplebucket -key JoinDomain.exe -file JoinDomain.exe
Invoke-Item C:/temp/JoinDomain.exe
</powershell>

Here's what it looks like in the AWS Console:

Follow the remaining steps to complete launching of the instance. The instance will launch, download the exe, execute it and restart.

Trust relationship failed

Run this command:

Reset-ComputerMachinePassword -Server DC01 -Credential Domain01\Admin01

Recipient isn't the expected type

Set-Mailbox fails, update "Send on behalf" exchange 2010, recipient isn't the expected type.

Issue:

When update "sent on behalf", you may face error "recipient isn't the expected type"

Cause:

"Send On Behalf" access has not been removed while disabling the mailbox, they accounts are still pointing to shared mailbox, it usually happens with migrated exchange.

Solution:

Go to AD Users & Groups

Select email account having problem

Click properties and select Attribute Editor

select PUBLIC DELEGATES and remove the mailbox account you seen in the error.

Error message when you try to set an IP address on a network adapter

This error often occurs when cloning/copying virtual machines. You may have come across this error:

The IP address XXX.XXX.XXX.XXX you have entered for this network adapter is already assigned to another adapter Name of adapter. Name of adapter is hidden from the network and Dial-up Connections folder because it is not physically in the computer or is a legacy adapter that is not working. If the same address is assigned to both adapters and they become active, only one of them will use this address. This may result in incorrect system configuration. Do you want to enter a different IP address for this adapter in the list of IP addresses in the advanced dialog box?

There's a simple fix to this and requires following these steps from the command prompt.

1. Type set devmgr_show_nonpresent_devices=1, and then press Enter.
2. Type Start DEVMGMT.MSC, and then press Enter.
3. Click View, and then click Show Hidden Devices.
4. Expand the Network Adapters tree.
5. Right-click the dimmed network adapter, and then click Uninstall.

Here's the link to Microsoft's site for this error, Microsoft KB 269155

ExMon Unknown Start Trace Error (183)

When trying to run Exchange Monitor (ExMon) you may run into this error:

Here are the steps to fix this common problem.

• From the CMD prompt, Query the running traces:
  • Logman query -ets
  • You should see one called "Exchange Event Trace"
• To stop the Exmon trace, run this:
  • Logman stop “Exchange Event Trace” -ets
• You should now be able to run ExMon

MS Exchange abnormal log generation

We came across an issue where one of our Exchange 2010 databases was quickly filling with logs.  The server was generating 10-20 times more logs than average, which caused the database drive to fill and took the database offline.

The root cause, a user had updated their iPhone iOS which resulted in a sync error with ActiveSync.  I discovered this by using the below tips to locate the mailbox generating the logs and disabling the ActiveSync feature for the mailbox.  With ActiveSync disabled the logs stopped.  So a quick wipe of the Exchange account from the iPhone, rebooting the iPhone, and recreating the Exchange account on the iPhone resolved the issue.

If you have a single mailbox database you are lucky, and the Exchange User Monitor should be all you need. Download Exchange User Monitor (ExMon) from Microsoft:
http://www.microsoft.com/en-us/download/details.aspx?id=11461

Run the ExMon.exe application from "C:\Program Files (x86)\Exchange User Monitor".  With ExMon you can see the users mailboxes, their CPU usage, Bytes In, Bytes out, Log Bytes, etc.  You may quickly find a user with a large consistent CPU usage and large Log Bytes compared to other users, this may be your culprit.  Try disabling their Exchange ActiveSync within the Mailbox Features.  This will only stop the ActiveSync communication, the mobile device will reconnect when you re-enable the ActiveSync, so no need to reconfigure the mobile device if this doesn't resolve the log issue.  With ActiveSync disabled for the suspected log generating mailbox, monitor the logs for the database and verify they have returned to a normal generation rate.  If so, you found the user.  Delete the email account from the mobile device, restart the device, and then re-add the email account.

If you ever run into the 'Unknown Start Trace Error (183)' when starting ExMon, see this post for a fix.

If you have multiple databases, ExMon is not as easy as it will list all mailboxes for the server.  To filter per database, us the Exchange Management Shell and this powershell script.

Get-StoreUsageStatistics -Database <DatabaseIdParameter> | export-csv c:\temp\LogonStats.csv

Modify the <DatabaseIdParameter> to the name of your database, and also change the path for the export of the csv file.  You can find additional information on this command here:
http://technet.microsoft.com/en-us/library/dd876852.aspx