AWS EC2 instance auto scale policy with spot instances and automatic attachment to ELB

Some time ago I was having a look how to deploy this in our infrastructure. I found some interesting examples in this link, and with the official manual I managed to get the right policy for us, launching spot instances on demand and automatically registering them into our ELB. Here's my final take.

My variables are:

Security group: EE-FE-Stack-SG
SNS Topic: EE-FE-Stack
Minimum servers: 1
Scale up: 80% + CPU Used for 5 min Cooldown 300 sec
Scale down: 65% - CPU Used for 5 min Cooldown 600 sec
In bold the lines where you need to specify your load balancer name, AMI images and availability zone where you want to launch the instances.
In italic Userdata contains some instructions to try to fetch the cf-helper debian package and install it.

When launching the template, you will be asked to introduce the operators email, max spot price, min servers, max servers, security keys and some other parameters. Some of these values can be modified later on when re configuring the policy via web interface or cli tools.


{
"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "EE-FE Stack. Cooldown 300 sec up, 600 down.  [EE-FE-Stack-SG secutirygroup, EE-FE SNS topic]",

"Parameters" : {
"KeyName" : {
    "Description" : "Security key",
    "Type" : "String"
},

"InstanceType" : {
    "Type" : "String",
    "Default" : "m1.small",
    "AllowedValues" : [ "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.xlarge", "cc1.4xlarge" ],
    "Description" : "EC2 instance type (e.g. m1.large, m1.xlarge, m2.xlarge)"
},

"SpotPrice": {
    "Description": "Spot price for application AutoScaling Group",
    "Type": "Number",
    "MinValue" : ".03"
},

"MinInstances" : {
  "Description" : "The minimum number of Workers",
  "Type" : "Number",
  "MinValue" : "0",
  "Default"  : "0",
  "ConstraintDescription" : "Enter a number >=0"
},

"MaxInstances" : {
  "Description" : "The maximum number of Workers",
  "Type" : "Number",
  "MinValue" : "1",
  "Default"  : "4",
  "ConstraintDescription" : "Enter a number >1"
},

"OperatorEmail": {
  "Description": "Email Address",
  "Type": "String"
}
},

"Mappings" : {
"AWSInstanceType2Arch" : {
  "t1.micro"    : { "Arch" : "64" },
  "m1.small"    : { "Arch" : "64" },
  "m1.medium"   : { "Arch" : "64" },
  "m1.large"    : { "Arch" : "64" },
  "m1.xlarge"   : { "Arch" : "64" },
  "m2.xlarge"   : { "Arch" : "64" },
  "m2.2xlarge"  : { "Arch" : "64" },
  "m2.4xlarge"  : { "Arch" : "64" },
  "m3.xlarge"   : { "Arch" : "64" },
  "m3.2xlarge"  : { "Arch" : "64" },
  "c1.medium"   : { "Arch" : "64" },
  "c1.xlarge"   : { "Arch" : "64" },
  "cc1.4xlarge" : { "Arch" : "64HVM" },
  "cc2.8xlarge" : { "Arch" : "64HVM" },
  "cg1.4xlarge" : { "Arch" : "64HVM" }
},

"AWSRegionArch2AMI" : {
  "us-east-1"      : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "us-west-2"      : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "us-west-1"      : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "eu-west-1"      : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "ap-southeast-1" : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "ap-southeast-2" : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "ap-northeast-1" : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" },
  "sa-east-1"      : { "32" : "NOT_YET_SUPPORTED", "64" : "<Your AMI here>", "64HVM" : "NOT_YET_SUPPORTED" }
}
},

"Resources" : {
"NotificationTopic": {
  "Type": "AWS::SNS::Topic",
  "Properties": {
    "DisplayName" : "EE-FE-Stack",
    "Subscription": [ {
        "Endpoint": { "Ref": "OperatorEmail" },
        "Protocol": "email" } ]
  }
},

"WebServerGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "AvailabilityZones" : [ "us-west-1a" ],
    "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
    "MinSize" : { "Ref" : "MinInstances" },
    "MaxSize" : { "Ref" : "MaxInstances" },
    "LoadBalancerNames" : [ "<your load balancer here>"],
    "NotificationConfiguration" : {
      "TopicARN" : { "Ref" : "NotificationTopic" },
      "NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_LAUNCH_ERROR","autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"]
    }
    }
},

"CfnUser" : {
    "Type" : "AWS::IAM::User",
    "Properties" : {
        "Path": "/",
        "Policies": [ {
            "PolicyName": "root",
            "PolicyDocument": { "Statement": [ {
                "Effect":"Allow",
                "Action":"cloudformation:DescribeStackResource",
                "Resource":"*"
            } ] }
        } ]
    }
},

"HostKeys" : {
    "Type" : "AWS::IAM::AccessKey",
    "Properties" : {
        "UserName" : { "Ref" : "CfnUser" }
    }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "Comment" : "Create a single webserver",
    "AWS::CloudFormation::Init" : {
      "config" : {
        "packages" : {
            "yum" : {

            }
        },
        "files" : {

        }
      }
    }
  },
  "Properties" : {
    "KeyName" : { "Ref" : "KeyName" },
    "SpotPrice" : { "Ref" : "SpotPrice" },
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
                                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", {     "Ref" : "InstanceType" },
                                      "Arch" ] } ] },
    "SecurityGroups" : [ "WEB" ],
    "InstanceType" : { "Ref" : "InstanceType" },
    "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash\n",
      "wget -O /opt/aws-cfn-bootstrap.deb http://pkg.camptocamp.net/staging/pool/sysadmin/a/aws-cfn-bootstrap/aws-cfn-bootstrap_1.3-1_all.deb\n",
      "dpkg -i /opt/aws-cfn-bootstrap.deb\n",
      "# Install the Worker application\n",
      "/opt/aws/bin/cfn-init ",
      "         --stack ", { "Ref" : "AWS::StackId" },
      "         --resource LaunchConfig ",
      "         --configset ALL",
      "         --region ", { "Ref" : "AWS::Region" }, "\n"
    ]]}}      
  }
},


"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "300",
    "ScalingAdjustment" : "1"
  }
},
"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "600",
    "ScalingAdjustment" : "-1"
  }
},

  "WorkerThreadHigh": {
   "Type": "AWS::CloudWatch::Alarm",
   "Properties": {
      "AlarmDescription": "Scale-up if Worker Thread Vs. Idle Percent > 80% for 5min",
      "MetricName": "CPUUtilization",
      "Namespace": "AWS/EC2",
      "Statistic": "Average",
      "Period": "300",
      "EvaluationPeriods": "2",
      "Threshold": "80",
      "AlarmActions": [ { "Ref": "WebServerScaleUpPolicy" } ],
      "Dimensions": [
        {
          "Name": "AutoScalingGroupName",
          "Value": { "Ref": "WebServerGroup" }
        }
      ],
      "ComparisonOperator": "GreaterThanThreshold"
    }
  },
  "WorkerThreadLow": {
   "Type": "AWS::CloudWatch::Alarm",
   "Properties": {
      "AlarmDescription": "Scale-down if CPU < 65% for 5 minutes",
      "MetricName": "CPUUtilization",
      "Namespace": "AWS/EC2",
      "Statistic": "Average",
      "Period": "300",
      "EvaluationPeriods": "2",
      "Threshold": "65",
      "AlarmActions": [ { "Ref": "WebServerScaleDownPolicy" } ],
      "Dimensions": [
        {
          "Name": "AutoScalingGroupName",
          "Value": { "Ref": "WebServerGroup" }
        }
      ],
      "ComparisonOperator": "LessThanThreshold"
    }
  }
}

}

DNS error after migrating Chef from version 10 to 11

This week I'm testing any possible errors if we migrate our Chef to version 11.0.6 - I know there's version 11.0.8 available but is giving out a ruby timezone error on ubuntu that I need to deal with yet.

So far the migration seems smooth, following the steps on the official guide. Restoring the backup was ok and a testing client was able to communicate with the server. However, after modifying a cookbook I was getting this error when applying the template change:

FATAL: SocketError: template[/etc/rsyslog.d/22-XXXXXX.conf] (Deploy_configuration_na_olive-logging-rsyslog::default line 13) had an error: SocketError: Error connecting to https://ip-10-XX-XX-XX.us-west-5.compute.internal/bookshelf/organization-00000000000000000000000000000000/checksum-b3f32a70cedbe6de9ac38?AWSAccessKeyId=XXXXXXXXX&Expires=1382603029&Signature=XXXXXXXX - getaddrinfo: Name or service not known

Straigh forward we see the message getaddrinfo: Name or service not known. Being an AWS EC2 instance, it was unable to resolve the name because the servers are in different geographic regions. However that's not the issue, the problem is that the chef server is using the internal DNS for it's settings.

To rectify this, modify (or create) the file /etc/chef-server/chef-server.rb with this content:

server_name = "<your chef server external DNS>"
bookshelf['url'] = "https://#{server_name}"
bookshelf['vip'] = server_name
nginx['url'] = "https://#{server_name}"
nginx['server_name'] = server_name
lb['api_fqdn'] = server_name
lb['web_ui_fqdn'] = server_name
api_fqdn = server_name

then, as the migration user execute a reload of settings and a server restart:

$ sudo chef-server-ctl reconfigure ; sudo chef-server-ctl restart

Now try again to apply the changes on the client. If it doesn't work, try to kill all chef processes and start them from scratch - for some reason I needed to do this as my chef processes were not being killed..

Creating your own Debian packages

With this example I'll create a basic Tor browser bundle debian package - there are real maintained debian packages for tor this is only for testing purposes.

We will install the content into /usr/local and will create some links to launch the software into /usr/local/bin.

First download the latest Tor package (I assume we are using 64 bits) and extract the files in /tmp/tor-browser_en-US

Now in your home (or testing) folder, create a subfolder named "tor-browser-bundle". Now create the subfolder DEBIAN and usr/local:

$ mkdir -p tor-browser-bundle/DEBIAN

$ mkdir -p tor-browser-bundle/usr/local

Now copy the /tmp/tor-browser_en-US to the folder we just created:

$ mv /tmp/tor-browser_en-US tor-browser-bundle/usr/local

 Now, we create the file DEBIAN/control containing this template:

Package: tor-bundle-browser
Priority: optional
Section: devel
Installed-Size: 120
Maintainer: Andreu Martin
Architecture: amd64
Version: 0.1
Depends: libc6 (>= 2.0)
Description: Tor bundle browser test

Now we create the file tor-bundle-browser/DEBIAN/postinst with the post installation tasks. This file must have permissions  >=0555 and <=0775.

$ cat tor-browser-bundle/DEBIAN/postinst 

ln -s /usr/local/tor-browser_en-US/start-tor-browser /usr/local/bin/

cat <<-EOF > /usr/local/bin/start-tor-firefox

#!/bin/sh

/usr/local/tor-browser_en-US/App/Firefox/firefox -no-remote -profile /usr/local/tor-browser_en-US/Data/profile

EOF

chmod 755 /usr/local/bin/start-tor-firefox

          $ chmod 775 tor-browser-bundle/DEBIAN/postinst 

 Now we are ready to build the .deb:

          $ dpkg-deb -z9 -Zgzip --build tor-browser-bundle

(-z9 specifies compression level - 0 to 9, -Zgzip for compression type - gzip, xz, bzip2, lzma, or none)

 Now we have a tor-browser-bundle.deb ready to go.

Additional reading on debian packages on the official manual.