Auto register chef clients with EC2 auto scaling

I recently deployed an auto scale policy at work increasing front ends on demand. One of the challenges I found is to register each chef client on boot up and get it removed on termination.

First thing, the ami must contain - or must obtain on boot time trough scripts, there's some cool stuff the people have post in internet about this - these things:

• a chef client installation
• a valid client.rb file
• a role file
• the validation certificate (based on your chef setup)

In my case, my ami already has a chef client installed but with only these three files (the ones mentioned above) in the /etc/chef folder. My client.rb loos like this:

$ cat /etc/chef/client.rb

log_level        :warn

log_location     STDOUT

chef_server_url  "http://ec2-.us.compute.amazonaws.com:4000"

validation_client_name "chef-validator" 

My json file specifing the role - just one rule, frontend. Other cookbooks and roles can be added.

$ cat /etc/chef/first-boot.json

{"run_list":["role[frontend]"]}

As you may have noticed already my client.rb does not specify any node_name. We add this parameter on boot time - each node, different node_name. We also create a knife configuration file to delete the node and client on shutdown - I place it on /root/.chef but you can choose anywhere else:

$ sudo cat /root/.chef/knife.rb

log_level                :info

log_location             STDOUT

client_key               '/etc/chef/client.pem'

validation_client_name   'chef-validator'

validation_key           '/etc/chef/validation.pem'

chef_server_url          'http://ec2-us-west-1.compute.amazonaws.com:4000'

cache_type               'BasicFile'

Ok, we have the skeleton ready. Now we just need to validate the client in order to register the client and node. For this, I use a script named /etc/init.d/chef-register :

#!/bin/sh
### BEGIN INIT INFO
# Provides:           chef-register
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:      2 3 4 5
# Default-Stop:      0 2 3 4 5 6
# Short-Description:  registers / deletes chef client
case $1 in
        start)
        echo "***** Registering node and client *****"
        instance_id=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`
        echo "node_name \"$instance_id\"" >> /etc/chef/client.rb
        echo "node_name '$instance_id'" >> /root/.chef/knife.rb
        /usr/bin/chef-client -j /etc/chef/first-boot.json
        ;;
        stop)
        echo "***** Deleting node and client *****"
        instance=`cat /etc/chef/client.rb | grep node_name | cut -d '"' -f 2`
        knife node delete $instance -c /root/.chef/knife.rb -y
        knife client delete $instance -c /root/.chef/knife.rb -y
        ;;
        *)
        echo $0 start or stop
        ;;
esac

Note that only once we validate the client we will obtain the client.pem on the /etc/chef folder.

To enable this script, we execute chmod +x /etc/init.d/chef-register  and update-rc.d chef-client defaults (for redhat based distros you will need to use chkconfig). On start, it will base the client name on the instance-id and put this parameter into both client.rb and knife.rb. On termination, it will delete the node and client - otherwise gets annoying when receiving timeouts deploying settings to the role.

That's it !

Optimizing linux TCP settings

Optimizing the TCP parameters can be tricky, and more when the servers are receiving hundreds of consecutive connections - i.e. load balancers.

Optimizing these settings depends a lot on the environment and nature of connection. As an instance, these are some of the settings I have:

The FIN in the tcp protocol defaults in 60 seconds. I tend to reduce it to 20, some people reduce it to less. In the end, is just a goodbye  from one IP to another :)

net.ipv4.tcp_fin_timeout = 20

For the TCP buffers I have the following settings:

(r = receive, w=send)
# 8MB for core mem max, default 65K
net.core.rmem_max = 8388608 
net.core.wmem_max = 8388608
net.core.rmem_default = 65536
net.core.wmem_default = 65536

#tcp socket buffers, minimum 4096K, initial 87K and max 8 MB
net.ipv4.tcp_rmem = 4096 87380 8388608 
net.ipv4.tcp_wmem = 4096 65536 8388608 
net.ipv4.tcp_mem = 8388608 8388608 8388608

Bear in mind that tcp_wmem overried net.core.wmem_default, in my case both are 65K.

Also, enable the tcp window scale

net.ipv4.tcp_window_scaling = 1
In my debian I have this file loading the settings on boot time:

$ cat /etc/sysctl.d/20-TCP-Tuning.conf

#FIN timeout to 20 sec
net.ipv4.tcp_fin_timeout = 20
# 8MB for core mem max, default 65K
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.rmem_default = 65536
net.core.wmem_default = 65536
#tcp sucket buffers, minimum 4096K, initial 87K and max 8 MB
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
#Enable TCP window scaling
net.ipv4.tcp_window_scaling = 1

To hot load these settings we can use sudo sysctl -p /etc/sysctl.d/20-TCP-Tuning.conf

Bash shell history commands

Some handy tips while working with bash history on linux.

The basics, defining the history file:

$ export HISTFILE = ~/.bash_history

Disabling history for your session:

$ export HISTSIZE=0

HISTSIZE affects the in-memory store, to set how many lines the file can contain we use HISTFILESIZE.

Adding a time stamp on the history list:

$ export HISTTIMEFORMAT='[%F %T]  '

History will look like this:

    1  [2013-06-09 10:40:12]   cat /etc/issue
    2  [2013-06-09 10:40:12]   clear
    3  [2013-06-09 10:40:12]   find /etc -name *.conf

%F Equivalent to %Y - %m - %d
%T Equivalent to time ( %H : %M : %S )

We can omit some comands to be displayed on the history file, such as the ones starting with a space, duplicates, or both:

$ export HISTCONTROL= < ignorespace | ignoredups | ignoreboth >

And to ignore certain command - in this case, command history:

$ export HISTIGNORE="history"

To repeat previous commands, we can press CTRL + R and start to write the command, the match in the history record will come up.

To get a full list of the current history we can use the history utility:

$ history[..] 1787  ssh sec02
 1788  ssh sec03
 1789  history
$

To delete the history:

$ history -c$ history
  791  history
$

to delete just one line, for example 791:

$ history -d 791

The bang bang (!!) is an existing feature to allow executing the latest commands easily. For example:

$ echo uno$ echo dos$ echo tres
$ history 1799  clear
 1800  echo uno
 1801  echo dos
 1802  echo tres
 1803  history
$

to execute the last command (history):

$ !!1799  clear
 1800  echo uno
 1801  echo dos
 1802  echo tres
 1803  history
$

To execute the second command in the list, starting from the last one:

$ !-2echo trestres$

We have something similar with the arguments, using !$. For example, we ping a host and then we telnet the argument using !$:

$ ping localhostPING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_req=1 ttl=64 time=0.036 ms
[...]
$ telnet !$ 22telnet localhost 22
Trying ::1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.0p1 Debian-4

There are other combinations with bang, for more information have a look to the linux documentation project.