PTP with NTP backup with Solarflare PTP daemon

Recently I witnessed an unexpected issue with reundant PTP devices: an IGMP snooping issue on the switch side made both time appliances unavailable.

For some time I wanted to add NTP redundancy to my PTP setup, this was a good excuse to put some time on it. It is not the same, but when you are working with certain hardware it is a good failsafe mechanism.

From what I read in the official documentation, Chrony could be used with SFPTPD but with a few caveats. SFPTPD needs to be able to control Chrony either via external helper (prior to 3.8) or via socket on the newest releases.

Disclaimer: This configuration is still experimental and under testing.

This is my SFPTPD config file:

[general]
sync_module ptp ptp1
sync_module crny crny0
message_log syslog
stats_log syslog
clock_control no-step
selection_holdoff_interval 60
[ptp1]
ptp_mode slave
ptp_delay_mechanism end-to-end
ptp_network_mode hybrid
ptp_domain 28
priority 10
[ptp]
interface core
[crny0]
clock_control on
priority 20

The basic additions are Chrony sync module and its configuration, with higher priority than the PTP domain. Also, a hold off interval in between kicking Chrony in or out.

On the Chrony configuration side (without the comments):

allow 127.0.0.1 # only for local purposes
server 169.254.169.123 iburst prefer # AWS ntp or your preferred ntp source
rtcsync # keep the bios updates, useful for older hardware
bindcmdaddress /var/run/chrony/chronyd.sock # local control via socket
makestep 0.0 0 # no stepping
maxslewrate 10000 # roughtly 1 hour to slew 1 minute

 

After restarting both daemons, we can see SFPTPD recognising the additional clocksource and failing over to Chrony if PTP is not available:

systemd[1]: Started sfptpd.service - Solarflare Enhanced PTP Daemon.
sfptpd[3388701]: ntp: changed state from ntp-listening to ntp-disabled
sfptpd[3388701]: crny: unblocking system clock
sfptpd[3388701]: crny: changed state from ntp-listening to ntp-selection
sfptpd[3388701]: crny: changed state from ntp-selection to ntp-slave
sfptpd[3388701]: selection: rank 1: crny0 by rule state (2) <- BEST
sfptpd[3388701]: selection: rank 2: ptp1 <- WORST
sfptpd[3388701]: will switch to sync instance crny0 in 10 seconds if ptp1 does not recover
sfptpd[3388701]: ptp ptp1: failed to receive Announce within 12.000 seconds
sfptpd[3388701]: crny: enabled chronyd clock control
sfptpd[3388701]: selected sync instance crny0 (ptp1 was active for 16.355s)

It can be observed in SFPTPD logs the block / unblock chrony events to config it is able to talk to chrony via socket (you can also replicate with chronyc -h <socket path>):

$ sudo journalctl -u sfptpd -l | egrep -i block sfptpd[2343906]: crny: blocking system clock sfptpd[2343906]: crny: unblocking system clock

I need to do more tests on this configuration, but so far looks promising.

Additional documentation sources:

• /usr/share/doc/sfptpd/config/ptp_slave_chrony_fallback.cfg (local file on your SFPTPD server)
• https://docs.amd.com/r/en-US/ug1602-ptp-user/ptp_slave_chrony_fallback.cfg
• https://docs.amd.com/r/en-US/ug1602-ptp-user/Chronyd

 

Istio Ingress Gateway for direct node ingress with Kubernetes applications

Recently I migrated a few market data applications from nomad to EKS. 

These services are optimised to run within a small, simple footprint where SSL is offloaded to other components. Before the migration, this used to be an nginx companion using a sidecar pattern. On the new setup, Istio is the ingress tool of choice so there is no need to keep a sidecar.

With a traditional Istio setup, pretty soon we came to the realisation that we are introducing additional hops in our network and increasing service latency:

The network circuit was:

• Consumers connect to the NLB / Service definition in Istio
• The NLB sends the requests to the worker nodes running Istio Ingress
• Istio Ingress accesses the application service definition and routes the traffic to one of the instances
• The MD instance receives the request, processes the information
• The HTTP response is returned to the client

The new platform would be slower, since now we need to contact a newly introduced NLB and the intermediate Istio worker node.

Looking around at what was possible with Istio, there is a feature that allows you to configure edge ingress with Istio Ingress Gateway Summing up the additional configuration:

• Make use of external-dns for direct service resolution
• Make use of nodeport to proxy into this service type with automatic pod resolution
• Make use of Ingress Gateway helm deployment to have Istio running on the application nodes

The resulting picture is:

The resulting network circuit is now:

• Consumers resolve the endpoint via DNS which external-dns populates 
• The Edge Istio container receives the request, routes it internally or sends it to the next MD APP node
• The MD instance receives the request, processes the information
• The HTTP response is returned to the client

The implementation is relatively simple. Since I already had an Istio deployment, I created a new helm chart deploying the ingressgateway component of Istio:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
      istio: ingressgateway
  template:
    metadata:
      labels:
        app: istio-ingressgateway
        istio: ingressgateway
      annotations:
        inject.istio.io/templates: gateway
    spec:
      serviceAccountName: istio-ingress # shared account with Istio
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet # This will be needed for edge access

      nodeSelector:
        mylabels/ingressgateway: "true" # What labels will run Istio Ingress Gateway
      tolerations:
        # Any tolerations?

      containers:

  [... from here pretty much same as the original Istio ...]

After getting Istio Ingress Gateway running on the desired labels / nodes, we need to add the necessary settings to on the Market Data application deploymenty side. The application previously had these relevant components:

• Service definition (ClusterIP)
• Gateway
• VirtualService

First, modified the service to add the external-dns annotation adding the additional internal DNS. For this setup the application will have two endpoints:

• mymarketdata.localdomain.service.internal - the original dns endpoint
• igw-mymarketdata.localdomain.service.internal - Istio Ingress Gateway dns endpoint

Note: This is an addition to an existing service, so I will only highlight the modifications to a traditional service setup and not the entire setup.

On the existing Service definition, we need to add an annotation for external-dns:

apiVersion: v1
kind: Service
metadata:
  name: market-data-v1
  namespace: marketdata
  annotations:
    external-dns.alpha.kubernetes.io/hostname: igw-mymarketdata.localdomain.service.internal

 [...]

Since this is is a ClusterIP service type we will use nodeport to automatically map the pod IPs on the target DNS records. Sample nodeport configuration:

apiVersion: v1
kind: Service
metadata:
  name: nodeport-market-data-v1
  namespace: istio-system
  annotations:
    external-dns.alpha.kubernetes.io/hostname: igw-mymarketdata.localdomain.service.internal
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    istio: ingressgateway # or the app labels if more convenient or have ingress gateway running in more places
  ports:
  - name: https
    port: 443 
    targetPort: 443 # 8443 is the default unless we bind envoy to 443
    nodePort: 38443   # valid NodePort

Note: Nodeport operates on the range 30000-32767, and by default will bind to port 8443 unless envoy runs as root or is allowed to bind to low ports such as 443.

Now that we have the service modification and nodeport, we need to create the new Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-market-data-v1
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - igw-mymarketdata.localdomain.service.internal

      tls:
        mode: SIMPLE
        credentialName: [your cert config]

 and VirtualService:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-market-data-v1
  namespace: marketdata
spec:
  hosts:
    - igw-mymarketdata.localdomain.service.internal

  gateways:
    - istio-system/gateway-market-data-v1
  http:
    - route:
        - destination:
            host: market-data-v1.marketdata.svc.cluster.local

            port:
              name: [your container port label or port number with "number:"]

Now Istio should start to publish the running pod's IP on the new dns name. By default it will be a round robin, but we can expand this to least connections, random, etc with a DestinationRule.

CPU affinity and Kubernetes

These days I'm working on a fun project: migrate some trading components from other orchestrators to EKS. Pretty soon we hit a problem - network packet loss.

Applications that are part of a trading engine can be very picky with the underlying infrastructure, specially single threaded low latency components. They like cpu affinity, a fancy network space with expensive network cards whith kernel bypass, and they don't like interruptions.

The initial approach was checking network configuration, drivers, solarflare firmware upgrade, onload version upgrade, sockperf tests. All looking decent. CPU affinity configured as usual - isolcpu, nohz, pstates disabled, frequency scaling off etc.

Then started to look into what was the process doing. Some of the findings were:

• /proc/interrupts was showing interruptions from on the isolated cores from the network cards, which was weirf
• perf commands were showing cpu interruptions on the isolated cores
• Solarflare support confirmed the interruptions in the onload stacks being the probable cause of network loss

The first issue was easily solved, configuring irqbalance to ban the isolared cores from the list with IRQBALANCE_BANNED_CPUS.

The core interruption took a bit longer to figure out - it was all on the resource definition in Kubernetes and the cgroups. With other orchestrators, cpu isolation is easier to handle. But in Kubernetes we have to consider the QoS classes:

• Burstable class - when there are different values for CPU requests and limits
• Guaranteed class - when there are the same values for CPU / memory requests and limits
• Best effort class - when we omit the CPU resource definition for both requests and limits

Depending on the QoS class we land, we will or will not get shared cpu time on the cgroups.

Kubelet will orchestrate the cgroup slices dynamically based on our deployment settings. From what I observed with my tests, the first two classes will cause cpu interruptions. The last one, will be just fine.

Looking into the cgroups for the first two options:

Burstable manifest definition:

resources:

  limits:

    cpu: "2"

  requests:

    cpu: "1"

# cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podXXXXX.slice/cri-containerd-YYYYY.scope/cpu.max

200000 100000

Guaranteed menifest definition:

resources:

  limits:

    cpu: "1000m"

    memory: "1Gi"

  requests:

    cpu: "1000m"

    memory: "1Gi"

# cat /sys/fs/cgroup/kubepods.slice/kubepods-podYYYYY.slice/cri-containerd-XXXXXscope/cpu.max

100000 100000 

On both definitions there is a specific CPU runtime allocated per a interval. Looking at the onload stack we can see some interrupts:

onload_stackdump lots | grep interrupts

interrupts: 148378

When switching to best effort, we see some changes:

limits: (empty below)

# cat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-AAAAA.slice/cri-containerd-BBBBB.scope/cpu.max

max 100000

We can see a difference where the max possible runtime is assigned. With this configuration, no interruptions were observed.

Looking again at the onload stack stats, we can see can confirm we don't have any more interrupts:

onload_stackdump lots | grep interrupts

interrupts: 0

This is only part of the story, because the cgroups have all the cores added to the cpuset (not using the cpu management policy in Kubernetes yet). But due to our cpu isolation policies only processes with cpu affinity settings will go to the designated cores, else this would need addressing.

After removing all the cpu resource definitions and applying the new irqbalance configuration, the network looks happy and healthy.

Other approaches to this could have been testing kubelet cgroup configurations to define a custom slice, and / or disable the cpu and cpuset controllers from the slices - altho not ideal as slices will change over deployments. A manual temporary approach could be disabling the cpuset and cpu controllers from the slices:

echo '-cpuset' > /sys/fs/cgroup/kubepods.slice/kubepods-<CLASS>.slice/kubepods-XXXXX.slice/cgroup.subtree_control

echo '-cpu' > /sys/fs/cgroup/kubepods.slice/kubepods-<CLASS>.slice/kubepods-XXXXX.slice/cgroup.subtree_control

The migration journey to EKS has just begun, and I expect to find more problems to solve. From what I heard, some exchanges migrated to EKS then migrated back due to performance complications. This is going to be fun.

Internal clock issues, TSC falling back to hpet in Linux kernel 4.X

Recently I was having some issues at work with internal clocks.

For high performance systems, you might prefer using TSC (time stamp counter) instead of HPET (high precession event timer). There's a good explanation on these two in this Red Hat documentation page.

In my case, running kernel 4.19.0-XX-amd64, I was occasionally seeing this error in the logs:

kernel: [517300.909751] clocksource: timekeeping watchdog on CPU15: hpet retried 2 times before success

This means the cpu watchdog did not get a timely answer from the CPU, for 2 attempts. This can escalate further if there are more than 3 attempts (or the max attempts defined in clocksource.max_cswd_read_retries kernel parameter). The max skew allowed for this event is defined in the kernel, with the value of WATCHDOG_MAX_SKEW:

(extract of kernel 4.19 source kernel/time/clocksource.c)

/*

* Interval: 0.5sec Threshold: 0.0625s

*/

#define WATCHDOG_INTERVAL (HZ >> 1)

#define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4)

/*

 * Maximum permissible delay between two readouts of the watchdog

 * clocksource surrounding a read of the clocksource being validated.

 * This delay could be due to SMIs, NMIs, or to VCPU preemptions.

 */

#define WATCHDOG_MAX_SKEW (100 * NSEC_PER_USEC)

If the readings take longer than the defined max attempts, you may see something like this:

kernel: [4027805.681972] clocksource: timekeeping watchdog on CPU8: hpet read-back delay of 113166ns, attempt 4, marking unstable

The internal clock will change to hpet, bringing down the performance of the server. In this case the watchdog answer was about 13 ish microseconds too late.

I couldn't find the reason why this random event would happen, even on an idle server. I tried changing CPU isolation settings, idle states, cpu frequency governor settings, etc. Moreover, being an unpredictable event, it was difficult to make sure the new settings had any real effect on the problem.

Eventually I was able to manually reproduce this with the following stress-ng command:

sudo taskset -c 0-15 stress-ng --timeout 180 --times --verify --metrics-brief --ioport 32 --sysinfo 32 --aggressive --schedpolicy 40 --cpu-load-slice 100

This was on a server with a single processor, 16 cores. It would fall back to HPET within 1 minute. But it has to pin to all available cores, else it won't flip.

Eventually the solution was upgrading to kernel 5.10. It has new kernel options to interact with the clock watchdog, like clocksource.verify_n_cpus. However, just updating kernel made this issue go away for me. The source looks fairly similar on the new kernel:

(extract of kernel 5.10 source kernel/time/clocksource.c)

/*

 * Threshold: 0.0312s, when doubled: 0.0625s.

 * Also a default for cs->uncertainty_margin when registering clocks.

 */

#define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 5)

/*

 * Maximum permissible delay between two readouts of the watchdog

 * clocksource surrounding a read of the clocksource being validated.

 * This delay could be due to SMIs, NMIs, or to VCPU preemptions.  Used as

 * a lower bound for cs->uncertainty_margin values when registering clocks.

 */

#define WATCHDOG_MAX_SKEW (100 * NSEC_PER_USEC)

The threshold is increased a bit, but the max skew still 100 microseconds. In kernel 6 we can see this value increased to 125. An alternative if this issue would happen again could be custom build my own kernel, increasing the max skew to something more accommodating to what my servers are reporting. Although that could complicate a bit upgrading kernels later on, and perhaps have an impact on overall performance.

In the end, besides making my clock source issue go away, I'm also observing more user time in the cpu usage space, and less IO wait. A good reminder of the benefits of trying new kernels.

Creating your own dynamic DNS domain in under 10 minutes

A few years back I subscribed to a discounted dynamic DNS service - which has been very useful tbh. It was time to renew (or keep updating those hosts manually each month), then realized I could do my own thing for a fraction of the service subscription cost (and the domain name would be mine for as long as I needed).

My solutions consists on a AWS account making use of route53, a domain purchased from namescheap and a cron job. For dynamic dns on surveillance cameras or storage appliances this might not work out of the box.

First thing, the domain. I wanted something super cheap and found an offer of less than a euro per year for a ".site" domain:

Went straight for 5 years, which was something around 5 euros.

I already had a working AWS account, if not you can find the steps in Amazon Web Services site.

Create a route53 public zone, with the name of the domain you purchased:

 Configure your hosting to make use of the DNS servers provided by AWS (red box):

Add the A records for your devices - i.e. AWS host, or home computer:

Now we have to configure an IAM user capable of updating the DNS records. Go to IAM accounts, groups, create a new one (i.e. dnsupdater) and provide a security policy similar to this one (depending on your needs):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:Get*",
        "route53:List*",
        "route53:Update*",
        "route53:ChangeResourceRecordSets",
        "route53:TestDNSAnswer"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Create a user, get the keys and add it to the group.

In your computer, install AWS cli tools with pip install awscli and follow the console instructions to use your credentials.

Setting up the script, I have two files in the folder; template.json and updatedns.sh

The template looks as follows:

{
   "Comment": "Updated record",
   "Changes": [
       {
           "Action": "UPSERT",
           "ResourceRecordSet": {
               "Name": "home.mytest.site",
               "Type": "A",
               "TTL": 300,
               "ResourceRecords": [
                   {
                       "Value": "XXX.XXX.XXX.XXX"
                   }
               ]
           }
       }
   ]
}

You'll have to enter your zone ID in the script:

#!/bin/bash

hostname="home.mystest.site"
current=`host home.mytest.site | awk '{ print $4}'`
external_ip=`curl -s checkip.dyndns.org | sed -e 's/.*Current IP Address: //' -e 's/<.*$//'`
template="/opt/scripts/dns/template.json"
zone_id="/hostedzone/<enter your zone ID>"
dns_template="/opt/scripts/dns/template.json"

echo "My DNS: $current"
echo "My IP: $external_ip"

if [ $current != $external_ip ]; then
       echo "updating zone!"
       sed -i -E "s/\"Value\"\: \".+\"/\"Value\": \"$external_ip\"/g" $dns_template
       aws route53 change-resource-record-sets --hosted-zone-id $zone_id --change-batch file://$dns_template  
fi

Then add it as a cronjob:

$ crontab -l
*/30 * * * *    /opt/scripts/dns/updatedns.sh > /dev/null


All set !

Openshift CI images: S2i image builder + gitlab CI webhook 3 / 3 (Gitlab)

To configure the deployment key in gitlab, go to:

your repository -> settings -> Repository ->Deploy keys

Introduce the public key in the box that will open. Consider using these keys as read only.


Next, we have to configure the CI integration with the webhook URL we obtained before - you will need Gitlab runner.

Add this to your .gitlab-ci.yml:

openshift_job:
  stage: deploy
  tags:
   - docker
   script:
   - "https_proxy= curl -k -XPOST -H'Content-Type: application/json' -d'{}' <your build CI webhook URL>"
   only:
    - master

We rebuild the image:

$ oc start-build mydocsproject

And it should all work now :)

Openshift CI images: S2i image builder + gitlab CI webhook 2 / 3 (OpenShift)

Next step is creating the credentials. We'll use them to make openshift talk to gitlab - this will necessary only if your repository is authenticated.

$ ssh-keygen -b 4096 -t rsa -f mykey Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):  Enter same passphrase again:  Your identification has been saved in mykey.
Your public key has been saved in mykey.pub.

We upload the private key to openshift:

$ oc secrets new mygitlab-key ssh-privatekey=mykey

$ oc secrets add serviceaccount/builder secrets/mygitlab-key

$ oc secrets link builder mygitlab-key

Next we create the app configuration. You may need to customize the image location and your gitlab user / address. The new OpenShift project will be mydocsproject:

$ oc new-app <your repository>/builderimagename:latest:latest~user@gitlab.com:your_repo.git --name mydocsproject

This will generate a build image, that will fail because we haven't given access to our authenticated gitlab repository. We'll configure OpenShift to use the newly created credentials - gitlab configuration in the next post:

$ oc set build-secret --source bc/mydocsproject mygitlab-key

To provide gitlab with the webhook to trigger a deployment on push, we will need the build config's webhook:

$ oc describe bc/mydocsproject | grep -A1 "Webhook Generic"   

Webhook Generic:
       URL:           https://youropenshifturl/oapi/v1/namespaces/blablabla/generic

Next is to configure gitlab.

Openshift CI images: S2i image builder + gitlab CI webhook 1 / 3 (Docker image)

This week I had to migrate an old static site that was hosting a documentation repository. It worked using a git hook that ran rsync to the hosting server every time there was a push. Local git server, physical shared hosting server.

New location: openshift and gitlab.

Laziness within sent thoughts of running git clone from inside the container, but eventually decided to give a try to S2I.

First thing, setting an image with a basic nginx service:

.
├── build
├── Dockerfile
├── etc
│   ├── nginx.conf
│   ├── server_status.conf
│   └── staticdocs.contoso.com.conf
└── test
    └── test-app
        └── index.html

My Dockerfile:

FROM my-favourite-repository/centos-7:latest
MAINTAINER Andres Martin <andreu.antonio@gmail.com>

LABEL io.k8s.description="Platform for serving static HTML files" \      io.k8s.display-name="Nginx latest" \
     io.openshift.expose-services="8080:http" \      io.openshift.tags="builder,html,nginx"

RUN yum install -y nginx git vim curl && \
   yum clean all -y

LABEL io.openshift.s2i.scripts-url=image:///usr/local/s2i
COPY ./.s2i/bin/ /usr/local/s2i


COPY ./etc/nginx.conf /etc/nginx/nginx.conf
COPY ./etc/staticdocs.contoso.com.conf /etc/nginx/conf.d/staticdocs.contoso.com.conf
RUN mkdir -p /etc/nginx/inc.d/internal 

COPY ./etc/server_status.conf /etc/nginx/inc.d/internal/server_status.conf


EXPOSE 8080

#copy paste from other Dockerfiles nginxRUN mkdir -p /var/www/html
RUN chmod 777 /var/www/html # because why notRUN mkdir -p /var/log/nginx
RUN chmod g+xrw /var/log/nginxRUN rm -f /etc/nginx/sites-available/default.conf
RUN mkdir /nginx/cache -p
RUN mkdir /nginx/run -p
RUN chmod g+xrw -R /nginx


USER 1001120000

CMD ["/usr/local/s2i/usage"]

 My nginx.conf to allow non-privileged execution:

worker_processes 1;
daemon off;
pid        /nginx/run/nginx.pid;

events {
   worker_connections 1024;
}

http {

   include       /etc/nginx/mime.types;
   default_type  application/octet-stream;

   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '                      '"$http_user_agent" "$http_x_forwarded_for"';

   access_log  /nginx/access.log  main;
   error_log /nginx/error.log error;
   client_body_temp_path /nginx/cache 1 2;
   proxy_temp_path /nginx/cache 1 2;
   fastcgi_temp_path /nginx/cache 1 2;
   uwsgi_temp_path /nginx/cache 1 2;

   sendfile        on;

   keepalive_timeout  65;

   #gzip  on;

   client_max_body_size 25m;

   server_tokens off;

   include /etc/nginx/conf.d/*.conf;

   index   index.html index.htm;

}

After installing the s2i tools, you will have something like this in your folder:

├── .s2i
│   └── bin
│       ├── assemble
│       ├── run
│       ├── s2i
│       ├── sti -> s2i
│       └── usage

And you might have this in your run script:

exec nginx -c /etc/nginx/nginx.conf

At this point, to make sure all the above is right you can try a manual run:

Build your image as usual (sudo docker build . -f Dockerfile -t latest --pull; sudo docker tag latest my-registry/builderimagename:latest; sudo docker p
ush my-registry/builderimagename:latest)

$ sudo s2i build <your git repo> <builder image name:test> <result image name:latest>

i.e. my syntax:


$ sudo s2i build https://gitlab.com/myproject.git myrepo/mybuilderimage:latest myrepo/amartin-sample-app

$ sudo docker run -d -p 8080:8080 myrepo/amartin-sample-app <result image>

Now is time to configure OpenShift in the next post.

Customizing built-in modules of your Nginx deb packages

The other day I had to update my customer's Nginx due to CVE-2016-4450. Their installation is a bit fussy as they use the "set-misc" module, which is not included in the nginx-extras package in Ubuntu by default.

First step is download the source code:

$ apt-get source nginx-extras

You should get a nginx-<version> folder. Inside you can find the source code, debian control files, etc.

To add the "set-misc" module we download the latest build from the official repository inside debian/modules folder:

$ cd debian/modules

$ sudo wget https://github.com/openresty/set-misc-nginx-module/archive/master.zip && sudo unzip -d . master.zip && sudo rm -f master.zip

To add the module to the build options we need to edit the file debian/rules and add our options on the flavor configuration (in this case, nginx-extras adding --add-module=/home/amartin/nginx-1.4.6/debian/modules/set-misc-nginx-module-master \ at the end):

It would be a good idea to delete modules not being used to harden and light up the software.

Now we can build the packages:

$ sudo dpkg-buildpackage -b

Now you should have a set of built in packages with the options you have specified.

Detecting torrent clients (uTorrent and Azureus) on your local network with perl and nmap

This week I had some torrent issues at one of my customer's office - torrents consuming the network bandwidth.

Normally you would start to analyze the traffic and enumerate the top consumers, but this was not possible as the network equipment's maintenance was outsourced to a third party and was not possible to tinkle with it - funny thing, the  third party couldn't do anything about these torrents tho.

So I had only my desktop to deal with the issue. Based on the two most popular torrent clients I built a script that detects whether there's an Utorrent or Azureus client - under normal circumstances using high ports. I tried the versions released during the time I was writing this article, not sure if it will work with previous or immediate future versions.

You may want to change the email from and to in bold (email function) and install the additional perl packages using cpan (Email::MIME and LWP::UserAgent).

Syntax is $ perl script.pl <ip or network, nmap format> <port range, nmap format>

#!/usr/bin/perl

use strict;
use warnings;
use WWW::Curl::Easy;
use Email::MIME;
require LWP::UserAgent;


my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
$ua->agent('Petardo 1.0');

my @strings;
my @targets;
my $text;
my $server="";
my $pointer="";
my $agent="";


my $nmaptarget= $ARGV[0];
my $nmapports= $ARGV[1];
my $tmpfile="/tmp/torrents-Searcher.log";
my $logfile="/tmp/lolo3";
my $nmap="";


if ( ! defined $nmaptarget ) {
        print "Please give me a network to scan ( i.e. 172.172.172.0/24) \n";
        exit (1);
}

if ( ! defined $nmapports ){
        print "You didn't give me a port list to scan (nmap format). Will use 10000-65535\n";
        $nmapports="10000-65535";
}
scan();

open(FILE, "<", $tmpfile) || die "TMP File not found";
my @textlog = <FILE>;
close(FILE);

foreach (@textlog) {

        @strings = split / /;

        my $pointer=1;
        my $server=$strings[$pointer];
        while (defined $strings[$pointer] && $strings[$pointer] !~ /[1-9]{1,3}\.[1-9]{1,3}\.[1-9]{1,3}\.[1-9]{1,3}/ ) {
                $pointer++;
                next;
                }
        $server=$strings[$pointer];

        if (defined $server) {
                $server=~s/\(//;
                $server=~s/\)//;
        }
        $pointer++;

        my $arrSize = @strings;

        while ($pointer < $arrSize) {
                if ($strings[$pointer] ne "" && $strings[$pointer] =~ /[0-9]{1,5}\/tcp/) {
                        $strings[$pointer] =~ s/\/tcp//;
                        findutorrent($server,$strings[$pointer]);
                        findazureus($server,$strings[$pointer]);
                }
                $pointer++;
        }

}

sub findutorrent {
        my $url = "http://" . $_[0] . ":" . "$_[1]" . "/version";
        my $response = $ua->get($url);
        if ($response->is_success) {
                if ( $response->decoded_content =~ /uTorrent/ ) {
                        print $response->decoded_content;
                        print "Utorrent check URL " . $url . " gave code " . $response->code . "\n";
                        print "Found possible utorrent on IP ". $_[0] . " and port " . $_[1] . "\n";
                        emailme($_[0],$_[1],"utorrent");
                }
                else {
                        print "Probe success (" . $response->code . ") but no valid uTorrent response detected on port " . $_[1] . "\n";
                }

         }
         else {
             print "No joy on uTorrent: " . $response->status_line . " on port $_[1] \n";
         }
}

sub findazureus {
        my $url = "http://" . $_[0] . ":" . "$_[1]" . "/service/request1.php?p=789C258CB10EC2300C44FFC5334D44C74A0831302216BA554269E3A61124B1D2A4A022FE1D876EBEF7CEF70152062BAF14B5E1176706A6FD7FBA5BD9D19A5980BD2C129EB39F789A2283D3104C782723F043F5A93E393F594123552964255D46884C6F9910229FD2E2F72C6B8D801E566";
        my $response = $ua->get($url);
        if ($response->is_success) {
                if ( $response->content =~ /[^[:ascii]]/ && length($response->decoded_content ) > 100 && length($response->decoded_content) < 350 && ! $response->decoded_content =~ /SSH/ ) {
                        print $response->content . "\n";
                        print "Lenght: " . length($response->decoded_content) . "\n";
                        print "Azureus check URL " . $url . " gave code " . $response->code . "\n";
                        print "Found possible Azureus on IP ". $_[0] . " and port " . $_[1] . "\n";
                        emailme($_[0],$_[1],"azureus");
                }
                else {
                        print "Probe success (" . $response->code . "," . length($response->decoded_content) . ") but no valid Azureus response detected on port " . $_[1] . "\n";
                }

        }
         else {
             print "No joy on Azureus: " . $response->status_line . " on port $_[1] \n";
         }    
}


sub emailme {

        my $message = Email::MIME->create(
        header_str => [
                From    => 'no.reply@mydomain.com',
                To      => 'andres.martin@mydomain.com',
                Subject => 'torrent client found?',
                ],
        attributes => {
                encoding => 'quoted-printable',
                charset  => 'ISO-8859-1',
                },
        body_str => "Found possible ". $_[2] . " client on IP ". $_[0] . " and port " . $_[1] . "\n",
        );

        # send the message
        use Email::Sender::Simple qw(sendmail);
        sendmail($message);
}

sub scan {

        open (my $fh, '>', $tmpfile) or die "Can't write to file '$tmpfile' $!";
        print "Nmap scanning network " . $nmaptarget . " and ports " . $nmapports . "\n";
        my $nmap = <<`SHELL`;
/usr/bin/nmap -Pn -sT -T5 --open -max-rtt-timeout 50ms --host-timeout 10m -p $nmapports $nmaptarget \n
SHELL
        $nmap =~ s/\n/ /g;
        print $fh $nmap;
        print "Nmap finished scanning\n";
        close $fh
}

Deploying wordpress using a chef recipe and AWS OpsWorks

Although I am not a fan of Wordpress, many people are. Lately I had to set up multiple instances for my colleges, and even tho installing an instance takes just a few minutes, ends up being is a bit of a hassle to do it all by hand. I thought would be nice to automate Wordpress deployments in my environment.

Initially thought of a golden AMI image, with a set of instructions to update Wordpress and the OS on boot time, but using chef seemed more flexible. I did some research and found a nice cookbook from Kenta Yasukawa based on Apache. I copied most of his work, doing some modifications to use nginx + php fpm instead and other minor changes. My cookbook can be found at my github. You can configure your own user and passwords in the file attributes/default.rb:

Even tho MySQL by default is not exposed to the world, it is safer to establish your own user and passwords - you can clone the repo, modify the files the upload to your own repository or use S3:

default["mysql"]["root"] = "yourrootpass"
default["mysql"]["pass"] = "passforwpuser"
default["mysql"]["user"] = "yourwpuser"

With AWS OpsWorks, we can create our own stack of Wordpress servers. First, go to OpsWorks and create your first stack. Select your preferences, and after clicking in Advanced you will be able to specify the URL for the chef recipe (https://github.com/AndreuAntonio/chef-wp.git) in this case:

Create your layer with your personal preferences. I selected custom, as DB and FE are going to reside within the same server, but there are a lot more possibilities here:

Now edit the layer and specify what recipe you want to use and in what stage:

We will use the recipe Deploy_software_wordpress_nginx-mysql56::default during setup. Remember to check on the Security tab to specify the right security groups for this (i.e. HTTP open).

Now we can start deploying servers. Check the Instances tabs and deploy a new server:

Start the server. It takes a while to spin up. When ready, the status field will become a green online.

Click then on the public IP and Wordpress setup should show up:

To add more Wordpress servers, just deploy new instances and in a few minutes you'll have them running :)

This is just a test lab, for a real production environment there's other things that we should take care of, like load balancing, redundancy, backups etc. Don't take this example as it is to a production environment.

Fixing reports not showing in OpenVAS 8

Last week I was upgrading my OpenVAS installation and I realized the reports sections was empty. I tried to google for a solution but couldn't find anything useful, so I decided to share my findings here.

At first I thought could be an issue with GreenBone so I tried to fetch the report using the cli tools, but no joy:

$ omp -v -u amartin -w XXXXX -R e6feb760-c9c3-425d-9ef5-a861d0dad6d2 -f a3810a62-1f62-11e1-9219-406186ea4fc5

WARNING: Verbose mode may reveal passwords!

Will try to connect to host 127.0.0.1, port 9390...
Failed to get report.

After some debugging with strace I realized a command was being executed by openvasmd:

/bin/sh -c "su nobody -c "/bin/sh /usr/local/share/openvas/openvasmd/global_report_formats/c402cc3e-b531-11e1-9163-406186ea4fc5/generate ....

Checked the permissions on those files, user nobody had no rights to execute or read any files in /usr/local/share/openvas/openvasmd/global_report_formats. Open the permissions:

$ sudo chmod a+xr /usr/local/share/openvas/openvasmd/global_report_formats/ -R

And then OpenVAS reports were working again.

Fixing missing locale warnings in bash

Recently I have been getting some locale warnings when doing an scp (while trying the bash completion):

$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_SG.UTF-8
LANGUAGE=en_SG:en
LC_CTYPE="en_SG.UTF-8"
LC_NUMERIC="en_SG.UTF-8"
LC_TIME="en_SG.UTF-8"
LC_COLLATE="en_SG.UTF-8"
LC_MONETARY="en_SG.UTF-8"
LC_MESSAGES="en_SG.UTF-8"
LC_PAPER="en_SG.UTF-8"
LC_NAME="en_SG.UTF-8"
LC_ADDRESS="en_SG.UTF-8"
LC_TELEPHONE="en_SG.UTF-8"
LC_MEASUREMENT="en_SG.UTF-8"
LC_IDENTIFICATION="en_SG.UTF-8"
LC_ALL=

To fix it, just type sudo dpkg-reconfigure locales and select the locales you want to rebuild (in my case, en_SG.UTF-8). The warning should be gone.

Amazon VPC with Chef server in a separate VPC

This week I had to create a separate AWS account for an specific platform - isolated from the Chef server's network, accounting purposes. Since being at it, the new AWS account would have several VPCs with different environments (staging, live).

First complication with this is that the command knife ssh won't work for the servers in the new account. As long as the nodes have internet access they will be able to register into Chef and install the recipes all right, but they will register with the following information:

$ knife node show dev-http-01Node Name:   dev-http-01
Environment: _default
FQDN:   ip-10-0-1-89.us-west-2.compute.internal  <---****
IP:          10.0.1.89 <---****
Run List:    role[dev]
Roles:       dev
Recipes:     chef-client, keys-us-west-2, autoupdate_apt, ntp, Deploy_package_apache2-latest
Platform:    ubuntu 14.04
Tags:        

With my VPC settings (using amazon DNS and DHCP servers), even tho an elastic IP has been assigned it will register itself with the internal address.

One approach would be create a proxy or vpn connection to the chef server, so it can talk to this internal network. However I just need to knife ssh into a few hosts, so I created this Chef recipe that updates the FQDN with the actual external IP:

$ cat Deploy_script_setIP_VPC/recipes/default.rb #
# Cookbook Name:: Deploy_script_setIP_VPC
# Recipe:: default
#
# No Copyright
# Andres Martin andreu.antonio@gmail.com
template "/etc/init.d/if-config" do
 source "if-config.erb"
 owner "root"
 group "root"
 mode "754"
end
service "if-config" do
      supports :restart => true, :start => true, :stop => true, :reload => true
      action [ :enable, :start]
    end

$ cat Deploy_script_setIP_VPC/templates/default/if-config.erb
#!/bin/sh
case $1 in
        start)
        URL="http://ifconfig.me/"
        IP=`curl $URL`
        if [ -n "`nslookup $IP`" ]; then
                echo "IP resolved to $IP, setting hostname..."
                name=`nslookup $IP | awk '{ print $4 }' | grep amazonaws.com | cut -d "." -f 1`
                hostname $name
                fi
        ;;
        stop)
        echo "this won't work..."
        ;;
        *)
        echo "Only for start"
        ;;
esac

This script relies on the public service ifconfig.me (thanks guys for this website). The output should change as soon as the Chef client contacts Chef again:

$ knife node show dev-http-01Node Name:   dev-http-01
Environment: _default
FQDN:  ec2-XX-XX-XX-XX.us-west-2.compute.amazonaws.comIP:          10.0.1.89
Run List:    role[dev]
Roles:       dev
Recipes:     chef-client, keys-us-west-2, autoupdate_apt, ntp, Deploy_package_apache2-latest
Platform:    ubuntu 14.04
Tags:      

This script shouldn't be used in critical services tho - is not foolproof, just a quick fix for a certain scenario.