Install NSQ on Debian with init.d and NGINX

NSQ is an awesome and extremely simple distributed message queue. You can simply use it by publishing messages — that is arbitrary text, I often use JSON — on a so called "topic". A second process can than attach to a topic to consume messages on a so called "channel". Each messages is forwarded into each channel (at least once). This blag post covers deployment of NSQ in the simplest scenario: one box running everything. There are three different services you need to run for a minimal NSQ deployment: nsqd, nsqlookupd, and nsqadmin. nsqd is the "main service". You are supposed to run it alongside every message producer and it communicates with the configured nsqlookupd which in turn then enables infrastructure discovery: it can be contacted by consumers to figure out where to connect to. Finally, nsqadmin is just a Web UI to carry out administrative tasks and get some high-level insights (like total number of processed messages). # Preparation and Download Let's first create a new user called nsq and download the self-contained binary package and unpack it into a sensible location while adjusting the permissions:

useradd -r -s /bin/false nsq

mkdir -p /usr/local/nsq/bin
chown -v nsq:nsq /usr/local/nsq

cd /tmp
wget 'https://s3.amazonaws.com/bitly-downloads/nsq/nsq-1.2.1.linux-amd64.go1.16.6.tar.gz'
tar -xzf nsq-*.tar.gz
rm nsq-*.tar.gz

mv -v nsq-*/bin /usr/local/nsq
rmdir -v nsq-*

chown -v nsq:nsq /usr/local/nsq/bin /usr/local/nsq/bin/*

The plan is to have the binaries in /usr/local/nsq/bin and any data NSQ needs to write to disk — it does so in case of a shutdown — to /usr/local/nsq/data. The next step now is to figure out the command line arguments necessary for the three services to properly run. Before we can actually do that, we need to make a couple of network topology decisions (and I really hope, I made the right ones): * Use client-side SSL certificates for authentication (I didn't look into nsq's internal auth mechanisms, but it is probable that I wouldn't have trusted them enough anyway). * Expose nsqadmin and the nsqd itself through NGINX which serves as a reverse proxy (there is a "NGINX with client-side certificate"-sidequest here). # NGINX: Client-Side Certificates Again, this is just a sidequest, so I'll leave you with the two scripts I wrote to create the certificate authority (CA) and the individual client-side certificates. Starting with creating the CA:

#!/bin/bash
set -xe
CURVE=secp384r1

# certificate authority (ca) key and certificate (crt)
openssl ecparam -out ca.pem.key -name $CURVE -genkey
openssl req -new -x509 -days 365 -sha512 -key ca.pem.key -out ca.pem.crt -subj '/C=CA/ST=British Columbia/L=Vancouver/O=Foo/OU=Security Department/CN=Bar'

# server key and certificate signing request (csr)
openssl ecparam -out server.pem.key -name $CURVE -genkey
openssl req -new -key server.pem.key -out server.pem.csr -subj "/C=CA/ST=British Columbia/L=Vancouver/O=Foo/OU=Security Department/CN=Bar"

# sign sever certificate with ca
openssl x509 -req -days 365 -sha512 -in server.pem.csr -CA ca.pem.crt -CAkey ca.pem.key -set_serial 01 -out server.pem.crt

The above script is not intended to just be copied&pasted, read it and understand what the different lines do and adjust them to your needs. There is a rabbit hole within this sidequest: I obviously wanted to use Elliptic-curve cryptography (ECC) because RSA sucks (factoring being sub-exponential and key sizes growing too fast and all that jazz). And with ECC you need to choose an actual curve you want to use. But not all browsers have support for all curves. So I ended up with a relatively lame choice of secp384r1 (don't ask my why I find this choice lame, it's a gut-feeling, probably because all my friends told me to use a brainpool curve). And then there's a second script I use to create the actual client-side certificates:

#!/bin/bash
set -xe
CURVE=secp384r1

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <client name>"
    exit 1
fi

# client key and csr
openssl ecparam -out client_$1.pem.key -name $CURVE -genkey
openssl req -new -key client_$1.pem.key -out client_$1.pem.csr -subj "/C=CA/ST=British Columbia/L=Vancouver/O=Foo/OU=Security Department/CN=$1"

# sign certificate
openssl x509 -req -days 365 -sha512 -in client_$1.pem.csr -CA ca.pem.crt -CAkey ca.pem.key -set_serial 01 -out client_$1.pem.crt

# convert to pkcs
openssl pkcs12 -export -in client_$1.pem.crt -inkey client_$1.pem.key -out client_$1.p12 -name "Foo client-side certificate for $1"

Apart from the client_*.pem.key (the client private key) and client_*.pem.csr (the client certificate) files, the above script will also bundle up a handy .p12 file that you can just double click under Windows to install it. All these scripts need some love: you should probably make the second one safe against spaces in $1 and also somehow increment the serial number after the 365 days. But this blag post is about NSQ! # How to run NSQ services Let's start with the lookup service:

/usr/local/nsq/bin/nsqlookupd -http-address 127.0.0.1:4161 -tcp-address 127.0.0.1:4160

Not much to say here, I just specified the host names to make sure to only bind to localhost and not expose this service directly to the internet (that is, without the NGINX reverse proxy). Now that we have a port fixed for nsqlookupd, we can start the main service:

/usr/local/nsq/bin/nsqd -http-address 127.0.0.1:4151 -tcp-address 127.0.0.1:4150 -lookupd-tcp-address 127.0.0.1:4160 -broadcast-address nsqd.example.com -broadcast-http-port 80

As above, I specified the host name to only bind to localhost as well as the previously configured lookupd address and port. I played around with the last two command line switches and now think it works like this: I told nsqd to announce itself under this publicly reachable host name which will be used by nsqadmin for example. Requests to the plain http port 80 will be redirected to https by NGINX. I didn't specify 443 directly because when I did, some component wanted to speak plain HTTP with an HTTPS endpoint. Again: there's some "opportunity for understanding", but we are already in a rabbit hole of a sidequest. Ok, so let's finally start the admin UI:

/usr/local/nsq/bin/nsqadmin -http-address 127.0.0.1:4171 -lookupd-http-address 127.0.0.1:4161 -http-client-tls-cert /path/to/client/certs/client_nsqadmin.pem.crt --http-client-tls-key /path/to/client/certs/client_nsqadmin.pem.key

Initially I thought the admin UI performs requests directly to the publicly available nsqd endpoints. But as it turns out, the server-component of the admin UI proxies those requests. That's why it needs a client-side certificate (which can be easily generated by calling ./create-client-certificate.sh nsqadmin). The following table gives a quick overview about the different ports and services: | Service | Port | Protocol | Public Endpoint | |--- |--- |--- |--- | | nsqlookupd | 4161 | HTTP | - | | nsqlookupd | 4160 | TCP | - | | nsqd | 4151 | HTTP | https://nsqd.exmaple.com | | nsqd | 4150 | TCP | - | | nsqadmin | 4171 | HTTP | https://nsqadmin.exmaple.com | # NGINX: Reverse Proxy Configuration For completeness sacke I will include the NGINX configuration here as well:

server {
    server_name nsqadmin.example.com;
    listen [::]:80;
    listen 80;

    # Let's Encrypt challenges
    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt/;
    }

    location / {
        return 308 https://nsqadmin.example.com$request_uri;
    }
}

server {
    server_name nsqadmin.example.com;

    listen 443 ssl;
    # Let's Encrypt certificate stuff here

    # client side certificate
    ssl_client_certificate /path/to/client/certs/ca.pem.crt;
    ssl_verify_client optional; # access restriction handled in location block below

    access_log /var/log/nginx/nsqadmin.example.com_access.log;
    error_log /var/log/nginx/nsqadmin.example.com_error.log info;

    charset utf8;

    location / {
        if ($ssl_client_verify != SUCCESS) {
            return 403;
        }
        proxy_pass http://127.0.0.1:4171;
    }
}

The configuration for nsqd looks exactly the same but with nsqd instead of nsqadmin and 4151 instead of 4171. # init.d Services In order to properly turn the above commands into services in init.d, we need to write service scripts able to spawn them into the background and also find the service to properly shut them down. For this, the start-stop-daemon command seems to be the way to go. Below is the /etc/init.d/nsqd script, the other two can simply be created by some smart copy&pasting:

#!/bin/bash

### BEGIN INIT INFO
# Provides:          nsqd
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: nsqd service
# Description:       nsqd is the daemon that receives, queues, and delivers messages to clients.
### END INIT INFO

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting nsqd..."
    start-stop-daemon --start --background --oknodo --user nsq --pidfile /run/nsqd.pid --make-pidfile --name nsqd --exec /usr/local/nsq/bin/nsqd --chdir /usr/local/nsq/data --chuid nsq -- -http-address 127.0.0.1:4151 -tcp-address 127.0.0.1:4150 -lookupd-tcp-address 127.0.0.1:4160 -broadcast-address nsqd.example.com -broadcast-http-port 80
    ;;
  stop)
    echo "Stopping nsqd..."
    start-stop-daemon --stop --oknodo --user nsq --pidfile /run/nsqd.pid --remove-pidfile --name nsqd --retry 5
    sleep 2
    ;;
  *)
    echo "Usage: /etc/init.d/nsqd {start|stop}"
    exit 1
    ;;
esac

exit 0

# Testing Functionality You can now fire up an NSQ tail command:

/usr/local/nsq/bin/nsq_tail -channel foo -topic test -nsqd-tcp-address 127.0.0.1:4150

And use curl to populate the test topic:

curl -d 'hello NTF' 'http://127.0.0.1:4151/pub?topic=test'

Full-disclosure here: using nsq_tail with the -lookupd-http-address 127.0.0.1:4161 switch instead of -nsqd-tcp-address 127.0.0.1:4150 doesn't work because nsqd announces itself under it's publicly available hostname (as instructed) and to access that, a client-side certificate would be necessary.