Apache, Varnish, Nginx SSL and Let's Encrypt

So, the title gives it sway, I just wanted to get what is simple setup working.... 
Well... It turned out to be not such a simple setup afterall, so I will share my experience!

Background

In the company I work we serve numerous Drupal websites using a "traditional" LAMP stack in the backend with Varnish for caching proxy and optionally Nginx when the SSL termiantion is needed.
If you're interested You can see on github the bolerplate for our varnish congifuration, and here below a simple schema of our network, as you can see it's nothing crazy, just an usual setup.

Let's Encrypt, Certbot and .well-known

At this point we needed to add SSL to some small services and we decided that we would use Let's Encrypt for it. Since I am in charge of the infrastructure I did the only thing i could do: RTFM!
I decided to use the suggested certbot to manage the certificate and actually certbot-auto since we run Ubuntu 14.04.

Since the certbot wouldn't be able to bind to neither port 80 (in use by varnish) nor on port 443 (in use by nginx) I decided to use the webroot plugin to validate the domain and get the certificates.
In this way the certboots creates the acme-challenge file into the .well-known directory in the document root of your virtual-host. Rather quickly then I realised that, once more, I encountered an issue with this approach: the document root of ell sites was on a completely differet machine and in the current server where the nginx deamon runs (which needed the certificate) I had nothing but that and varnish, both running as reverse proxy.

The solution: Varnish and Nginx magic!

Any request for files inside .well-known coming to Varnish (listening on port 80) would be redirected to the local nginx which would attempt to serve them from the local webroot. Follows here my Nginx + Varnish + Certbot configuration, keep in mind that Varnish config si based on the boilerplate i mentioned before (available on github).

1. Nginx .well-known host.

Port 80 is already taken from Varnish, so i decided to have nginx locally listening on port 8080, this would be available only for local connection sicne i would serve it via varnish. Also I wanted to have a single virtual host for all the small services that we had running so I decided to use the domain *.ssl.example.com

Create the nginx virtual-host configuration: ​# vim /etc/nginx/sites-available/star.ssl.example.com-8080

# This server is used to get the SSL certificates from Let's Encrypt
server {
    listen 8080;
    server_name ~^(?<shortname>.+)\.ssl\.example\.com$;

    # Use the host domain as directory webroot.
    location /.well-known {
            alias /var/www/nginx-letsencrypt/$shortname/.well-known;
    }
}

This allows me to simply have one single virtual host for all the domain, and for example if I need to have ssl on the subdomain esolitos.ssl.example.com I can just run mkdir -p /var/www/nginx-letsencrypt/widget/.well-known and use the certboot (see below) to get the cerficate.

2. Varnish catch and redirect .well-known requests

Next step is to have varnish redirecting the ACME requests to the nginx host. This is a simple matter of defining a backend and redirecting the request to that.

First I needed to define the backend: # vim /etc/varnish/backends/local-acme-catcher.vcl

# Custom local backend to handle theb ACME requests
# from Let's Encrypt
backend local_acme_catcher {
  .host = "127.0.0.1";
  .port = "8080";
}

Remember that the backend should be loaded and included: # echo 'include "backends/local-acme-catcher.vcl";' >> /etc/varnish/backends.vcl

Then somewhere in the config I had to catch the request and select this appropriate backend, i decided to do this in hosts.vcl since it's where I was already switching backends: # /etc/varnish/hosts.vcl

#
# Virtual Host Definition
#
# Switch the backend / director based on the required host

sub virtualhost__recv {

  #
  # [...] Several if/then/else to switch backends based
  # on hostname and other arbitrary parameters
  #

  # And finally catch ACME requests and set the local acme catcher as backend.
  if ( req.http.host ~ ".+\.ssl\.example\.com" && req.url ~ "^/.well-known/.*" ) {
    set req.backend = local_acme_catcher;
  }
}

3. Reload services, run Certbot and get the certificate!

It is now all set, I jsut nedeed to reload nginx and varnish (using the test/reload script budled in our bolierplate)

root@varnish-acme:~# service nginx reload
 * Reloading nginx configuration nginx
[ OK ]

root@varnish-acme:~# /etc/varnish/extra/varnishconfigtest.sh /etc/varnish/default.vcl
Config Check: All good!
Do you want to reload Varnish? [y/n]: y
 * Reloading HTTP accelerator varnishd
   ...done.

The last step is to gather the certificate and setup the final nginx virtualhost with it!

root@varnish-acme:~# /usr/local/bin/certbot-auto certonly --webroot -w /var/www/nginx-letsencrypt/esolitos/ -d esolitos.ssl.example.com

This will create the csr, key and certificate in the /etc/letsencrypt/live/$HOSTNAME directory.

root@varnish-acme:~# ls /etc/letsencrypt/live/esolitos.ssl.example.com
cert.pem  chain.pem  fullchain.pem  privkey.pem  README

Now it's just a matter of creating the actual virtual host for the site, I normally use the great Mozilla SSL Configuration Generator as bolierplate: # vim /etc/nginx/sites-available/esolitos.ssl.example.com

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name esolitos.ssl.example.com;

    ssl_certificate /etc/letsencrypt/live/esolitos.ssl.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/esolitos.ssl.example.com/privkey.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    # modern configuration. tweak to your needs.
    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

}

This is basically all there is to know!