Introduction

This week I had the opportunity to implement mutual authentication in a project that I'm working on. I've always been interested in authentication and cryptography, however I'd never explored this topic in any detail before, let alone implemented it myself on both the client side and server side.

Mutual authentication, or mTLS, is simply a way for both a client and a server to authenticate each other when they open a connection or communicate. The server proves to the client, by means of a certificate, that it is who the client expects, and vice versa. This provides a greater level of security, and protects against most person-in-the-middle (PITM) attacks. For a much better and more detailed explanation see the page on the subject by Cloudflare, Mutual TLS authentication.

In the end, it didn't take me that long to get a workable production implementation in place, however a few of the guides I found online seemed to be lacking in small ways, and didn't work for my particular environment and use case. Due to this, I thought it would be worth writing this blog post to help others who may be struggling with similar issues to the ones I encountered.

Overview

I'll begin by discussing the topic of certificate generation. This covers the generation of a root certificate, a server certificate, and a client certificate. This post only considers the 'self-signed' case, however it would be very similar when using an actual, recgonised, certificate authority. Following this, I'll discuss web server configuration, specifically using Nginx. I'll then discuss client side implementation in Flutter. Next, I'll discuss some of the common problems that I came across while implementing mutual authentication, and finally I'll wrap up with a summary.

I lent heavily on the information in the blog posts by Muhammet GÜMÜŞ, namely mTLS Client Authentication with NGINX  and Client Certificate Authentication (mTLS) with Flutter, and by Darshit Patel, How To Implement Two Way SSL With Nginx. They formed the basis for this blog post, I just made tweaks and improvements to suit my use case.

Certificate Generation

1. Generate root private key and certificate

# Root CA key
openssl genrsa -aes256 -out ca.key 2048
# Generate the certificate without using a CSR
openssl req -new -x509 -days 365 -key ca.key -out ca.crt
Generate private key and root certificate

-aes256 :  This is the encryption algorithm to use when encrypting your private key. You will be required to provide a password. If you leave this parameter off, your private key will not be encrypted, this is NOT recommended. For more info, see this discussion.

2048 : This is the keysize to use for your private key. This is the minimum size you should use, as 1024 is now considered too weak (reference).

-days 365 : The validity of the new ceritificate, one year is a safe default, especially for self-signed certificates.

After running these two commands and entering the required information you will have a self-signed root certificate and its associated private key. Keep these, and the private key password safe, as they will be used to sign server and client certificates in the next steps.

2. Generate server private key and certificate

# Server key and certificate sign request (CSR)
openssl genrsa -aes256 -out server.key 2048
openssl req -new -key server.key -out server.csr
# Sign the CSR with the CA certificate and key
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 1 -out server.crt
Generate private key and server certificate

-sha256 : This specifies the signature digest algorithm to use for the new certificate. Without this, OpenSSL may default to a weak algorithm, see Common Problems for more information.

-CA and -CAkey : These two parameters specify the filename of the root certificate and its corresponding private key, that were generated in step 1.

Important: When entering the CSR information, make sure you enter a DIFFERENT Organisation than you used in the root certificate, in step 1. For example, you could use <COMPANY NAME> for the root certificate, and "<COMPANY NAME> - Server" for the server. If you use the same organisation, verification of the certificate will fail.

You should now have the private key and a newly generated server certificate.

3. Generate client private key and certificate

# Client key and CSR
openssl genrsa -aes256 -out client.key 2048
openssl req -new -key client.key -out client.csr
# Sign the client CSR with CA certificate and key
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt
Generate private key and client certificate

Important: Make sure you use a different Organisation name from your root certificate, as described in step 2.

At this stage you can verify the client, or server, certificate against the root certificate, by using:

openssl verify -verbose -CAfile ca.crt client.crt
Verify a certificate

This command will produce a failure if you generated your client certificate with the same Organisation name as your root certificate.

You should now have a private key and client certificate.

Web Server (Nginx) Configuration

Now that you have the required certificates and their private keys, the next step is to setup your webserver in order to utilize and validate them.

In this case I use Nginx, setup as a reverse proxy. It will forward authenticated incoming requests to a backend service to handle them. It's also possible to set this up in different ways depending on your requirements.

Below is a example of a minimal Nginx configuration file that you can use in order to implement mutual authenticattion. You will likely need to incorporate the relevant config options in your existing Nginx file. Make sure you copy server.crt, server.key, and ca.crt,  to a location where Nginx can access it, e.g. under /etc/nginx/certs/.

http {
    server {
        listen 443 ssl;
        server_name <SERVER_NAME>;

        # See https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name
        proxy_ssl_server_name on;

        # Server SSL certificate and private key
        ssl_certificate     /etc/nginx/certs/server.crt;
        ssl_certificate_key /etc/nginx/certs/server.key;
        # Contains the password to decrypt the key above
        ssl_password_file /var/lib/nginx/ssl_passwords.txt;

        # The CA cert against which the client will be validated
        # The Client certificate might be created from a different
        # CA than the Server one      
        ssl_client_certificate /etc/nginx/certs/ca.crt;

        # Validate client certificate (or optional_no_ca for no validation)
        ssl_verify_client on;

        # Number of intermediate certificates to verify.
        # https://cheapsslsecurity.com/p/what-is-ssl-certificate-chain/
        ssl_verify_depth 2;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
        ssl_prefer_server_ciphers off;

        location / {
            proxy_set_header   X-Forwarded-For $remote_addr;
            proxy_set_header   Host $http_host;
            
            # certificate verification information
            # If client cert is verified, this will be 'SUCCESS'
            proxy_set_header VERIFIED $ssl_client_verify;
            # client certificate information (DN)
            proxy_set_header DN $ssl_client_s_dn;
            # Replace with the URL of the service
            proxy_pass         "http://<BACKEND_API_HOST>:<PORT>";
        }
    }
}
Sample Nginx configuration

Some of the options in this config were set using guidance from Mozilla, you can customise it however you like.

The most important configuration options for mutual authentication are:

  1. ssl_certificate <LOCATION OF SERVER CERTIFICATE>
  2. ssl_certificate_key <LOCATION OF SERVER PRIVATE KEY>
  3. ssl_password_file <LOCATION OF PRIVATE KEY PASSWORD FILE>
  4. ssl_client_certificate <LOCATION OF CLIENT CERTIFICATE AUTHORITY>
  5. ssl_verifiy_client on
  6. proxy_set_header VERIFIED $ssl_client_verify
  7. proxy_pass <URL_TO_FORWARD_TO>

Decrypting the server's private key

As you may have noticed above, we set the ssl_password_file config option to the path of a password file. This is necessary because when we created the server private key, we chose to encrypt it and provide a password. In order for Nginx to decrypt the private key, it needs this password.

There are various ways this can be done, but one of the simplest is to create a new file, e.g. ssl_passwords.txt and place the plaintext password for the server private key in it. Then copy it somewhere accessible to Nginx, and ensure you reflect its path in the config. More information on this subject can be found here.

Testing with cURL

At this point, you should be able to test your server configuration for Nginx by utilising cURL and providing the necessary certificates. This is described in more detail in this article.

curl -v -k --cacert ca.crt \
     --key client.key \
     --cert client.crt \
     https://<SERVER_HOST>:<SERVER_PORT>
cURL command for testing mutual authentication

This command will complete successfully if you've set everything up properly. Be sure to setup another server to handle the proxied request, which Nginx will pass to it (proxy_pass). If it does not work, review the above steps, and consult the Nginx debug error log for more detail on what the problem might be.

Client (Flutter) Configuration

I'll now cover how to implement the client side of mutual authentication in Dart/Flutter. I used the Dio library as it is an interesting project, and I found the implementation to be the easiest. It is discussed in more detail here. The process will be very similar in other languages. The general approach is to include the client certificate and private key in the client project, then read them in before use (providing the client key password), and finally instantiate a new HTTP client, which will be used for authenticated requests to the server.

// First read in the cert and key bytes
final List<int> certificateChainBytes =
      (await rootBundle.load('lib/assets/certs/client.crt')).buffer.asInt8List();
final List<int> keyBytes =
      (await rootBundle.load('lib/assets/certs/client.key')).buffer.asInt8List();

// Create Dio and set it up with a new HttpClient
Dio dio = new Dio();
(_dio.httpClientAdapter as DefaultHttpClientAdapter).onHttpClientCreate =
      (client) {
    SecurityContext sc = new SecurityContext(withTrustedRoots: true);
    sc.useCertificateChainBytes(certificateChainBytes);
    // Replace CLIENT_KEY_PASS with your client key password
    sc.usePrivateKeyBytes(keyBytes, password: CLIENT_KEY_PASS);
    HttpClient httpClient = new HttpClient(context: sc);
    // Allow self-signed certs, change this to false to disallow this
    httpClient.badCertificateCallback =
        (X509Certificate cert, String host, int port) {
      return true;
    };
    return httpClient;
  };
}

// Then perform an HTTP request using Dio
// Anything besides a 200 and 304 will result in a DioError
try {
  Response response = await dio.get("<URL TO QUERY>");
} on DioError catch (e) {
  throw e;
}
Code snippet for client side implementation in Flutter

Common Problems

Here I'll cover some of the problems that I came across while implementing mutual authentication between an Nginx reverse proxy and a Flutter client.

1. Weak Signature Digest Algorithm

As mentioned in the Generating Certificates section, if you leave off the -sha256 parameter in the openssl x509 command, you could run into trouble. This is due to the fact that, by default, openssl will use a signature algorithm that the latest version of Nginx considers weak, resulting in a verification error.

In my case, Nginx would fail with the following error:

[info] 28#28: *3 client SSL certificate verify error: (68:CA signature digest algorithm too weak) while reading client request headers, client: 172.29.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:443"

You don't have to use -sha256 here, des3 is also another acceptably secure option.

This error seems fairly common and is described in more detail here.

2. Nginx HTTP Response 495

This Nginx HTTP response code means there was an error with the SSL certificate. The error returned by Nginx is not very descriptive, even in the error logs. This code will be returned when Nginx fails to verify a client certificate for some reason.

In this case however, this error could mean that you generated your root and client certificates with the same Organisation, because of this Nginx considers it 'self-signed', as usually these two should not be the same.

[info] 21#21: *3 client SSL certificate verify error: (18:self signed certificate) while reading client request headers, client: 172.29.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:443"

This error also seems to be fairly common, as can be seen in this discussion.

Summary

In conclusion, a brief summary of all of the steps that I have discussed here, which are required in order to implement mutual authentication:

  1. Generate a root certificate and a private key.
  2. Generate server and client certificates and private keys.
  3. Configure Nginx to use the root and server certificates, and to verify client certificates in incoming requests.
  4. Implement reading in and sending the client certificate with HTTPS requests from the client

Thanks for reading. Any constructive feedback is most welcome 😊

Future Work

There are various aspects of mutual authentication between a server and a client that I believe can be optimised further than I have done here. I plan to spend some time researching these topics, and perhaps writing corresponding blog posts, in the future.

  1. Use of the ECDSA signature algorithm over RSA when generating certificates. Reference: https://blog.cloudflare.com/ecdsa-the-digital-signature-algorithm-of-a-better-internet/
  2. A more secure storage of the server private key password. For example, this could be fetching the key via a secrets vault distributed via encrypted environmental variables.
  3. A more secure way of password storage and key and certificate use in the client, for example: fetching the client private key password from a secrets vault or remote config service such as Firebase.
  4. TLSv1.3 support. For some reason Flutter/Dio does not negotiate a connection properly when only TLSv1.3 is enabled in Nginx. Investigate why this is and remedy it.
  5. Using certificates that are signed by a non-self-signed root cerificate, e.g. LetsEncrypt. With special consideration paid to the validity period.

Bonus!

A sample Dockerfile for Nginx to help you on your way:

FROM nginx:1.19

COPY nginx.conf /etc/nginx/nginx.conf

COPY server.crt /etc/nginx/certs/
COPY server.key /etc/nginx/certs/
COPY ca.crt /etc/nginx/certs/
COPY ssl_passwords.txt /var/lib/nginx/ssl_passwords.txt

RUN chmod 400 /etc/nginx/certs/server.key
RUN chmod 400 /var/lib/nginx/ssl_passwords.txt

EXPOSE 443
Dockerfile for Nginx

When running your Nginx container, use the following command if you need to debug:

[nginx-debug, '-g', 'daemon off;']
Nginx debug command

And, add this to your Nginx config file to enable debug level logging:

error_log /var/log/nginx/error.log debug;
Nginx debug level config