Photo by Silas Köhler on Unsplash

Best Practices Of Storing Encryption Keys in AWS Secrets Manager

“The only secrets are the secrets that keep themselves”- George Bernard Shaw

Leveraging a service like AWS Secrets Manager, to outsource secured storage and life-cycle management of secrets (like passwords, API keys, tokens, encryption keys, etc.) is becoming quite commonplace. Essentially, this practice keeps the application code clean and devoid of any sensitive information that might get leaked otherwise. The idea is to either use DevOps pipeline to fetch secrets and inject them at the time of deployment (primarily as environment variables) or use AWS SDK to retrieve secrets during application runtime and use them.

Normally, creation and retrieval of sensitive values from Secrets Manager is pretty straightforward. AWS has done a really good job in creating multiple ways to do so- AWS console, SDK, CLI & REST APIs. However, when it comes to storing/retrieving encryption keys (symmetric or asymmetric), one has to be a tad careful.

For instance, a very common error (in Node.js applications) resulting due to an incorrectly stored PEM encoded encryption key in AWS Secrets Manager, is shown below. Any character translation corrupts the key and following error is directly thrown from PEM libraries when we attempt to use the incorrectly stored key in a scenario like mutual TLS authentication (mTLS) or otherwise.

{ 
   “library”: “PEM routines”, 
   “function”: “get_name”, 
   “reason”: “no start line”, 
   “code”: “ERR_OSSL_PEM_NO_START_LINE”
}

In essence, encryption key files should be treated as sacrosanct. Hence we have to resort to other methods of creating and accessing these encryption keys as a best practice.

Use base64 encoding

For encryption keys, we’d like to avoid any character translation like removal of white spaces or control character or sequence of control characters. A reliable method is to encode the key with ‘base64’. In this following example, we assume that the private key file is named private.key (primarily for lack of imagination on my part 🙂) and it is PEM encoded. We could use a couple of commands to create base64 encoded secret.

(Note: Here we are assuming that AWS CLI has been installed & configured and a utility like base64 exists, which happens to be the case for Linux distributions)

b64key=$(base64 private.key)
aws secretsmanager create-secret --name myprivatekey --description "Private key file" --secret-string “$b64key”

The thing to watch for is that, here we are storing the key as a ‘secret-string’ . We could also create the secret as JSON object with multiple keys like private and public key-pair. However both these keys must be base64 encoded independently.

{ 
 “key”: “<base64 encoded private key contents>”,
 “cert”: “<base64 encoded certificate file contents>”
}

That way, there are no character loss due to unwanted translations and the values of these keys remain intact during storage-retrieval cycles.

Retrieving a base64 encoded secret

Retrieval and use of base64 encoded secret using AWS CLI is just a matter of getting hold of the secret and then passing it through a base64 decode cycle.

aws secretsmanager get-secret-value --secret-id myprivatekey --query ‘SecretString’ --output text | base64 -decode

Storing and retrieving SecretString using AWS SDK has been well documented by AWS and others. We will not cover the same in this post.

Typically in Node.js, the libraries that support usage of encryption keys (say for performing mTLS authentication, etc.) either accept these keys as a string representing a path to a file OR a Buffer. In the code snippet that follows, we are trying to create a Buffer from the private key retrieved from AWS Secrets Manager with the knowledge that it is base64 encoded. At this point, the ‘privateKeyBuffer’ could be used by any library to leverage the encryption key.

var privateKeyBuffer = Buffer.from(privateKey, 'base64');

Use a binary secret

Another way to safely store keys is to create binary secrets in AWS Secrets Manager. Unlike JSON based secrets, which can be easily created from AWS console, binary secrets does warrant some special treatment. For one, they cannot be created from AWS console. We will have to leverage either AWS CLI or AWS SDK to create a binary secret in AWS Secrets Manager. The best part is that, binary secrets are transparently encoded with base64 when they are stored in AWS Secrets Manager.

Here’s how to use AWS CLI to store a binary secret:

aws secretsmanager create-secret --name "myprivatekey" 
   --description "Private key file" 
   --secret-binary fileb://./private.key

Note that, here we are using ‘secret-binary’ option, instead of ‘secret-string’. Also we are passing a binary file to be stored. Alternatively, we can leverage AWS SDK to store binary secrets. Here is a sample Node.js code snippet that creates a binary secret from a private key.

const AWS = require('aws-sdk');
const fs  = require('fs');
const REGION_CODE = process.env.REGION_CODE || 'ap-south-1’;
var secretsmanager = new AWS.SecretsManager({region: REGION_CODE});
var params = {
    Name: 'myprivatekey’,
    Description: 'Private key file’,
    SecretBinary: fs.readFileSync(’./private.key’);
};
secretsmanager.createSecret(params, function(err, data){
    if(err){
        console.error("Error >> ", err, err.stack);
    }
    else{
        console.log("Data >> ", data);
    }
});

Retrieving a binary secret

We will have to issue the following command to retrieve a binary secret using AWS CLI:

aws secretsmanager get-secret-value --secret-id myprivatekey --query 'SecretBinary' --output text --profile sm_user | base64 --decode

Here is a sample function (in Node.js) that retrieves a binary secret from Secrets Manager and returns the corresponding Buffer that contains the private key.

const AWS = require('aws-sdk');
const REGION_CODE = process.env.REGION_CODE || 'ap-south-1’;
const fetchKey= async(secretid)=>{
    const SECRET_BINARY = 'SecretBinary';
    let request = new AWS.SecretsManager({region: REGION_CODE})
                              .getSecretValue({SecretId: secretid});
    let secretBinary=null;
    let t1 = new Date().getTime();
    await request.promise()
        .then(function(data){
            if(SECRET_BINARY in data){
                secretBinary = Buffer.from(data.SecretBinary); 
                let t2 = new Date().getTime();
                console.log("Keys fetched in "+ (t2-t1) + " msecs");
            }
            else{
                console.error("Expecting binary secret..not found");
                throw Error("Invalid secret- not binary");
            }
        })
        .catch(function(err){
            throw err;
        });
    return secretBinary;
};

Conclusion

Encryption keys are sensitive to character translation, hence it’s wise to base64 encode them before storing in AWS Secrets Manager. As such, either explicit base64 encoding/decoding schemes will have to be carried out or these could be stored as binary secrets which are automatically base64 encoded by AWS Secrets Manager.

Binary secrets lack the relative ease of storing more than one encryption key in a structured format like JSON. However, they do provide an easy way to automatically base64 encode the secrets during storage.

On the other hand, storing secrets as normal secret-string will have to be backed by explicit base64 encoding/decoding cycles, but they provide the versatility of composing logically related sensitive values as a single secret (for example: private and public key-pair), which in turn makes handling of such values easy with minimal remote calls to AWS Secrets Manager.