Using secrets in Azure Container Instances

Introduction


At work I am involved in one of our internal project development. This project is a relatively small (less than 30 pages) ASP.NET Core 3.1 Razor Pages application. From it’s first days application is hosted in Azure using Azure Container Instance, Azure SQL Database and Azure Key Vault (continues integration and deployment pipelines are powered by Azure DevOps).

There is nothing special in such setup, however as it happens with setups of this kind, there is one thing that has to be managed prior to implementing it – “How to securely pass secrets to Azure Container Instances and securely read secrets stored in Azure Key Vault (ex. read database connection string) from within Azure Container Instances?”

Finding the answer was my responsibility, so I have created a dummy project and dived into Azure documentation. This post contains results of my findings composed, as I hope, in easy and understandable format.

Disclaimer

All approaches described here represent a secure way to pass secrets to Azure Container Instances (ACI) and read secrets from within ACI. These approaches are secure in a meaning that passed secrets are protected from the unauthorized access and aren’t included in any kind of logs of read queries. However, they don’t address role-based access control (RBAC) configuration, so as it will be demonstrated, a person with full access to ACI will be able to execute container exec command to view and manipulate these secrets.

Passing secrets using secret volume


Supported operating systems: Linux

secret volume is a temporary RAM storage[1] which is mounted and made accessible to containers inside a container group. All secrets are stored on volume in form of files where file name is treated as secret name and file content is treated as secret value. These files aren’t accessible through Azure API and their contents aren’t included in any logs or diagnostics.

Secret volume is configured as part of ACI deployment.

The way of interpreting a directory as config where file name is a key and file content is a value is known and ‘key per file’ approach. This approach becomes more and more popular, so it has a built in support in .NET ecosystem in form of configuration provider.

Author’s note

Let’s see how we can pass two secrets: secret_1 and secret_2 to ACI using Azure CLI’s az container create command:

az container create `
  --resource-group <resource group>`
  --name <name> `
  --image <image> `
  --os-type Linux `
  --secrets secret_1="value_1" secret_2="value_2" `
  --secrets-mount-path /mnt/secrets

The secrets are specified as space-separated list of key=value[2] items in –secrets parameter (line: 6) while the volume mount path is specified in –secrets-mount-path[3] (line: 7).

When deployment is finished we can use az container exec command to examine content of secret volume:

az container exec `
  --resource-group <resource group> `
  --name <name> `
  --exec-command "/bin/bash"

# Bash command 'cat'

>:/app# cat /mnt/secrets/secret_1

# Outputs

value_1

In .NET application code these secrets can be consumed using Microsoft.Extensions.Configuration.KeyPerFile package:

// This code uses the following NuGet packages:
// - Microsoft.Extensions.Configuration.KeyPerFile

public class Program
{
  public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
      .ConfigureWebHostDefaults(webBuilder =>
      {
        webBuilder.ConfigureAppConfiguration(
          (ctx, config) =>
          {
              config.AddKeyPerFile("/mnt/secrets", false);
          });
        webBuilder.UseStartup<Startup>();
      });
}

The false argument in config.AddKeyPerFile() (line: 13) invocation indicates that this configuration source is optional, and, in cases when /mnt/secrets directory doesn’t exist, can be ignored.

Contention points


The main benefit of passing secrets using secret volume is a built in support from Azure ecosystem. However there are several points of contention.

Secrets are specified as part of deployment


Please see Appendix A. Common contention points section for more details.

Application code is tied to how secrets are stored


Please see Appendix A. Common contention points section for more details.

Passing secrets using secure environment variables


Supported operating systems: Windows, Linux

Environment variables are widely used for dynamic application configuration, however passing sensitive information through environment variables is a security risk, because they aren’t designed to keep sensitive information and usually aren’t protected by infrastructure. To leverage simplicity of environment variables and mitigate the security risks ACI supports secure environment variables.

Secure environment variables are environment variables which are protected by Azure infrastructure i.e. they aren’t exposed through Azure API and their contents aren’t included in any logs or diagnostics.

Secure environment variables are configured as part of ACI deployment.

Here is an example of passing one native environment variables public_1 and one secure environment variable secret_1 using az container create command:

az container create `
  --resource-group <resource group>`
  --name <name> `
  --image <image> `
  --os-type <os type> `
  --environment-variables public_1="public_2" `
  --secure-environment-variables secret_1="secret_1"

Both native and secure environment variables are specified as space-separated list of key=value[2] items in –environment-variables (line: 6) and –secure-environment-variables (line: 7) parameters correspondingly[4].

When deployment is finished we can examine ACI environment variables using az container show command:

az container show `
  --resource-group <resource group> `
  --name <name> `
  --query 'containers[].environmentVariables'

# Outputs 

[
  [
    {
      "name": "public_1",
      "secureValue": null,
      "value": "public_2"
    },
    {
      "name": "secret_1",
      "secureValue": null,
      "value": null
    }
  ]
]

As you can see show command conceals value of secret_1 environment variable. However, we still can examine the ACI environment variables using using az container exec command:

az container exec `
  --resource-group <resource group> `
  --name <name> `
  --exec-command "/bin/bash"

# Bash command 'env'

>:/app# env

# Outputs

public_1=public_2
secret_1=secret_1

In .NET application code these secrets can be consumed in the same way as native environment variables (because in fact, secure environment variables are native environment variables) using Microsoft.Extensions.Configuration.EnvironmentVariables package:

// This code uses the following NuGet packages:
// - Microsoft.Extensions.Configuration.EnvironmentVariables

public class Program
{
  public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
      .ConfigureWebHostDefaults(webBuilder =>
      {
        webBuilder.ConfigureAppConfiguration(
          (ctx, config) =>
          {
              config.AddEnvironmentVariables();
          });
        webBuilder.UseStartup<Startup>();
      });
}

It should be noted, that in most .NET applications there is no need to manually include environment variables configuration source (i.e. reference Microsoft.Extensions.Configuration.EnvironmentVariables package and invoke config.AddEnvironmentVariables method) because it is automatically added in default host builder configuration (i.e. when IHostBuilder is created using Host.CreateDefaultBuilder method).

Contention points


Secrets are specified as part of deployment


Please see Appendix A. Common contention points section for detailed explanation.

Application code is tied to how secrets are stored


Please see Appendix A. Common contention points section for more details.

Accessing secrets in Azure Key Vault using Managed Identity


Managed identity – is a feature of Azure Active Directory (hereinafter AAD) which allows you to assign an identity to a Azure resource and then use this identity to access other Azure resources. This allows to authorize environment rather than application and by such avoid passing any kind of secrets, access tokens, …, etc.

There are two types of managed identities and ACI supports both or them:

  • user-assigned – is created as a standalone Azure resource (created inside AAD tenant and bound to a resource group), independent of any particular Azure resource it is intended to be used with. Therefore user-assigned identity life-cycle is managed separately from the life-cycle of Azure resources to which it’s assigned. This type of managed identity can be assigned to multiple Azure resources (or in our case multiple ACI) which results in a one time need to configure RBAC access policies.
  • system-assigned – is enabled directly on Azure resource. When it’s enabled, Azure automatically creates an identity in the AAD tenant. Therefore system-assigned identity life-cycle is directly tied to the resource it’s enabled on, which makes it impossible to be reused and therefore results in a need to configuration RBAC access policies on required resources for each system-assigned identity.

Both user-assigned and system-assigned identities are assigned / enabled at ACI deployment time.

Here is an example of how to configure user-assigned / system-assigned identity using az container create command:

az container create `
  --resource-group <resource-group> `
  --name <name> `
  --image <image> `
  --assign-identity <identity resource id or blank to use system identity>

System-assigned identity

There is one important moment to understand when using system-assigned identity. Each time you create a new ACI with system-assigned identity enabled you create a new identity in the AAD. However if you re-deploy ACI, Azure will re-use previously created system-assigned identity:

# Initial deployment

az container create `
  --resource-group <resource-group> `
  --name <name> `
  --image <image> `
  --assign-identity

# Re-deployment with new image (ex. update)

az container create `
  --resource-group <resource-group> `
  --name <name> `
  --image <new-image> `
  --assign-identity

When ACI container group is assigned with either user-assigned or system-assigned identity the application code executing inside a container could request resource access token from Azure Instance Metadata Service (hereinafter AIMS).

Here is an example of retrieving Azure Key Vault (https://vault.azure.netaccess token from AIMS and using it to retrieve a secret:

# The URL is split into multiple lines for readability purposes

#           AIMS endpoint
#                ↓ 
curl http://169.254.169.254/metadata/identity/oauth2/token`
     ?api-version=2018-02-01` # ← Should be set to 2018-02-01
     &resource=https%3A%2F%2Fvault.azure.net ` # ← Resource URI
     -H Metadata:true -s
#          ↑
#       Set HTTP 'Metadata' header as required by endpoint contract

# Result (JSON)
#
#{
#  "access_token": "<access token>",
#  "refresh_token": "",
#  "expires_in": "<value>",
#  "expires_on": "<value>",
#  "not_before": "<value>",
#  "resource": "https://vault.azure.net/", # ← Resource URI
#  "token_type": "Bearer"
#}

# Use token to retrieve a secret
# The URL is split into multiple lines for readability purposes

#              Resource URI         Secret name
#                  ↓                    ↓
curl https://vault.azure.net/secrets/secret/`
     ?api-version=2016-10-01` # ← Should be set to 2016-10-01
     -H "Authorization: Bearer $access_token"
#             ↑
#        Pass "access token" in authorization header

# Result (JSON)
#
#{
#  "value": "secret value"
#  ...
#}

In .NET application code you can leverage managed-identity benefits to access Azure resources using NuGet packages from Microsoft or by manually retrieving resource access tokens using the .NET implementation of the above example.

Here is an example of how you can read a secret from Azure Key Vault using Microsoft.Azure.KeyVault and Microsoft.Azure.Services.AppAuthentication packages:

// This code uses the following NuGet packages from Microsoft:
// - Microsoft.Azure.KeyVault
// - Microsoft.Azure.Services.AppAuthentication

public Task<string> GetSecretAsync() 
{
  var azureServiceTokenProvider = new AzureServiceTokenProvider();

  var keyVaultClient =
    new KeyVaultClient(
      new KeyVaultClient.AuthenticationCallback(
        azureServiceTokenProvider.KeyVaultTokenCallback));

  var secret = await keyVaultClient
    .GetSecretAsync("https://vault.azure.net", "secret")
    .ConfigureAwait(false);

  return secret.Value;
}

… and here is an example of how to retrieve resource access token from AIMS endpoint:

var httpClient = new HttpClient
{
  // Set Metadata: true HTTP header
  DefaultRequestHeaders = { { "Metadata", "true" } }
};

// AIMS_ENDPOINT = 169.254.169.254
// AIMS_VERSION  = 2018-02-01
// resource      = https://vault.azure.net

var uri = $"http://{AIMS_ENDPOINT}/metadata/identity/oauth2/token"
        + $"?api-version={AIMS_VERSION}"
        + $"&resource={HttpUtility.UrlEncode(resource)}";

var response = await this.httpClient.GetAsync(uri);
if (response.StatusCode == HttpStatusCode.NotFound || 
    response.StatusCode == (HttpStatusCode) 429 /* Too many requests */)
{
  throw new Exception(
    $"The AIMS returned '{response.StatusCode}' status code.");
}

var content = await response.Content.ReadAsStringAsync();

JObject resultObject;
try
{
  resultObject = JObject.Parse(content);
}
catch (Exception e)
{
  throw new Exception(
    "Invalid JSON response received from AIMS",
    ExceptionDispatchInfo.Capture(e).SourceException);
}

if (response.StatusCode != HttpStatusCode.OK)
{
  // ERROR_IDENTIFIER = "error"
  // ERROR_DESCRIPTION = "error_description"

  var errId = resultObject[ERROR_IDENTIFIER].Value<string>();
  var errDescr = resultObject[ERROR_DESCRIPTION].Value<string>();
  throw new Exception(
    $"The AIMS returned '{response.StatusCode}' status code "
  + $"with the following error: '{errId}'/'{errDescr}'");
}

return resultObject[ACCESS_TOKEN].Value<string>();

This token now can be used to access Azure Key Vault directly through the API.

Contention points


Access policy level of control


Using managed identity means we could control access to the resource only on the level resource allows it. In case of Azure Key Vault we could restrict identity to be able to get, set or list secrets, but even if we restrict access to “get secrets” we still won’t be able to prevent it from getting any secrets it knows by name (this is an opposite side of specifying secrets at deployment time).

While in most cases such behavior is considered an advantage it is still worth thinking of.

Access from Azure Virtual Network


Currently ACI doesn’t support usage of managed-identities from within Azure Virtual Network (see here for more details).

Application code is tied to how secrets are stored


Please see Appendix A. Common contention points section for more details.

Appendix A. Common contention points


This appendix contains description contention points shared by multiple approaches described in this post.

Secrets are specified as part of deployment


This contention point is common for passing secrets using secret volume and passing secrets using secure environment variables approaches. They both require you to specify well-defined set of secrets and their values as part of deployment script.

Let’s analyze what advantages and disadvantages this leads to:

Advantages

  • Allows to pass only those secrets that are required by application.

    This is very important if you want to have granular control on what secrets your application code can access.
  • Allows to abstract both application code and deployment script from secret values source.

    From the application code side you can choose a way to consume secrets configuration and stick to it regardless of where these secrets come from. The very same thing applies to deployment script which bounds itself to input arguments.

Disadvantages

  • In case of automated deployment, additional cautions should be taken to ensure no secrets information is included into build or deployment scripts, or logged as part of deployment process.
  • Secrets passed to application at deployment time could become outdated which would required application re-deployment.

The first disadvantage could be mitigated by using a build system which support secure variables. Secure variables aren’t logged or displayed by a build system actions, so they can be safely used as parameters for build and deployment scripts.

In Azure DevOps you can create secure variables manually as part of build pipeline configuration, however, besides creating them manually we also can retrieve secrets from Azure Key Vault and store them as secure variables during the build.

Let’s consider a small build pipeline where:

  • Azure Key Vault Pipeline Task is used to read secrets from Azure Key Vault into Azure DevOps secure variables (ex. a connection string into ConnectionString secure variable)[1]
  • Azure CLI Task is used to invoke inline deployment script.

Here is an inline deployment script (executed by Azure CLI Task) which uses secure environment variables retrieved from Azure Key Vault to pass connection string information (through $(ConnectionString) variable):

az container create `
  --resource-group <resource group>`
  --name <name> `
  --image <image> `
  --secure-environment-variables 'ConnectionString=$(ConnectionString)'

Here is a sample from build pipeline logs:

<time> [command]/usr/bin/az cloud set -n AzureCloud
<time> [command]/usr/bin/az login --service-principal ***
<time> [
<time>   {
<time>     "cloudName": "AzureCloud",
<time>     "homeTenantId": "***",
<time>     "id": "<id>",
<time>     "isDefault": true,
<time>     "managedByTenants": [],
<time>     "name": "<name>",
<time>     "state": "Enabled",
<time>     "tenantId": "***",
<time>     "user": {
<time>       "name": "***",
<time>       "type": "servicePrincipal"
<time>     }
<time>   }
<time> ]
<time> [command]/usr/bin/az account set --subscription <subscription>
<time> [command]/bin/bash /***/azureclitaskscript***.sh
                               ^
                  This is all output we have from Azure CLI Task

<time> [command]/usr/bin/az account clear

You can see that there isn’t any notion of $(ConnectionString) value. You may wonder that this is due to Azure CLI Task implementation, however, even if we use our own build script with something similar to echo '$(ConnectionString)' inside it, $(ConnectionString) value still won’t be disclosed:

<time> ##[section]Starting: Bash Script
<time> =======================================================
<time> ...
<time> =======================================================
<time> Generating script.
<time> Script contents:
<time> echo '***'
       ^
       This is how $(ConnectionString) is printed
<time> ============== Starting Command Output ================
<time> ...
<time> ##[section]Finishing: Bash Script

This behavior is automatically enforced by Azure DevOps for all secure variables.


The second disadvantage could be partially mitigated by storing application secrets in a dedicated centralized secret store and passing it’s access token as secret to the application (using either secret volume of secure environment variables). Application then will use this access token to retrieve secrets directly from the secret store. This solution adheres a trade off between having immutable, well-defined set of secrets abstracted from secret store (advantage #1) and application ability to read latest version of secrets directly from secret store.

This is important to note that this approach assumes application secret store contain only application secrets for particular environment. Ideally this secret store should also be accessible only from application’s virtual network. 

Author’s note

Application code is tied to how secrets are stored


Regardless of how secrets are stored there should be some application code to read them. This issues can’t be mitigated completely, however it’s impact can be significantly reduced if application framework or third-party libraries provides support for selected approach.

Let’s consider a very simple example of ASP.NET Core 3.0 application which is configured to read secrets from a secret volume in Staging and from Azure Key Vault in Production:

// This code uses the following NuGet packages:
// - Microsoft.Extensions.Configuration.KeyPerFile
// - Microsoft.Extensions.Configuration.AzureKeyVault

public class Program
{
  public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
      .ConfigureWebHostDefaults(webBuilder =>
      {
        webBuilder.ConfigureAppConfiguration(
          (ctx, config) =>
          {
            if (ctx.HostingEnvironment.IsStaging())
              /*
               *  The call to .AddKeyPerFile configures 
               *  application to use configuration provider capable of
               *  reading values from secret volume.
               */
              config.AddKeyPerFile("/mnt/secrets", false);

            if (ctx.HostingEnvironment.IsProduction())
              /*
               *  The call to .AddAzureKeyVault() configures 
               *  application to use configuration provider capable of 
               *  reading secrets from a Azure KeyVault.
               */
              config.AddAzureKeyVault("https://project.vault.azure.net/");
          });

        webBuilder.UseStartup<Startup>();
      });
}

Requirement to consume secrets from different sources in different environments causes application to have environment specific code i.e. in Staging we expect secrets to come from secret volume mounted on certain path and in Production we expect secrets to come from Azure Key Vault.

This exact situations was the reason I’ve started of CoherentSolutions.Extensions.Configuration.AnyWhere project some months ago. This project provides application an ability to dynamically inject configuration providers based on the information passed through environment variables.

Here is how the above code can be modified when using  CoherentSolutions.Extensions.Configuration.AnyWhere. All required configuration sources are are included as configuration adapters NuGet packages and enabled using environment variables:

// This code uses the following NuGet packages from Coherent Solutions:
// - CoherentSolutions.Extensions.Configuration.AnyWhere
// - CoherentSolutions.Extensions.Configuration.AnyWhere.AdapterList
// - CoherentSolutions.Extensions.Configuration.AnyWhere.KeyPerFile
// - CoherentSolutions.Extensions.Configuration.AnyWhere.AzureKeyVault
//
// Environment variables (Staging):
// - ANYWHERE_ADAPTER_0_NAME=KeyPerFile
// - ANYWHERE_ADAPTER_0_DIRECTORY_PATH=/mnt/secrets
// - ANYWHERE_ADAPTER_0_OPTIONAL=false
//
// Environment variables (Production):
// - ANYWHERE_ADAPTER_1_NAME=AzureKeyVault
// - ANYWHERE_ADAPTER_1_VAULT=https://project.vault.azure.net/
// - ANYWHERE_ADAPTER_1_SECRETS=secret-one

public class Program
{
  public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
      .ConfigureWebHostDefaults(webBuilder =>
      {
        webBuilder.ConfigureAppConfiguration(
          (ctx, config) =>
          {
            config
              /*
               * This call configures a well known list
               * of configuration adapters - i.e. this allows to use 
               * ADAPTER_*_NAME variables with 'AzureKeyVault' or 
               * 'KeyPerFile' values instead of using 
               * fully qualified adapter type
               */
              .AddAnyWhereConfigurationAdapterList()
              /*
               * This call configures AnyWhere configuration
               */
              .AddAnyWhereConfiguration();
            }
          });

        webBuilder.UseStartup<Startup>();
      });
}

This way you can use as many configuration providers (included as adapters) as required without modifying application code.

Conclusion


Uhhh… This was a long post.

Thank you for reading and I hope you have enjoyed it! See you next time!

Footnotes


  1. To better understand what temporary RAM storage is please refer to tmpfs related articles and documentation. [↪]
  1. If you have to pass a complex secret [↪] or secure environment variable [↪] value i.e. a database connection string, you can escape it by surrounding the whole secret into single quotes: ‘key=value’.
  1. Besides specifying secret volume secrets and mount path directly through az container create command parameters you can configure them as part of the yaml or resource manager deployment templates. []
  1. Besides specifying secure environment variables directly through az container create command parameters you can configure them as part of the yaml deployment template. []
  1. “Azure Key Vault Pipeline Task” isn’t the only way to integrate Azure DevOps with Azure Key Vault, check out this post for details about how to link Azure Key Vault through Variable Groups[↪]

Acknowledgement


CoherentSolutions.Extensions.Configuration.AnyWhere is an open source project owned and maintained by Coherent Solutions Inc.

2 thoughts on “Using secrets in Azure Container Instances

  1. Hi Oleg,

    Great post, really helpful and very good explained. I have ACI deployed to VNET and I saw in documentation it is not possible to use managed identities to access Key Vault once ACI is in vnet? Do you maybe have an idea how can these managed identities be bypassed (if at all), cause I still want to use Key Vault…it seems as the best and logical solution for passing secrets.

    Like

    1. Hi Tanja,

      Yes, this is true. You can’t use managed identity to access Key Vault from ACI deployed to VNET. However, there is a work around which was partially described in the post.

      In general, you should have two Key Vaults: one is application specific Key Vault (in general there should be a separate application specific Key Vault for each environment) and one Key Vault used to store client id / client secret to other Key Vaults.

      During the build (in this post I used Azure DevOps because is has great integration with Key Vault) you retrieve a client id / secret pair to application specific Key Vault and pass it to ACI using either secure environment variables or secret volume. Then in your application code you can use these credentials to access application specific Key Vault.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s