Terraform on Azure Key Vault audit logs through diagnostic settings, and a Rust census

A vault with no diagnostic setting works fine and records nothing.

An Azure Key Vault records nothing about who reads its secrets until you attach a diagnostic setting to it. This lesson declares that setting in Terraform, then uses a small Rust program to find every vault in a subscription that still has no audit logging.

§I. Frame: the problem in plain words

Picture a key vault that holds database passwords and signing keys. Applications read from it all day. Six months later someone asks: which identity read the payments secret last Tuesday, and from which IP address? If nobody turned on logging, there is no answer. The vault worked the whole time, and it kept no record you can query.

That is the default. Azure Monitor sorts its data into three kinds, and the kind that holds a vault's own access records, called resource logs, "aren't collected by default. Create a diagnostic setting to collect resource logs." (Azure Monitor docs) The vault does not warn you. Creating it succeeds, it serves requests, and the silence looks the same as a healthy system.

This matters beyond Azure. The DevOps Handbook quotes Marcus Sachs on why breaches go undetected: "One of the primary reasons for this is that no one in the organization was regularly reviewing the log files." (PDF p. 433) The book also names two causes: "blind spots in our monitoring" and nobody examining "the relevant telemetry in their daily work." A vault with no diagnostic setting is a blind spot. A vault with logs that nobody can find or query is the second problem. The book's remedy is to make the data "adequately centralized to facilitate easy and meaningful analysis and response."

This lesson has two halves.

1. Declare the setting in Terraform, so each vault you manage sends audit events to a Log Analytics workspace.
2. Count the vaults that have no such setting, with a Rust program, because Terraform only knows about the vaults it manages.

A third piece packages the script that gathers the census input using Nix, so the script is checked at build time.

What was verified, stated plainly. The Terraform file was run through terraform init -backend=false, terraform validate and terraform fmt -check (Terraform 1.14.3, azurerm v4.81.0). Validation reported valid, with 0 errors and 0 warnings. It was never planned or applied, because no Azure credentials were used. The Rust census compiled with no warnings, and it ran only against a synthetic fixture shaped like the REST API response. It has not run against a real subscription. The Nix check passed on one system, aarch64-darwin. The script that builds the census input was built and linted but never run against live Azure.

§II. Foundations: what Azure Monitor collects, and what a diagnostic setting is

Three kinds of data, one that needs a setting

Only the third row needs you to act, and for Key Vault it is the row that matters: the AuditEvent records described below, one per operation against the vault and its keys, secrets and certificates, are resource logs.

What a diagnostic setting is

A diagnostic setting is a small configuration object that lives on one resource. It says "send these categories of data from this resource to these destinations." The rules, all from the Azure Monitor documentation:

• You "create a separate diagnostic setting for each resource that you want to collect data from."
• A single setting "can define no more than one of each destination type." To send to two Log Analytics workspaces, you create two settings.
• Each resource can have up to five diagnostic settings.
• The destination "must already exist before you create the diagnostic setting."

The destination types are a Log Analytics workspace (query with log queries and workbooks, alert on the data), an Azure Storage account (audit and archive, with immutable storage available "to prevent its modification"), Azure Event Hubs (stream to external systems such as a SIEM) and Azure Monitor partner solutions. (Azure Monitor docs)

For a workspace, only the workspace itself must exist. The tables inside it "are created automatically when the first data is sent." (Azure Monitor docs) For Storage accounts and Event Hubs on regional resources, the destination must be in the same region as the monitored resource. (Azure Monitor docs)

Categories versus category groups

A category is one named kind of log, such as AuditEvent. A category group is a predefined bundle of categories. If the group changes, "your log collection is modified automatically." (Azure Monitor docs) Two groups exist today:

• allLogs: all categories for the resource.
• audit: all resource logs that record customer interactions with data or the settings of the service. You don't need audit if you select allLogs.

Not every service supports groups, and if you use a group you can't also pick individual categories. (Azure Monitor docs)

What a Key Vault audit record answers

The Key Vault logging page lists the fields of each record. (Key Vault logging docs) The ones that answer incident questions:

So "who read the payments secret last Tuesday" becomes a query for SecretGet records on one vault, read from identity and callerIpAddress. The page says records are available "10 minutes (at most) after the key vault operation." (Key Vault logging docs) Separately, after you create a setting, data "should start flowing to your selected destinations within 90 minutes." (Azure Monitor docs) Expect a wait the first time.

§III. Mechanism: the Terraform resource

The azurerm provider's azurerm_monitor_diagnostic_setting "manages a Diagnostic Setting for an existing Resource." (azurerm docs) Here is kv-diag.tf, block by block.

The destination and the vault

# Destination: a Log Analytics workspace in the same region as the vault.
resource "azurerm_log_analytics_workspace" "audit" {
  name                = "law-audit-eastus"
  location            = azurerm_resource_group.sec.location
  resource_group_name = azurerm_resource_group.sec.name
  sku                 = "PerGB2018"
  retention_in_days   = 90
}

resource "azurerm_key_vault" "app" {
  name                       = "kv-app-eastus-111122"
  ...
  purge_protection_enabled   = true
}

The workspace comes first because the destination must exist before the setting does. Terraform handles the order for you, since the setting will reference the workspace's id.

The comment says "same region as the vault." The region rule in the Azure Monitor docs is stated for Storage accounts and Event Hubs. The file keeps everything in eastus anyway, so the comment is a conservative habit and the rule is not in doubt here.

Asking Azure which categories exist

# Ask Azure which log categories and groups this resource type offers.
data "azurerm_monitor_diagnostic_categories" "kv" {
  resource_id = azurerm_key_vault.app.id
}

Not every resource has category groups, so the provider offers this data source "to identify which categories are available for a given Resource." (azurerm docs) The file exposes the result as an output:

output "kv_log_category_groups" {
  value = data.azurerm_monitor_diagnostic_categories.kv.log_category_groups
}

After an apply, that output shows what Azure itself reports for the vault. No apply was run for this lesson, so there is no output to show here. This is the way to check that audit exists before you rely on it. The CLI has the same view: az monitor diagnostic-settings categories list lists the categories for a resource. (Azure CLI reference)

The setting itself

# Resource logs are not collected until a diagnostic setting exists.
resource "azurerm_monitor_diagnostic_setting" "kv_audit" {
  name                       = "kv-audit-to-law"
  target_resource_id         = azurerm_key_vault.app.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.audit.id

  enabled_log {
    category_group = "audit"
  }

  enabled_metric {
    category = "AllMetrics"
  }
}

The provider rules that shape this block (azurerm docs):

• At least one destination. One of eventhub_authorization_rule_id, log_analytics_workspace_id, partner_solution_id or storage_account_id must be set. Here it is the workspace.
• **At least one enabled_log or enabled_metric block.** The file has both.
• **Exactly one of category or category_group per enabled_log.** That mirrors the Azure rule that groups and individual categories can't be mixed.
• **name and target_resource_id force replacement.** Changing either "forces a new resource to be created." A rename means destroy and create.
• **The service trap.** If the name is service, "it will not be possible to fully delete the diagnostic setting," because of legacy API support. (azurerm docs) Pick a descriptive name instead.
• **log_analytics_destination_type.** For some resource types, Key Vault among them, "this field is unconfigurable." (azurerm docs) The file leaves it out.

Audit, allLogs, or AuditEvent?

Three spellings could express "send the audit logs":

This file uses audit. The Key Vault docs say AuditEvent is the only category in Key Vault logs, but no source here says the groups and that category contain exactly the same records, so this lesson does not claim it. Use the kv_log_category_groups output and the categories list to confirm on your own vault. The census in §IV accepts all three spellings, so you can choose without breaking the check.

On cost, the Azure Monitor docs advise: "Collect only the log categories that you need for each service." That points away from allLogs unless you need it. The same page says to skip platform metrics unless you need them in the workspace for complex analysis by log queries, because Azure Monitor collects platform metrics automatically and shows them in metrics explorer without a setting. (Azure Monitor docs) The AllMetrics block mirrors the provider doc's example, (azurerm docs) but by that guidance you could delete it, since the enabled_log block already satisfies the "at least one" rule.

Importing and deleting

If a vault already has a hand-made setting, you can bring it under Terraform. The import ID is {resourceId}|{diagnosticSettingName}, for example .../providers/Microsoft.KeyVault/vaults/vault1|logMonitoring1. The doc calls it "a Terraform specific Resource ID." (azurerm docs)

A warning from the Azure Monitor docs about the other direction: delete a resource's diagnostic settings if you delete or rename that resource, or migrate it across resource groups or subscriptions. If you don't, and the resource is re-created, "any diagnostic settings for the deleted resource could be applied to the new one." Settings can outlive their target, so clean them up deliberately.

§IV. The census: finding vaults Terraform does not know about

The file above guarantees one thing: the vault it declares has a setting. It says nothing about the other vaults in the subscription, such as one made in the portal, one from an older project, or one owned by another team. Terraform only knows what is in its state and configuration. To find the rest, ask Azure.

kv-diag-census.rs is a single-file Rust program. Single-file packages are an unstable Cargo feature: the manifest sits in a frontmatter block at the top of the .rs file, and you run it with cargo +nightly -Zscript file.rs. (Cargo book) The default target directory for these packages is $CARGO_HOME/target/<hash>. (Cargo book)

The input shape

The program reads one JSON file. Each vault carries the body returned by the Diagnostic Settings List call, GET {resourceUri}/providers/Microsoft.Insights/diagnosticSettings?api-version=2021-05-01-preview. (REST API reference) That body is a collection: {"value": [ {name, id, type, properties: {...}} ]}, where properties holds workspaceId, storageAccountId, eventHubAuthorizationRuleId, marketplacePartnerId, logs and metrics. (REST API reference)

#[derive(Deserialize, Default)]
#[serde(rename_all = "camelCase", default)]
struct LogSetting {
    category: Option<String>,
    category_group: Option<String>,
    enabled: bool,
}

The camelCase rename lets Rust's category_group match the API's categoryGroup. The default attribute means a missing field becomes None or false instead of a parse error. Settings often omit fields, so this is the forgiving choice.

Does this log entry cover audit events?

/// A log entry covers Key Vault audit events if it names AuditEvent or a group that contains it.
fn covers_audit(l: &LogSetting) -> bool {
    l.enabled
        && (l.category.as_deref() == Some("AuditEvent")
            || matches!(l.category_group.as_deref(), Some("audit") | Some("allLogs")))
}

It checks enabled first, because a listed category can be switched off. The allLogs group counts because the Azure Monitor docs say audit need not be selected when allLogs is.

Is there a real destination?

/// An empty string is not a destination: the API returns "" for unset ids.
fn has_destination(p: &Props) -> bool {
    [&p.workspace_id, &p.storage_account_id, &p.event_hub_authorization_rule_id, &p.marketplace_partner_id]
        .iter()
        .any(|d| d.as_deref().is_some_and(|s| !s.is_empty()))
}

Here lies the subtle bug the program avoids. A check like "is workspaceId present?" passes on an empty string. The API's own sample response contains "workspaceId": "" next to a populated storage account and event hub. (REST API reference) The field is present and means nothing. A setting that sends logs nowhere is exactly the failure you're hunting, so the check demands a non-empty string.

The three failure reasons

fn verdict(v: &Vault) -> Result<String, String> {
    let settings = &v.diagnostic_settings.value;
    if settings.is_empty() {
        return Err("no diagnostic setting (resource logs are not collected)".into());
    }
    for s in settings {
        if s.properties.logs.iter().any(covers_audit) && has_destination(&s.properties) {
            return Ok(s.name.clone());
        }
    }
    let audit_somewhere = settings.iter().any(|s| s.properties.logs.iter().any(covers_audit));
    if audit_somewhere {
        Err("audit log enabled but the setting has no destination".into())
    } else {
        Err(format!("{} setting(s), none with an enabled audit log", settings.len()))
    }
}

A vault passes only if one single setting has both an enabled audit entry and a non-empty destination. Audit enabled in one setting and a destination in a different one does not count, because the data would never flow. When no setting passes, the program picks the most useful of three explanations: no setting at all, audit enabled but no destination, or settings that exist but enable no audit log.

main prints one line per vault, counts the failures, and ends with ExitCode::from(2) if any vault failed and success otherwise. A pipeline step treats a non-zero exit as failure, so the census can block a release.

The real output

The fixture (fixture/kv-diag.json) is synthetic. It was written to match the shape of the REST API response. It describes four vaults, one per outcome. This is the real output of the program over it, with exit code 2:

subscription 00000000-0000-0000-0000-000000000000: 4 key vault(s)
ok         kv-app-eastus-111122  via kv-audit-to-law
ATTENTION  kv-payments-eastus  no diagnostic setting (resource logs are not collected)
ATTENTION  kv-legacy-westus  1 setting(s), none with an enabled audit log
ATTENTION  kv-batch-eastus  audit log enabled but the setting has no destination
4 vault(s), 3 without audit logs

Line by line, against the fixture:

• kv-app-eastus-111122 has setting kv-audit-to-law with categoryGroup: audit, enabled, and a full workspace ID. It matches what the Terraform file builds.
• kv-payments-eastus has "value": []. No setting, so no resource logs. This is the opening scenario.
• kv-legacy-westus has a setting named metrics-only. Its log entry is AuditEvent with enabled: false, and only AllMetrics is on. A setting exists, so someone thought about monitoring, but no audit events flow.
• kv-batch-eastus has allLogs enabled and "workspaceId": "". This is the empty-string case from the REST API sample. Without the non-empty check this vault would pass.

§V. Packaging the input builder with Nix

In production the census input comes from a live subscription. flake/flake.nix defines a script, kv-diag-input, that builds the JSON the Rust program reads. It uses az account show for the subscription ID, az keyvault list for the vaults, and az rest against the Diagnostic Settings endpoint for each one. The CLI references describe az keyvault list as "List Vaults and/or HSMs" and az rest as "Invoke a custom request." (Azure CLI reference) The query in the script keeps only entries whose type is Microsoft.KeyVault/vaults.

kv-diag-input = pkgs.writeShellApplication {
  name = "kv-diag-input";
  runtimeInputs = [ pkgs.azure-cli pkgs.jq ];
  text = ''
    sub="$(az account show --query id --output tsv)"
    ...
    jq -n --arg s "$sub" --argjson v "$vaults" '{subscription_id: $s, vaults: $v}'
  '';
};

Why writeShellApplication, and not a loose shell file?

• It "writes an executable shell script to /nix/store/<store path>/bin/<name> and checks its syntax with shellcheck and the bash's -n option."
• It sets errexit, nounset and pipefail by default. (Nixpkgs manual) The script stops on a failed command, an unset variable, or a failure inside a pipe.
• runtimeInputs adds azure-cli and jq "to the shell script's $PATH at runtime." (Nixpkgs manual) The script cannot run with a missing or different jq.

The rest is wiring. The flake defines the package for three systems (aarch64-darwin, x86_64-linux, aarch64-linux), then lists the same derivation under checks:

checks = forAll (pkgs: {
  kv-diag-input = self.packages.${pkgs.stdenv.hostPlatform.system}.kv-diag-input;
});

nix flake check "verifies that the flake ... can be evaluated successfully ... and that the derivations specified by the flake's checks output can be built successfully." (Nix manual) Because the check is the script, building it runs shellcheck and bash -n. Without --all-systems it checks the current system only. (Nix manual)

The passing run

building '/nix/store/4zsg9y0qw5ywg736fga3xxljba4rc60m-kv-diag-input.drv'...
kv-diag-input> structuredAttrs is enabled
✅ checks.aarch64-darwin.kv-diag-input
warning: The check omitted these incompatible systems: aarch64-linux, x86_64-linux
Use '--all-systems' to check all.

The last two lines are why the claim stays narrow. Only aarch64-darwin was checked. The Linux entries were skipped and were not run.

The deliberate break

To see the check fail, the quotes around "$sub" were removed in a scratch copy, so the last line read --arg s $sub. The build failed:

kv-diag-input> In /nix/store/s69gdhzjhxggqcpp4zkwymz5yxs55lcv-kv-diag-input/bin/kv-diag-input line 18:
kv-diag-input> jq -n --arg s $sub --argjson v "$vaults" '{subscription_id: $s, vaults: $v}'
kv-diag-input>               ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
...
❌ checks.aarch64-darwin.kv-diag-input
error: Cannot build '/nix/store/mng5267dldisi3xk2ac3vx9im2yhcdsc-kv-diag-input.drv'.
       Reason: builder failed with exit code 1.

Shellcheck labels SC2086 as "info", yet the build still failed, and nix flake check returned exit 1. An unquoted variable can split on spaces or expand as a glob, and the check refuses to produce a script with that flaw. The broken script never reaches the Nix store as a usable result.

This is the same idea as the book's advice on environment tests. The DevOps Handbook wants teams to "enforce the consistency and correctness" of environments, audit logging among them, and says "we can use the same testing frameworks that we use to test our code to also test that our environments are configured and operating correctly." (PDF p. 201) The census tests the environment, and the flake tests the tool that feeds it. Both belong in the pipeline that runs your code checks.

§VI. Running it in a pipeline

One ordering that fits the pieces, with what each step catches:

1. **nix flake check.** Builds kv-diag-input, so shellcheck and bash -n run. A broken script stops the pipeline here.
2. **Run kv-diag-input** against the subscription and write its JSON to a file. This is the step that needs an Azure session. It is also the step this lesson has not run end to end.
3. Run the census on that file. Exit 2 means at least one vault has no working audit logging, and the step fails.
4. **terraform plan saved to a file.** If the census flagged a vault you manage, the fix is in the configuration. If the vault was made outside Terraform, you can write the resource and import it using the {resourceId}|{diagnosticSettingName} ID from §III.
5. Review the saved plan.
6. **terraform apply** of that exact plan.

Steps 4 to 6 are the same flow covered in the paired lesson on saved plan files (terraform plan -out), which is not repeated here.

One honest limit. The census reads configuration. It shows a setting with an audit category and a destination exists. It does not show that records have arrived, and the Azure Monitor docs say first data can take up to 90 minutes. After you change a setting, give it time before you judge by a query.

§VII. Drills

1. A vault was created last month and has never had a diagnostic setting. An auditor asks who read a secret in the first week. Can you answer, and why? (No. Resource logs "aren't collected by default," so nothing was recorded until a diagnostic setting was created, per the Azure Monitor docs.)
2. You set category = "AuditEvent" and category_group = "audit" in the same enabled_log block. What does the provider do, and what is the Azure-side reason? (It rejects it: exactly one of category or category_group must be specified, per the azurerm docs. On the Azure side, the Azure Monitor docs say that if you use a category group you can't select individual categories.)
3. The census reports audit log enabled but the setting has no destination for a vault whose setting shows "workspaceId": "". Why does the code treat "" as missing instead of present? (The API returns empty strings for unset IDs, and its own sample shows "workspaceId": "" beside populated destinations in the REST API reference. An empty string sends the logs nowhere, so has_destination requires a non-empty value.)
4. You rename a vault in Terraform and let it re-create. What risk does the diagnostic-settings guidance warn about? (If the old settings aren't removed and a resource is re-created, "any diagnostic settings for the deleted resource could be applied to the new one," per the Azure Monitor docs. Delete the settings when you delete or rename the resource.)
5. In the Nix flake, why does removing the quotes around "$sub" fail nix flake check, even though shellcheck calls SC2086 an "info"? (The check is the kv-diag-input derivation, and writeShellApplication runs shellcheck at build time, per the Nixpkgs manual. The build failed with exit code 1 in the logged run, so the check failed.)

§VIII. Closing

A vault with no diagnostic setting works fine and records nothing, so you have to look for the gap on purpose. Declare the setting in Terraform for the vaults you own. Then run a census over the whole subscription for the ones you don't. Check the script that feeds the census in the same pipeline as everything else, so the monitoring you rely on is itself checked.

Related

• Microsoft Learn, "Diagnostic settings in Azure Monitor"
• Microsoft Learn, "Azure Key Vault logging"
• Terraform azurerm provider docs, azurerm_monitor_diagnostic_setting
• Azure REST API, "Diagnostic Settings - List" (api-version 2021-05-01-preview)
• Azure CLI reference: az keyvault list, az rest, az monitor diagnostic-settings
• Nixpkgs manual, writeShellApplication
• Nix manual, nix flake check
• Cargo unstable book, single-file packages (-Zscript)
• Kim, Humble, Debois, Willis, The DevOps Handbook, Part VI, "Create Security Telemetry"
• Kim, Humble, Debois, Willis, The DevOps Handbook, environment tests in the deployment pipeline
• Paired lesson: Terraform Associate: saved plan files, from plan -out to apply, show -json, and the stale plan