Generating Kubernetes YAML with Nickel selectors written once, contracts with real messages, and a server-side dry run

Write the labels once, let contracts explain mistakes, and let the API server judge the rest.

The problem

A team runs a web service on Kubernetes, which schedules containers onto machines, and hand-writes two objects in YAML (YAML Ain't Markup Language) for it. A Deployment asks Kubernetes to keep a number of copies of a Pod running, where a Pod is a group of one or more containers, and its Pod template describes each copy. A Service gives those Pods one stable address. The Service never lists Pods. It finds them through labels, which are key-value pairs attached to objects, and a selector, which is the set of labels a Service or Deployment looks for. In the words of The Kubernetes Book, "This loose coupling is how Services know which Pods to send traffic to." So the same labels get typed three times: in the Deployment selector, in the Pod template and in the Service selector.

Repeated text is where mistakes enter. In the lab behind this lesson, a Service selector spelled note-api instead of notes-api was accepted by kube-apiserver, the Kubernetes API (application programming interface) server that receives and validates every request. It answered service/notes-api-typo created (server dry run) and exit 0. Step 6 later showed that this selector matches no Pods while the correct one matches. A misspelled field such as replcias and an image tagged latest, a moving tag, are other everyday mistakes. This lesson builds a generator that removes the repeated labels and a gate that catches the rest.

The idea, in plain words

Each service is described once, as a small record in Nickel. Nickel is a configuration language, and a record is a set of named fields, much like a JSON (JavaScript Object Notation) object. The record is checked by contracts, which are checks that run when the value is evaluated and point at the faulty value when they fail. A function renders that record into the Deployment and the Service, deriving every label and selector from the same value, so the copies cannot disagree. The result is exported as YAML. A real API server then checks that YAML with a server-side dry run, which validates the request and runs admission, the server's final checks before an object is stored, without saving anything. Labels that change on every release stay out of selectors, because a Deployment selector cannot be changed after creation.

Concept A: one description, rendered

Writing the labels once starts with a file that holds only facts about the service. The Nickel file imports the library, builds a record, and applies the k8s.App contract with | before handing the record to k8s.render. Checking first means a bad record fails at once, because otherwise the render function would build objects out of unchecked input.

# apps/notes-api.ncl: the only file a service team edits
let k8s = import "../lib/k8s.ncl" in
k8s.render (
  {
    name = "notes-api",
    team = "platform",
    image = "ghcr.io/example/notes-api:1.4.2",
    port = 8080,
    replicas = 3,
    env = { RUST_LOG = "info" },
  } | k8s.App
)

A service team edits only this file: name, team, image, port, replicas and environment. No label appears anywhere in it, so there is no label text to misspell. The team value later becomes the app.kubernetes.io/part-of label. Next comes the library function, shown from let render_app to the end of lib/k8s.ncl, with the contracts above it left for the next section.

let render_app = fun app =>
  # A Deployment selector cannot change after the Deployment is created.
  let selector_labels = {
    "app.kubernetes.io/name" = app.name,
    "app.kubernetes.io/part-of" = app.team,
  }
  in
  # Everything else can change on every release.
  let tag =
    if std.string.contains "@sha256:" app.image then
      null
    else
      std.array.last (std.string.split ":" app.image)
  in
  let common_labels =
    selector_labels
    & (if tag == null then {} else { "app.kubernetes.io/version" = tag })
  in
  {
    apiVersion = "v1",
    kind = "List",
    items = [
      {
        apiVersion = "apps/v1",
        kind = "Deployment",
        metadata = { name = app.name, labels = common_labels },
        spec = {
          replicas = app.replicas,
          selector.matchLabels = selector_labels,
          template = {
            metadata.labels = common_labels,
            spec.containers = [
              {
                name = app.name,
                image = app.image,
                ports = [{ name = "http", containerPort = app.port }],
                env =
                  app.env
                  |> std.record.to_array
                  |> std.array.map (fun entry => { name = entry.field, value = entry.value }),
              }
            ],
          },
        },
      },
      {
        apiVersion = "v1",
        kind = "Service",
        metadata = { name = app.name, labels = common_labels },
        spec = {
          selector = selector_labels,
          ports = [{ name = "http", port = 80, targetPort = "http" }],
        },
      },
    ],
  }
in
{ App = AppContract, render = render_app }

selector_labels holds the two labels that identify the service: its name and its team. The tag is the text after the last colon in the image, found by splitting on : and taking the last piece. For an image pinned by digest (@sha256:) the function sets it to null and leaves the version label out; the lab rendered only tagged images, so that branch was not exercised. common_labels joins selector_labels with a version label using &, the record merge operator, and the Nickel manual describes merge as symmetric. The two records share no field, so the result holds all three labels.

Three places then take their labels from these two values. The Deployment's selector.matchLabels is selector_labels, the Pod template and the object metadata use common_labels, and the Service's selector is selector_labels. The two selectors are one value, so they cannot differ, and the Pod template carries every selector label, so the Pods match both. The container port is named http and the Service uses that name as its targetPort, so the port number is written once, in the app file. Kubernetes wants environment variables as a list of name and value pairs, so the env record is turned into an array and mapped entry by entry.

By default Nickel exports JSON, and the extra parameter --format yaml exports YAML instead. Step 1 ran that on the app file; the container's env and ports lines are trimmed here.

apiVersion: v1
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    labels:
      app.kubernetes.io/name: notes-api
      app.kubernetes.io/part-of: platform
      app.kubernetes.io/version: 1.4.2
    name: notes-api
  spec:
    replicas: 3
    selector:
      matchLabels:
        app.kubernetes.io/name: notes-api
        app.kubernetes.io/part-of: platform
    template:
      metadata:
        labels:
          app.kubernetes.io/name: notes-api
          app.kubernetes.io/part-of: platform
          app.kubernetes.io/version: 1.4.2
      spec:
        containers:
        - env:
          ...
          image: ghcr.io/example/notes-api:1.4.2
          name: notes-api
- apiVersion: v1
  kind: Service
  metadata:
    name: notes-api
  spec:
    ports:
    - name: http
      port: 80
      targetPort: http
    selector:
      app.kubernetes.io/name: notes-api
      app.kubernetes.io/part-of: platform
kind: List

The Deployment selector lists name and part-of. The Pod template labels add the version. The Service selector repeats the Deployment selector line for line, with no version, because both came from selector_labels. A Service needs a Pod to carry every label it selects on, and the Pod may have more, so these Pods satisfy it. (The Service's metadata.labels are trimmed here.)

Symptom: nickel export fails with error: infinite recursion and points at labels = labels.

Cause: Nickel records are recursive by default, so a field named like the outer variable refers to itself. The manual says that name = name is "interpreted as a self-recursive definition" because the field being defined shadows the original variable.

Fix: Give the outer variable a different name, as the lab did with common_labels, or write include labels, which takes the value from the outer scope. Step 2b ran both.

$ nickel export pitfalls/self-reference.ncl
error: infinite recursion
  ┌─ ~/lab/nickel-k8s/pitfalls/self-reference.ncl:3:12
  │
3 │ { labels = labels }
  │            ^^^^^^ recursive reference

[exit 1]
$ nickel export pitfalls/include.ncl
{
  "labels": {
    "app": "notes-api"
  }
}
[exit 0]

The first file is let labels = { app = "notes-api" } in { labels = labels }. The right-hand labels resolves to the field being defined, not to the variable, so evaluation never finishes and Nickel reports the exact span. The second file, { include labels }, copies the outer variable into a field of the same name, and the export succeeds.

Concept B: contracts that explain themselves

A contract is a check attached to a value with |. Nickel evaluates it at run time, while exporting, and when it fails the error blames the faulty value and shows where it came from. Custom contracts let a library author write the message a service team will read. These are the contracts from lib/k8s.ncl.

let DnsLabel =
  std.contract.from_validator (fun value =>
    if !(std.is_string value) then
      'Error { message = "expected a string" }
    else if std.string.length value > 63 then
      'Error { message = "`%{value}` is longer than 63 characters" }
    else if std.string.is_match "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" value then
      'Ok
    else
      'Error {
        message = "`%{value}` is not a valid DNS label",
        notes = ["Use lowercase letters, digits and '-', and start and end with a letter or digit."],
      }
  )
in
let PinnedImage =
  std.contract.from_validator (fun value =>
    if std.string.is_match "@sha256:[0-9a-f]{64}$" value then
      'Ok
    else if std.string.is_match ":latest$" value then
      'Error { message = "image `%{value}` uses the moving tag `latest`" }
    else if std.string.is_match ":[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$" value then
      'Ok
    else
      'Error {
        message = "image `%{value}` has no tag",
        notes = ["Use name:version or name@sha256:digest."],
      }
  )
in
let Port =
  std.contract.from_validator (fun value =>
    if std.is_number value && std.number.is_integer value && value >= 1 && value <= 65535 then
      'Ok
    else
      'Error { message = "port must be an integer from 1 to 65535" }
  )
in
let AppContract = {
  name | DnsLabel,
  team | DnsLabel,
  image | PinnedImage,
  port | Port,
  replicas | std.number.PosNat | default = 2,
  env | { _ : String } | default = {},
}
in

std.contract.from_validator takes a function that returns 'Ok or 'Error with an optional message and optional notes, and each note appears on its own line under the error. DnsLabel checks a DNS (Domain Name System) label, the lowercase-letters-digits-and-hyphens form that Kubernetes names must follow. PinnedImage accepts a digest or a real tag and refuses latest and tagless images, because both can change under the same name. std.number.PosNat is built into Nickel and rejects zero and negatives, while | default = 2 fills in a value when the team omits replicas. The record contract is closed by default, so a field it does not list is an error. Writing .. after the last field would make it open.

Step 2 fed each broken file to nickel export. Three errors are shown here, with the part that points at the evaluated value trimmed.

error: contract broken by the value of `name`
       `Notes_API` is not a valid DNS label
   ┌─ ~/lab/nickel-k8s/lib/k8s.ncl:41:10
   │
41 │   name | DnsLabel,
   │          -------- expected type
   │
   ┌─ ~/lab/nickel-k8s/broken/nickel/bad-name.ncl:5:12
   │
 5 │     name = "Notes_API",
   │            ^^^^^^^^^^^ applied to this expression
   │
   = Use lowercase letters, digits and '-', and start and end with a letter or digit.

error: contract broken by the value of `image`
       image `ghcr.io/example/notes-api:latest` uses the moving tag `latest`

error: contract broken by a value
       extra field `replcias`
   ┌─ ~/lab/nickel-k8s/broken/nickel/typo.ncl:4:3
   │
   = Have you misspelled a field?
   = The record contract might also be too strict. By default, record contracts exclude any field which is not listed.
     Append `, ..` at the end of the record contract, as in `{some_field | SomeContract, ..}`, to make it accept extra fields.

Each message names the field, quotes the bad value and, for the name, says what a valid one looks like. The error points at both the contract line in the library and the offending line in the app file. The typo case is caught because the record contract is closed: replcias is not a field of AppContract, so it is reported as an extra field, and the hint asks the right question. Two more contracts fired in the same step: port = 80800 printed "port must be an integer from 1 to 65535", and replicas = 0 printed "expected a positive integer". All of them exited 1.

Symptom: A contract failure prints only a generic message that does not say what was wrong.

Cause: The contract was built with std.contract.from_predicate, which returns only true or false. The manual notes that it "doesn't allow for custom error messages."

Fix: Build the contract with std.contract.from_validator and return 'Error { message, notes }, as DnsLabel does.

Concept C: what only the API server can tell you

A dry run is a request that is processed but not saved. A server-side dry run sends the request to the API server with a flag that says so. The Kubernetes docs state the guarantee: "Kubernetes guarantees that dry-run requests will not be persisted in storage or have any other side effects." Every stage still runs, including admission, defaulting and schema validation, and the docs say the response is "the final object that would have been persisted, or an error". That lets Kubernetes itself judge a file, which matters because the contracts cover only the rules the library author wrote down.

The lab ran a throwaway server for this. The script cluster.sh started etcd, the key-value database where the API server keeps its objects, and kube-apiserver, both on 127.0.0.1. It used a static lab-only token and AlwaysAllow authorization, which is acceptable for a throwaway local server only. It started no scheduler, controller manager or kubelet, the parts of a full cluster that place Pods, react to stored objects and start containers. So nothing ran containers, and no EndpointSlices were created. An EndpointSlice is the list of Pod addresses behind a Service. The commands below use kubectl, the command-line client for the API server. Step 4 sent the generated YAML.

$ kubectl apply --dry-run=server -f out/notes-api.yaml
deployment.apps/notes-api created (server dry run)
service/notes-api created (server dry run)
[exit 0]

Both objects were accepted. Step 5 then sent hand-written files with mistakes in them.

$ kubectl apply --dry-run=server -f broken/yaml/deployment-field-typo.yaml
Error from server (BadRequest): error when creating "broken/yaml/deployment-field-typo.yaml": Deployment in version "v1" cannot be handled as a Deployment: strict decoding error: unknown field "spec.replcias"
[exit 1]
$ kubectl apply --dry-run=server -f broken/yaml/deployment-selector-mismatch.yaml
The Deployment "notes-api" is invalid: spec.template.metadata.labels: Invalid value: {"app.kubernetes.io/name":"notes"}: `selector` does not match template `labels`
[exit 1]
$ kubectl apply --dry-run=server -f broken/yaml/service-selector-typo.yaml
service/notes-api-typo created (server dry run)
[exit 0]
$ kubectl apply --dry-run=client -f broken/yaml/deployment-selector-mismatch.yaml
deployment.apps/notes-api created (dry run)
[exit 0]

The first failure is strict decoding: the kubectl apply reference says its default --validate is strict and, when the API server supports it, validates on the server. The API docs say a Strict request is rejected with a 400 Bad Request "when it detects any unknown or duplicate fields". The second failure enforces the documented rule that .spec.selector must match .spec.template.metadata.labels or it "will be rejected by the API." The Service with the one-letter typo passed, because a selector that matches nothing is still valid. The last command is the client-side dry run; the reference says the client strategy will "only print the object that would be sent, without sending it." It accepted the same broken Deployment the server had just rejected, so client mode did not run the server's checks.

Step 6 applied the good objects and the typo Service for real, created a Pod by hand with the right labels, and asked which Pods each Service's selector finds.

--- service notes-api selects: app.kubernetes.io/name=notes-api,app.kubernetes.io/part-of=platform
$ kubectl get pods -l app.kubernetes.io/name=notes-api,app.kubernetes.io/part-of=platform -o name
pod/notes-api-manual
[exit 0]
--- service notes-api-typo selects: app.kubernetes.io/name=note-api,app.kubernetes.io/part-of=platform
$ kubectl get pods -l app.kubernetes.io/name=note-api,app.kubernetes.io/part-of=platform -o name
[exit 0]

The correct selector finds pod/notes-api-manual and the typo selector finds nothing. The Kubernetes docs say the controller for a Service "continuously scans for Pods that match its selector, and then makes any necessary updates to the set of EndpointSlices for the Service." In a full cluster, the typo Service would therefore get no endpoints. The lab did not run that controller, so it did not observe the missing endpoints directly; it observed only that no Pod matches the selector.

Symptom: The Service exists, kubectl apply succeeded, but requests to it fail and it has no endpoints.

Cause: The selector does not match the Pod labels, and the API server does not check that (step 5).

Fix: Derive the selector from the same value as the Pod labels, as in concept A, and check with kubectl get pods -l <selector>.

Concept D: selector labels versus release labels

The Deployment's label selector is immutable, as The Kubernetes Book puts it: "you can't change it once it's deployed." The Kubernetes docs add that it "cannot be updated via kubectl patch, kubectl edit, kubectl apply, or tools like helm upgrade." A version label changes with every release, so putting it in a selector sets up the second release to fail. The lab kept an unsplit copy, lib/k8s-naive.ncl, that differs from lib/k8s.ncl in two lines, apart from a comment.

-          selector.matchLabels = common_labels,
+          selector.matchLabels = selector_labels,
...
-          selector = common_labels,
+          selector = selector_labels,

In the naive library, both selectors use common_labels, which includes app.kubernetes.io/version. Both selectors therefore carry a third label that changes with every image tag. Creating the Deployment with that selector works; the trouble starts when the tag moves. Step 7 rendered versions 1.4.2 and 1.4.3 with it. It applied 1.4.2, then dry-ran 1.4.3 on the server, then applied 1.4.3 for real.

$ kubectl apply -f work/naive-1.4.2.yaml
deployment.apps/notes-api created
service/notes-api created
[exit 0]
$ kubectl apply --dry-run=server -f work/naive-1.4.3.yaml
service/notes-api configured (server dry run)
The Deployment "notes-api" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"notes-api","app.kubernetes.io/part-of":"platform","app.kubernetes.io/version":"1.4.3"}}: field is immutable
[exit 1]
--- the same apply for real: one object changes, the other is refused
$ kubectl apply -f work/naive-1.4.3.yaml
service/notes-api configured
The Deployment "notes-api" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"notes-api","app.kubernetes.io/part-of":"platform","app.kubernetes.io/version":"1.4.3"}}: field is immutable
[exit 1]
--- service notes-api now selects: app.kubernetes.io/name=notes-api,app.kubernetes.io/part-of=platform,app.kubernetes.io/version=1.4.3
$ kubectl get pods -l app.kubernetes.io/name=notes-api,app.kubernetes.io/part-of=platform,app.kubernetes.io/version=1.4.3 -o name
[exit 0]
$ kubectl get pods --show-labels
NAME               READY   STATUS    RESTARTS   AGE   LABELS
notes-api-manual   0/1     Pending   0          0s    app.kubernetes.io/name=notes-api,app.kubernetes.io/part-of=platform,app.kubernetes.io/version=1.4.2
[exit 0]

The server dry run predicted the failure. The real apply then did the same: kubectl sent the objects one by one, so the Service change landed and the Deployment change was refused. The Service now selects version 1.4.3, so kubectl get pods -l returned nothing, while the only Pod, the one created by hand in step 6, still carried version 1.4.2. The selector moved ahead of the Pods it was supposed to find.

The split library from concept A avoids this. Step 7 applied the 1.4.2 file, then dry-ran 1.4.3 and read back the stored selector.

$ kubectl apply --dry-run=server -f work/split-1.4.3.yaml
deployment.apps/notes-api configured (server dry run)
service/notes-api configured (server dry run)
[exit 0]
$ kubectl get deployment notes-api -o jsonpath={.spec.selector.matchLabels}{"\n"}
{"app.kubernetes.io/name":"notes-api","app.kubernetes.io/part-of":"platform"}
[exit 0]

Release 1.4.3 was accepted by the server, and the stored selector holds only name and part-of. The version appears only in metadata.labels, which can change freely. This release ran as a dry run only.

Symptom: The first release after adding a version label fails with "field is immutable", and at the same moment the Service stops matching the running Pods.

Cause: The version label is in both the Deployment selector and the Service selector. The Service update went through and the Deployment update was refused.

Fix: Keep selectors to labels that never change (name, part-of) and put the version only in metadata.labels. If a selector really must change, the Deployment has to be deleted and recreated, because in apps/v1 the selector is immutable after creation.

Putting it together

The last piece is a gate that a pull request runs in continuous integration (CI), the automated checks that run before a change merges. Contracts run first, then the API server.

#!/usr/bin/env bash
# check.sh: the gate a pull request runs. Contracts first, then the API server.
set -euo pipefail
mkdir -p out
for app in apps/*.ncl; do
  name=$(basename "$app" .ncl)
  nickel export --format yaml "$app" > "out/$name.yaml"
  kubectl apply --dry-run=server -f "out/$name.yaml"
done
echo "all services passed contracts and server-side validation"

For every app file the script exports YAML and then asks the server to dry-run it. set -e stops the script at the first failing command, so a contract error ends the run before kubectl is called. The echo on the last line prints only when every service passed both stages. Step 8 ran it on the good service, then copied the latest example into apps/ and ran it again.

$ ./check.sh
deployment.apps/notes-api unchanged (server dry run)
service/notes-api unchanged (server dry run)
all services passed contracts and server-side validation
[exit 0]
$ ./check.sh
error: contract broken by the value of `image`
       image `ghcr.io/example/notes-api:latest` uses the moving tag `latest`
...
[exit 1]

The first run reports "unchanged" because the same objects had already been applied earlier in step 7. The second run stopped on the contract error, and no kubectl line appears after it. The gate needs an API server to talk to, and the lab used a throwaway one. A server-side dry run also needs the same permissions as the real request, since "Authorization for dry-run and non-dry-run requests is identical."

Everything here ran on aarch64-darwin with nickel 1.18.0, kubectl and kube-apiserver v1.37.1, and etcd 3.6.15, using AlwaysAllow authorization and no nodes. The lab did not verify Pods running, EndpointSlices, admission webhooks, Linux, or any live cluster.

Check yourself

Sources

1. Nickel user manual, Contracts
2. Nickel user manual, Syntax (Include expressions)
3. Nickel user manual, Tutorial
4. Nickel user manual, Merging
5. Kubernetes docs, Deployments
6. Kubernetes docs, Service
7. Kubernetes docs, Labels and Selectors
8. Kubernetes docs, API concepts
9. Kubernetes docs, kubectl apply reference
10. Nigel Poulton, The Kubernetes Book, chapters 6 and 7