One resource, four writes resourceVersion, 409 Conflict, and Status errors in axum

Send back the version you read; a stale write gets 409 instead of erasing someone else's change.

The problem

Two operators manage the same ConfigMap named flags. A ConfigMap is a Kubernetes object that holds a small set of key/value settings, and this one holds a feature flag: mode: blue and rollout: 10. Operator A wants to switch the mode to green. Operator B, in another terminal, wants to raise the rollout to 50. Both start by reading the object, so both hold a copy that says mode: blue and rollout: 10.

Each operator edits their own copy and saves it by sending the whole object back with an HTTP PUT. A's PUT arrives first, and the server stores mode: green, rollout: 10. B's PUT arrives second. B's copy says mode: blue, rollout: 50, so the server stores that. The mode is blue again. A's change is gone, nobody received an error, and A's tool still reports success.

This is the lost update. The server did what each request asked. The defect is that a PUT replaces the entire object, and B's request was built from data that stopped being true the moment A's write landed. Nothing in the request told the server which state the edit was based on. This lesson builds a small axum server that closes that gap, so B's write is refused and B has to look again before trying.

The idea, in plain words

Every stored object carries a string called resourceVersion, and the server changes it on every write. A client that reads an object receives the current version along with the data. When it writes the object back, it must send the version it read. If the stored object has been written since, the version no longer matches. The server then refuses the write with HTTP 409 Conflict and changes nothing. The client re-reads the object, reapplies its edit to the fresh copy, and tries again with the new version. The check and the write happen as one step on the server. Two clients therefore cannot both succeed from the same starting version, and nobody's change is silently erased. This technique is called optimistic concurrency. Nobody takes a lock while editing. Clients assume nobody else will interfere, and the version check catches the cases where that assumption was wrong.

Concept A: The resource and its URL shape

Start with the vocabulary the Kubernetes API documentation uses. A resource type is the name used in the URL, such as configmaps. Every resource type has an object schema called a kind, and the kind for configmaps is ConfigMap. A list of instances of a resource type is a collection, and a single instance is a resource. The documentation also separates the verbs. To list means to return a collection, to get means to retrieve one resource, and an update (HTTP PUT) is different from a patch (HTTP PATCH).

URLs follow from those terms. Core resources use /api instead of /apis and leave out the group segment. A ConfigMap lives in a namespace, so the collection is /api/v1/namespaces/{ns}/configmaps and one instance is that path plus /{name}. The docs say: "Within a namespace, an object's unique identity is defined by the tuple of its API Group, Resource, Namespace, and Name." The lab serves one group and one resource type, so the identity it needs is the pair (namespace, name). Both parts are needed because two namespaces can each hold a ConfigMap called flags. Without the namespace in the key, one team's flags would overwrite another's.

Here are the two structs the lab uses to model the object:

#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "camelCase")]
pub struct ObjectMeta {
    pub name: String,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub namespace: Option<String>,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub resource_version: Option<String>,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub uid: Option<String>,
}

#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "camelCase")]
pub struct ConfigMap {
    #[serde(default = "api_version")]
    pub api_version: String,
    #[serde(default = "kind")]
    pub kind: String,
    pub metadata: ObjectMeta,
    #[serde(default)]
    pub data: BTreeMap<String, String>,
}

ConfigMap is the whole object. apiVersion and kind default to "v1" and "ConfigMap" when a client omits them. metadata is required, and data is a sorted map of string keys to string values that defaults to empty. ObjectMeta holds the bookkeeping. name is required. namespace, resourceVersion and uid are Options so that a client can leave them out of a create request and the server can fill them in. skip_serializing_if keeps absent fields out of the JSON instead of printing null. The attribute that matters most is rename_all = "camelCase". Rust calls the field resource_version, and the attribute makes the JSON name resourceVersion, which is what Kubernetes clients send and expect.

The router maps the URL shapes onto handlers:

pub fn app() -> Router {
    Router::new()
        .route("/api/v1/namespaces/{ns}/configmaps", get(list).post(create))
        .route(
            "/api/v1/namespaces/{ns}/configmaps/{name}",
            get(read).put(replace).delete(remove),
        )
        .with_state(AppState::default())
}

Each route call pairs a path with a method router. Braces mark captures in axum 0.8, so {ns} and {name} each match one path segment and hand it to the handler. The collection path takes GET (list) and POST (create). The instance path takes GET (read), PUT (replace) and DELETE (remove). with_state attaches the shared AppState, an Arc<Mutex<Store>> whose map is keyed by (namespace, name). Inside a handler, State extracts that shared state and Path extracts the captures. It yields one String for one capture and a tuple for two. Json parses the request body. It consumes the body, so it has to be the last parameter.

Common mistake: wrong names. Symptom: a client that sent resourceVersion on a PUT gets 400 "update requires metadata.resourceVersion", and responses contain resource_version, which no Kubernetes client looks for. Cause: the rename_all = "camelCase" attribute is missing. In this code the incoming resourceVersion is then an unknown key that serde skips, so the Rust field stays None. A second variant is a body whose metadata.name differs from the {name} in the URL, which the replace handler rejects with 400. Fix: put the attribute on every struct with multi-word fields, and make the body's name equal the URL's name.

Concept B: resourceVersion and the conditional PUT

resourceVersion is a string in metadata. For an object it identifies the version at which that object was last modified. The documentation describes resource versions as strings that identify the server's internal version of an object, and it is firm about handling: "Resource versions must be passed unmodified back to the server."

The reason this exists is the PUT rule. The docs say that for a PUT, "it is the client's responsibility to specify the resourceVersion (taking this from the object being updated)". The server uses it "so that the API server can detect lost updates and reject requests made by a client that is out of date". On a stale version it returns 409 Conflict. A server that skipped the comparison would be back in the opening scenario, because B's old copy would simply win.

The server assigns the version, so create is where a version first appears:

async fn create(
    State(st): State<AppState>,
    Path(ns): Path<String>,
    Json(mut cm): Json<ConfigMap>,
) -> Result<(StatusCode, Json<ConfigMap>), ApiError> {
    if cm.metadata.resource_version.is_some() {
        return Err(ApiError::new(StatusCode::BAD_REQUEST, "BadRequest",
            "create cannot set resourceVersion"));
    }
    let mut s = st.0.lock().unwrap();
    let key = (ns.clone(), cm.metadata.name.clone());
    if s.objects.contains_key(&key) {
        return Err(ApiError::new(StatusCode::CONFLICT, "AlreadyExists",
            format!("configmaps \"{}\" already exists", cm.metadata.name)));
    }
    s.rv += 1;
    cm.metadata.namespace = Some(ns);
    cm.metadata.resource_version = Some(s.rv.to_string());
    cm.metadata.uid = Some(format!("uid-{}", s.rv));
    s.objects.insert(key, cm.clone());
    Ok((StatusCode::CREATED, Json(cm)))
}

A client that supplies its own resourceVersion on create gets 400, because only the server hands out versions. After that, the handler takes the lock and checks whether (namespace, name) already exists. If it does, the reply is 409 with the reason AlreadyExists. Otherwise it bumps the shared version number, stamps the namespace from the URL, the version and a uid onto the object, stores it and returns 201 with the stored object. The lab keeps one version number for every ConfigMap, so each write anywhere gets a new value. The check and the insert both happen while the lock is held. Two simultaneous creates of the same name therefore cannot both pass the duplicate check.

Replace is the handler where the lost update is prevented:

async fn replace(
    State(st): State<AppState>,
    Path((ns, name)): Path<(String, String)>,
    Json(mut cm): Json<ConfigMap>,
) -> Result<Json<ConfigMap>, ApiError> {
    if cm.metadata.name != name {
        return Err(ApiError::new(StatusCode::BAD_REQUEST, "BadRequest",
            "metadata.name does not match the URL"));
    }
    let Some(sent) = cm.metadata.resource_version.clone() else {
        return Err(ApiError::new(StatusCode::BAD_REQUEST, "BadRequest",
            "update requires metadata.resourceVersion"));
    };
    let mut s = st.0.lock().unwrap();
    let key = (ns.clone(), name.clone());
    let Some(current) = s.objects.get(&key) else {
        return Err(ApiError::new(StatusCode::NOT_FOUND, "NotFound",
            format!("configmaps \"{name}\" not found")));
    };
    if current.metadata.resource_version.as_deref() != Some(sent.as_str()) {
        return Err(ApiError::new(StatusCode::CONFLICT, "Conflict", format!(
            "the object has been modified; sent resourceVersion {sent}, current {}",
            current.metadata.resource_version.as_deref().unwrap_or("")
        )));
    }
    let uid = current.metadata.uid.clone();
    s.rv += 1;
    cm.metadata.namespace = Some(ns);
    cm.metadata.uid = uid;
    cm.metadata.resource_version = Some(s.rv.to_string());
    s.objects.insert(key, cm.clone());
    Ok(Json(cm))
}

The handler checks the request in order of cheapness. The body's name must match the URL (400). A resourceVersion must be present (400). The object must exist (404). The decisive line is the comparison of the stored version against the one the client sent. If they differ, the handler returns 409 with reason Conflict, and the message names both versions. If they match, the handler keeps the stored uid, bumps the shared version, stamps it on the new body and stores it. Copying the uid from the stored object means a PUT can change data but never identity: whatever uid the client sends is overwritten. The reference server shown later is stricter and answers a mismatched uid with 409 Conflict and the message "UID is immutable". The stored object is replaced wholesale, so any key missing from the client's body disappears from the stored object. The comparison runs while the lock is still held, so no other write can slip between the check and the store.

Keep two rules about versions in mind. First, the lab compares for equality only. It asks "is this the same string?" and never "is this older or newer?". Second, the lab's versions look like small integers (1, 2, 3), which tempts clients to parse them or compute the next one. Don't. The documentation says versions are opaque and must be passed back unmodified. A client that guesses rv + 1 is relying on an internal detail that the server is free to change.

The lab also refuses a PUT that has no resourceVersion at all, with a 400. That is this lab's choice, and the reference server shown below makes the same choice for most objects. This lesson makes no claim about how any other server treats an unversioned PUT.

Here is the run for the scenario from the top of the lesson. Client A and client B both GET flags and both hold version 1. Then each sends a PUT:

PUT by A (rv 1)              200  rv=2   reason=-
PUT by B (stale rv 1)        409  rv=-   reason=Conflict
{
  "apiVersion": "v1",
  "code": 409,
  "kind": "Status",
  "message": "the object has been modified; sent resourceVersion 1, current 2",
  "metadata": {},
  "reason": "Conflict",
  "status": "Failure"
}
PUT by B after re-read       200  rv=3   reason=-

A's PUT carries version 1, which matches the stored version, so it succeeds and the object moves to version 2. B's PUT also carries version 1. The stored version is now 2, so the server answers 409 and stores nothing. The message in the body says exactly what happened: B sent 1 and the current version is 2. B then does the correct thing. It GETs the object again, which returns A's green value at version 2, applies its own edit to that fresh copy and sends it. The server accepts it and the object moves to version 3. The test stale_put_is_conflict_and_reread_succeeds asserts every step, including that after the 409 a fresh read still shows A's green. A's write survived. In the lab both clients change the same key, but the protection is identical when they change different keys, as in the opening story.

Re-reading and re-editing also avoids a second hazard the docs mention: "You might accidentally drop fields if you decode an object locally ... and then drop them as part of your update." The retry should apply the edit to the object just fetched, not to an old local copy. The docs also say a patch "is typically appropriate if the change that the client wants to make isn't conditional on the existing data". This lesson covers only the conditional replace.

Common mistake: retrying the stale body. Symptom: a client gets 409 Conflict on its PUT, retries, and keeps getting 409 until it gives up. Cause: the retry resends the same body, and that body still carries the old resourceVersion, so the comparison fails every time. The docs put the burden on the client: "Kubernetes always detects the conflict, but you as the client author need to implement retries." Fix: on a 409 with reason Conflict, GET the object again, reapply the edit to the fresh copy, and PUT that copy. Cap the number of attempts.

Concept C: Errors as Status objects through IntoResponse

When a Kubernetes API call has no object to hand back, as with a failed write, the body is a Status. The reference describes it as "a return value for calls that don't return other objects." Its fields include kind, apiVersion, metadata, a status that is either "Success" or "Failure", a human-readable message, a code with the suggested HTTP status (0 if not set), and a reason, a machine-readable description of why the operation failed.

The reason matters because clients branch on it. Parsing English messages breaks the first time someone rewords one. A fixed shape with a short reason string lets a client write match reason. The reference puts the relationship exactly: "A Reason clarifies an HTTP status code but does not override it." The code stays authoritative, and the reason says which failure behind that code occurred.

This lab has two failures that share code 409. AlreadyExists means a create collided with an existing object, so the client usually reads that object and carries on, a create-or-get. Conflict means a replace used a stale version, so the client re-reads and retries as in Concept B. The status code alone cannot tell them apart. Only the reason can.

In axum, a handler may return anything that implements IntoResponse, and Result<T, E> implements it when both T and E do. The lab defines one error type and implements the trait once:

#[derive(Debug)]
pub struct ApiError {
    code: StatusCode,
    reason: &'static str,
    message: String,
}
impl ApiError {
    fn new(code: StatusCode, reason: &'static str, message: impl Into<String>) -> Self {
        Self { code, reason, message: message.into() }
    }
}
impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        let body = json!({
            "kind": "Status", "apiVersion": "v1", "metadata": {},
            "status": "Failure", "message": self.message,
            "reason": self.reason, "code": self.code.as_u16(),
        });
        (self.code, Json(body)).into_response()
    }
}

ApiError holds three things: the HTTP code, a static reason string and a message. into_response builds the Status JSON with status set to "Failure" and code copied from the HTTP status, so the body and the status line cannot disagree. The final line pairs the StatusCode with the JSON body, which sets both the status line and the content type. Because every handler returns Result<_, ApiError>, any branch can write return Err(ApiError::new(...)) and the client receives a Status. The reason is a &'static str, so each reason is a literal in the source and the set of reasons is easy to find.

Two successful responses also follow the Kubernetes shape. The list handler wraps its items in a collection object:

async fn list(State(st): State<AppState>, Path(ns): Path<String>) -> Json<serde_json::Value> {
    let s = st.0.lock().unwrap();
    let items: Vec<&ConfigMap> =
        s.objects.iter().filter(|((n, _), _)| *n == ns).map(|(_, v)| v).collect();
    Json(json!({
        "kind": "ConfigMapList", "apiVersion": "v1",
        "metadata": { "resourceVersion": s.rv.to_string() },
        "items": items,
    }))
}

The handler locks the store, keeps only entries whose namespace equals the captured {ns}, and returns them under items. The detail to notice is metadata.resourceVersion on the list itself. For a collection, that value identifies the version at which the collection was constructed. In the lab run the list reports 3, because the last write before it produced version 3. Individual items still carry their own versions. The list cannot fail, so it returns a bare Json rather than a Result. The delete handler returns a Status too, but with "status": "Success" and a details object naming the deleted resource. A delete of something that is not there falls through to a 404 with reason NotFound.

Common mistake: unlabelled errors. Symptom: a client sees a plain-text body or a bare 500, and cannot tell a duplicate create from a stale write, or a missing object from a server bug. Cause: the handler returned a string or StatusCode::INTERNAL_SERVER_ERROR instead of a Status, or reused one reason for different failures. Fix: route every failure through ApiError, pick the HTTP code first, then give each distinct failure its own reason.

Putting it together

The whole cargo run scenario, with what each line shows about the opening problem:

The fourth row is the fix for the problem at the top. B's copy was based on version 1, the server held version 2, and the write was refused instead of erasing A's change.

The open-source Rust Kubernetes API server hedronetes-h3s (Apache-2.0) performs the same check in crates/h3s-apiserver/src/write.rs:

        let old_stored = fetch(self.api, k, "object not found").await?;
        // Only Helm-owned ConfigMaps and Secrets accept unconditional PUT.
        // Other writes must carry the revision they observed.
        let rv = match value["metadata"]["resourceVersion"]
            .as_str()
            .filter(|s| !s.is_empty())
        {
            Some(s) => {
                let rv = s
                    .parse::<u64>()
                    .map_err(|_| bad("invalid resourceVersion"))?;
                if old_stored.revision != rv {
                    return Err(Failure::new(
                        409,
                        "Conflict",
                        "resourceVersion precondition failed",
                    ));
                }
                rv
            }
            None if matches!(self.target.resource.kind, "ConfigMap" | "Secret")
                && strategy::helm_owned(value, self.target.resource.kind) =>
            {
                old_stored.revision
            }
            None => return Err(bad("update requires metadata.resourceVersion")),
        };

It first loads the stored object, then looks for a non-empty metadata.resourceVersion in the request. If one is present, it is parsed and compared with the stored revision. A mismatch produces a 409 with the reason Conflict, the same pair the lab returns. If none is present, the request is rejected with "update requires metadata.resourceVersion", the same refusal the lab gives. The one exception is spelled out in the comment: Helm-owned ConfigMaps and Secrets accept an unconditional PUT. That is a choice made by that project, not something the lab copies. The same file also rejects a create that sets a version, with "create cannot set resourceVersion", and on create it stamps a random UUID v4 as uid plus a creationTimestamp, where the lab uses the placeholder uid-1. This server parses its own versions as integers internally, which is its right as the issuer. Clients must still treat them as opaque.

Two details separate the lab from a real server. The lab's std::sync::Mutex is held only inside synchronous sections, never across an .await, so no task is suspended while holding the lock. A real server also keeps the version number in storage (etcd in Kubernetes) and not in process memory, whereas the lab's version restarts at zero on every run. Versions also drive watches. A client can watch from a resourceVersion, and the docs note that clusters using etcd 3 preserve changes for the last 5 minutes by default. When the requested history is gone, the client must recognise the status code 410 Gone, clear its local cache, do a fresh get or list, and start watching from the resourceVersion that came back. The collection version on the list response exists for that purpose.

How to run it

Tested with rustc 1.99.0, axum 0.8.9. The dependencies in Cargo.toml are axum 0.8 with the json feature, tokio with macros and rt-multi-thread, tower 0.5 with util, serde with derive, serde_json and http-body-util 0.1.

cargo test
cargo run

cargo test runs five tests, and the last line of the test output for them reads:

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

cargo run executes the scenario in the table above. The driver in src/main.rs calls the router in process through tower::ServiceExt::oneshot, so it opens no port and there is nothing to curl.

Check yourself

1. In the lab run, why does B's PUT return 409 and not 200? Answer: B sent resourceVersion 1, but A's write had already moved the stored version to 2. The versions differ, so the server refuses the write and changes nothing.

1. Both AlreadyExists and Conflict come back with HTTP 409. How does a client tell them apart, and what should it do in each case? Answer: It reads the reason field of the Status body. The reason clarifies the code and does not override it. For AlreadyExists the client reads the existing object and continues. For Conflict it re-reads, reapplies its edit and retries.

1. Why should a client not compute the next version as the current one plus one? Answer: Resource versions are opaque strings that must be passed back unmodified. The lab happens to use small integers, but that is an internal detail the server may change.

1. What does the lab return for a PUT with no resourceVersion, and for a create that sets one? Answer: Both return 400 with reason BadRequest. The first message is "update requires metadata.resourceVersion" and the second is "create cannot set resourceVersion".

Sources

• Kubernetes API Concepts, sections "Terminology", "Object names", "API verbs", "Resource URIs", "Updates to existing resources", "Resource versions" and "Efficient detection of changes". https://kubernetes.io/docs/reference/using-api/api-concepts/
• Kubernetes API reference, "Status" (common definitions). https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/status/
• axum 0.8.9 documentation, crate overview (Router::route, path captures, extractors, IntoResponse, with_state). https://docs.rs/axum/0.8.9/axum/
• axum 0.8.9 documentation, module extract (Json must be the last extractor). https://docs.rs/axum/0.8.9/axum/extract/index.html
• hedronetes-h3s (Apache-2.0), crates/h3s-apiserver/src/write.rs at commit 612b4d2. https://github.com/VirtualMachinist/hedronetes-h3s