One pass, one write a ReplicaSet controller read in Rust
Adopt matching orphans, release drifted Pods, never steal another owner's Pod, cap writes at eight per pass, and pick scale-down victims in a fixed order.
The problem
You want three Pods running. The cluster has some other number. Of the Pods that exist, one carries your labels but belongs to nobody, one carries your labels but belongs to a different owner, and one is yours but someone edited its labels so it no longer matches. A controller has to look at that mess and decide what to write on each pass. It must not take a Pod that belongs to someone else, and it must not count a Pod twice or count one that should not count.
Some terms first. A Pod is the smallest unit a cluster runs: one or more containers scheduled together. A ReplicaSet is an object that says "keep N Pods like this template alive, and recognize them with this selector." A label selector is a set of key/value conditions tested against the labels on a Pod. The apiserver is the HTTP service that stores every cluster object and accepts reads and writes. A controller is a loop that watches those objects and writes changes to move the observed state toward the desired state. One reconcile is one pass of that loop for one object.
Ownership is recorded in a Pod's metadata.ownerReferences, a list naming the objects that own it. The upstream docs say: "A ReplicaSet is linked to its Pods via the Pods' metadata.ownerReferences field, which specifies what resource the current object is owned by." One entry may be flagged controller: true, which marks the single object managing that Pod. A Pod with no controller entry is an orphan. To adopt an orphan is to add your own entry to it. To release a Pod is to remove your entry. The docs describe adoption this way: a Pod "that has no OwnerReference or the OwnerReference is not a Controller and it matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet."
Usually you do not write ReplicaSets yourself: "Usually, you define a Deployment and let that Deployment manage ReplicaSets automatically." Reading a ReplicaSet controller is reading the layer underneath.
The code here is from hedronetes-h3s, a small Kubernetes-compatible control plane written in Rust (VirtualMachinist/hedronetes-h3s at commit 612b4d2, crate h3s-controllers). The files are crates/h3s-controllers/src/workload/replicaset.rs and crates/h3s-controllers/src/workload/mod.rs.
The wiring
use super::*;
use futures_util::StreamExt;
use kube::runtime::{controller::Action, watcher, Controller};
use std::sync::Arc;
pub async fn run_replicaset_controller(client: Client) -> Result<(), Error> {
Controller::new(
Api::<ReplicaSet>::all(client.clone()),
watcher::Config::default(),
)
.owns(Api::<Pod>::all(client.clone()), watcher::Config::default())
.run(
|rs, client: Arc<Client>| async move {
replicaset_once(
(*client).clone(),
&rs.namespace()
.ok_or(Error::Invalid("ReplicaSet namespace missing"))?,
&rs.name_any(),
)
.await?;
Ok(Action::requeue(Duration::from_secs(2)))
},
|_, _: &Error, _| Action::requeue(Duration::from_secs(5)),
Arc::new(client),
)
.for_each(|r| async move {
if let Err(e) = r {
eprintln!("ReplicaSet controller: {e}");
}
})
.await;
Err(Error::Stopped)
}This uses the kube crate's runtime, pinned by the project at =2.0.1. Its docs describe a controller as "an infinite stream of objects to be reconciled" that "continuously calls out to user provided reconcile and error_policy callbacks whenever relevant changes are detected or if errors are seen from reconcile."
Controller::new watches ReplicaSets in all namespaces. .owns adds Pods as children. The docs say that "changes to managed child resources will also trigger the reconciler for the managing object by traversing owner references," and that "all owned Child objects must contain an OwnerReference pointing back to a K."
.run takes three arguments: a reconcile closure, an error policy closure, and a shared context, here the client. A requeue asks the runtime to call reconcile again for the same object after a delay. This code requeues after 2 seconds when the pass succeeds and after 5 seconds when it fails.
The reconcile closure does very little. It pulls the namespace and name out of the object it was handed and calls replicaset_once, which fetches the ReplicaSet again. The docs explain why this is the right shape: the mapping from child to parent "ultimately hides the reason for the reconciliation request, and forces you to write an idempotent reconciler." The reconciler cannot know whether it woke up because a Pod died, a ReplicaSet changed, or a timer fired. It has to compare the world to the goal every time, and running it twice in a row must be harmless.
The 2 second requeue matters for orphans. An orphan has no ownerReference, so the owns mapping has no path from it back to a ReplicaSet, and its arrival triggers nothing. The reading here, inferred from the code and not observed in the lab: a new matching orphan is found on the next timed pass, at most about 2 seconds after the previous one finished.
One pass
pub async fn replicaset_once(client: Client, namespace: &str, name: &str) -> Result<(), Error> {
let api = Api::<ReplicaSet>::namespaced(client.clone(), namespace);
let Some(rs) = api.get_opt(name).await? else {
return Ok(());
};
let rs = serde_json::to_value(rs)?;
if !active(&rs) {
return Ok(());
}
let pods_api = Api::<Pod>::namespaced(client, namespace);
let mut pods = list(&pods_api).await?;
let result = sync(&api, &pods_api, &rs, &mut pods).await;
let selected: Vec<_> = pods.iter().filter(|p| owned(p, &rs) && active(p)).collect();
let minimum = minimum_ready(&rs).unwrap_or(0);
let mut desired = json!({"observedGeneration":rs["metadata"]["generation"],"replicas":selected.len(),"fullyLabeledReplicas":selected.iter().filter(|p|rs["spec"]["template"]["metadata"]["labels"].as_object().is_none_or(|labels|labels.iter().all(|(k,v)|p["metadata"]["labels"][k]==*v))).count(),"readyReplicas":selected.iter().filter(|p|ready(p)).count(),"availableReplicas":selected.iter().filter(|p|available(p,minimum)).count()});
if let Err(error) = &result {
desired["conditions"] = json!([condition(
&rs["status"],
"ReplicaFailure",
"True",
"FailedManageReplicas",
&failure(error),
false,
false
)]);
}
status(&api, &rs, desired).await?;
result
}The steps are:
- Fetch the ReplicaSet by name. If it is gone, return
Ok(()), since there is nothing to manage. - Convert it to a JSON value. If it is not
active, return. A ReplicaSet already being deleted should not create Pods. - List the Pods in the namespace and call
sync, which makes the writes. - Compute status from what is now observed: the Pods that are owned by this ReplicaSet and active. That gives the replica count, the fully labeled count, the ready count and the available count.
- Write the status.
Status comes from the observed list, not from what sync intended. If sync failed, a ReplicaFailure condition is added and the status is still written. Only after that does the function return the sync result, so the error reaches the error policy and the 5 second requeue.
The decision core
async fn sync(
api: &Api<ReplicaSet>,
pods_api: &Api<Pod>,
rs: &Value,
pods: &mut Vec<Value>,
) -> Result<(), Error> {
let desired = replicas(rs)? as usize;
minimum_ready(rs)?;
let template = &rs["spec"]["template"];
let meta = template_metadata(template)?;
// Claim matching orphans and release our mismatches with CAS; never steal another controller's Pod.
for p in pods.iter_mut().filter(|p| active(p)) {
let matches = matches(&rs["spec"]["selector"], &p["metadata"]["labels"]);
let ours = owned(p, rs);
let orphan = !values(&p["metadata"]["ownerReferences"]).any(|o| o["controller"] == true);
if (ours && !matches) || (orphan && matches) {
if !live(api, rs).await? {
return Ok(());
}
let mut refs = p["metadata"]["ownerReferences"]
.as_array()
.cloned()
.unwrap_or_default();
refs.retain(|o| o["uid"] != rs["metadata"]["uid"]);
if orphan && matches {
refs.push(owner(rs));
}
p["metadata"]["ownerReferences"] = refs.into();
replace(pods_api, p).await?;
// Re-list next cycle before acting on adoption/release results.
return Ok(());
}
}
let mut selected: Vec<_> = pods
.iter()
.filter(|p| owned(p, rs) && active(p))
.cloned()
.collect();
if selected.len() < desired {
for _ in 0..(desired - selected.len()).min(BURST) {
if !live(api, rs).await? {
break;
}
let mut metadata = meta.clone();
let name = text(&rs["metadata"], "name")?;
metadata["generateName"] = format!("{}-", &name[..name.len().min(54)]).into();
metadata["namespace"] = rs["metadata"]["namespace"].clone();
metadata["ownerReferences"] = json!([owner(rs)]);
let pod: Pod =
serde_json::from_value(json!({"metadata":metadata,"spec":template["spec"]}))?;
let created = pods_api.create(&PostParams::default(), &pod).await?;
pods.push(serde_json::to_value(created)?);
}
} else if selected.len() > desired {
selected.sort_by_key(|p| {
(
ready(p),
p["spec"]["nodeName"]
.as_str()
.is_some_and(|s| !s.is_empty()),
std::cmp::Reverse(
p["metadata"]["creationTimestamp"]
.as_str()
.unwrap_or("")
.to_owned(),
),
p["metadata"]["uid"].as_str().unwrap_or("").to_owned(),
)
});
for pod in selected.iter().take((selected.len() - desired).min(BURST)) {
if !live(api, rs).await? {
break;
}
delete(pods_api, pod).await?;
// Status is refreshed from API below; conflicts cannot count as deletion.
}
*pods = list(pods_api).await?;
}
Ok(())
}The function has two phases.
Ownership first. For each active Pod it computes three booleans. matches says whether the Pod's labels satisfy the selector. ours says whether this ReplicaSet is the Pod's single controller. orphan says whether no ownerReference is flagged as a controller. A Pod needs attention if it is ours but no longer matches (release), or if it is an orphan that matches (adopt). Anything else is left alone. That includes a matching Pod controlled by someone else: it is neither ours nor an orphan, so neither branch fires.
When a Pod needs attention, the code checks live, rewrites that one Pod's ownerReferences, and returns immediately. The comment states the reason: "Re-list next cycle before acting on adoption/release results." After a rewrite the in-memory list may no longer match the cluster. Counting Pods from it would be a guess, so the pass ends and the next one starts from a fresh list.
Counting second. If no Pod needed a rewrite, selected holds the Pods that are owned and active. Too few: create (desired - selected.len()).min(BURST) Pods, where BURST is 8, so a request for 20 Pods creates at most 8 in one pass. Too many: sort, then delete the first min(excess, BURST) entries in sorted order. Equal: do nothing. Every create and every delete is preceded by live. After deleting, the code re-lists so the status step sees the real state.
The lab crate
The lab copies the pure helpers from the h3s source and replaces every API call in sync with a returned Plan value. That lets the decision run with no apiserver.
Cargo.toml:
[package]
name = "rs_reconcile_lab"
version = "0.1.0"
edition = "2021"
publish = false
[dependencies]
serde_json = "1"src/lib.rs:
//! Pure decision half of a ReplicaSet reconcile, lifted from
//! VirtualMachinist/hedronetes-h3s @ 612b4d2,
//! crates/h3s-controllers/src/workload/{mod.rs,replicaset.rs}.
//! The helpers below are copied verbatim (minus the error type); `plan`
//! mirrors the branch structure of `sync` with every API call replaced by
//! a returned step, so it runs with no apiserver.
use serde_json::{json, Value};
pub const LIMIT: u32 = 4096;
pub const BURST: usize = 8;
fn values(v: &Value) -> impl Iterator<Item = &Value> {
v.as_array().into_iter().flatten()
}
pub fn active(p: &Value) -> bool {
p["metadata"]["deletionTimestamp"].is_null()
&& !matches!(p["status"]["phase"].as_str(), Some("Succeeded" | "Failed"))
}
pub fn ready(p: &Value) -> bool {
active(p)
&& values(&p["status"]["conditions"]).any(|c| c["type"] == "Ready" && c["status"] == "True")
}
pub fn controller(v: &Value) -> Option<&Value> {
let mut owners = values(&v["metadata"]["ownerReferences"]).filter(|o| o["controller"] == true);
let first = owners.next()?;
if owners.next().is_some() {
None
} else {
Some(first)
}
}
pub fn owned(child: &Value, parent: &Value) -> bool {
controller(child).is_some_and(|o| {
o["uid"] == parent["metadata"]["uid"]
&& o["name"] == parent["metadata"]["name"]
&& o["kind"] == parent["kind"]
&& o["apiVersion"] == parent["apiVersion"]
}) && child["metadata"]["namespace"] == parent["metadata"]["namespace"]
}
pub fn owner(parent: &Value) -> Value {
json!({"apiVersion":parent["apiVersion"],"kind":parent["kind"],"name":parent["metadata"]["name"],"uid":parent["metadata"]["uid"],"controller":true,"blockOwnerDeletion":true})
}
pub fn matches(selector: &Value, labels: &Value) -> bool {
let label_count = selector["matchLabels"].as_object().map_or(0, |m| m.len());
let expr_count = values(&selector["matchExpressions"]).count();
label_count + expr_count > 0
&& selector["matchLabels"]
.as_object()
.is_none_or(|m| m.iter().all(|(k, v)| labels[k] == *v))
&& values(&selector["matchExpressions"]).all(|r| {
let Some(key) = r["key"].as_str().filter(|k| !k.is_empty()) else {
return false;
};
let present = labels.get(key).is_some();
let vals: Vec<_> = values(&r["values"]).collect();
match r["operator"].as_str() {
Some("In") => !vals.is_empty() && present && vals.contains(&&labels[key]),
Some("NotIn") => !vals.is_empty() && (!present || !vals.contains(&&labels[key])),
Some("Exists") => vals.is_empty() && present,
Some("DoesNotExist") => vals.is_empty() && !present,
_ => false,
}
})
}
pub fn replicas(v: &Value) -> Result<i64, &'static str> {
v["spec"]["replicas"]
.as_i64()
.filter(|n| (0..=i64::from(LIMIT)).contains(n))
.ok_or("replica count exceeds the current bounded controller range")
}
/// One reconcile pass's intended writes.
#[derive(Debug, PartialEq)]
pub enum Plan {
/// Rewrite this Pod's ownerReferences (adopt or release), then stop the pass.
Rewrite { pod: String, adopt: bool },
Create(usize),
Delete(Vec<String>),
Nothing,
}
fn name(p: &Value) -> String {
p["metadata"]["name"].as_str().unwrap_or("").to_owned()
}
pub fn plan(rs: &Value, pods: &[Value]) -> Result<Plan, &'static str> {
let desired = replicas(rs)? as usize;
for p in pods.iter().filter(|p| active(p)) {
let m = matches(&rs["spec"]["selector"], &p["metadata"]["labels"]);
let ours = owned(p, rs);
let orphan = !values(&p["metadata"]["ownerReferences"]).any(|o| o["controller"] == true);
if (ours && !m) || (orphan && m) {
return Ok(Plan::Rewrite { pod: name(p), adopt: orphan && m });
}
}
let mut selected: Vec<&Value> = pods.iter().filter(|p| owned(p, rs) && active(p)).collect();
if selected.len() < desired {
Ok(Plan::Create((desired - selected.len()).min(BURST)))
} else if selected.len() > desired {
selected.sort_by_key(|p| {
(
ready(p),
p["spec"]["nodeName"].as_str().is_some_and(|s| !s.is_empty()),
std::cmp::Reverse(p["metadata"]["creationTimestamp"].as_str().unwrap_or("").to_owned()),
p["metadata"]["uid"].as_str().unwrap_or("").to_owned(),
)
});
Ok(Plan::Delete(
selected.iter().take((selected.len() - desired).min(BURST)).map(|p| name(p)).collect(),
))
} else {
Ok(Plan::Nothing)
}
}tests/plan.rs:
use rs_reconcile_lab::*;
use serde_json::{json, Value};
fn rs(replicas: i64) -> Value {
json!({"apiVersion":"apps/v1","kind":"ReplicaSet",
"metadata":{"name":"web","namespace":"default","uid":"rs-uid-1","generation":1},
"spec":{"replicas":replicas,"selector":{"matchLabels":{"app":"web"}},
"template":{"metadata":{"labels":{"app":"web"}},"spec":{"containers":[]}}}})
}
fn pod(name: &str, app: &str, owners: Value, ready: bool, node: &str, created: &str) -> Value {
json!({"metadata":{"name":name,"namespace":"default","uid":format!("uid-{name}"),
"labels":{"app":app},"ownerReferences":owners,"creationTimestamp":created},
"spec":{"nodeName":node},
"status":{"phase":"Running","conditions":[{"type":"Ready","status": if ready {"True"} else {"False"}}]}})
}
fn mine(r: &Value) -> Value { json!([owner(r)]) }
#[test]
fn t1_orphan_with_matching_labels_is_adopted_and_the_pass_stops() {
let r = rs(3);
let pods = vec![pod("stray", "web", json!([]), true, "n1", "2026-10-26T09:00:00Z")];
let p = plan(&r, &pods).unwrap();
println!("t1 {p:?}");
assert_eq!(p, Plan::Rewrite { pod: "stray".into(), adopt: true });
}
#[test]
fn t2_pod_controlled_by_someone_else_is_never_stolen() {
let r = rs(1);
let other = json!([{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"other","uid":"rs-uid-9","controller":true}]);
let pods = vec![pod("theirs", "web", other, true, "n1", "2026-10-26T09:00:00Z")];
let p = plan(&r, &pods).unwrap();
println!("t2 {p:?}");
assert_eq!(p, Plan::Create(1));
}
#[test]
fn t3_owned_pod_whose_labels_drifted_is_released() {
let r = rs(1);
let pods = vec![pod("drift", "api", mine(&r), true, "n1", "2026-10-26T09:00:00Z")];
let p = plan(&r, &pods).unwrap();
println!("t3 {p:?}");
assert_eq!(p, Plan::Rewrite { pod: "drift".into(), adopt: false });
}
#[test]
fn t4_scale_up_is_capped_at_burst_per_pass() {
let p = plan(&rs(20), &[]).unwrap();
println!("t4 {p:?}");
assert_eq!(p, Plan::Create(8));
}
#[test]
fn t5_scale_down_picks_unready_then_unscheduled_then_newest() {
let r = rs(1);
let m = mine(&r);
let pods = vec![
pod("old-ready", "web", m.clone(), true, "n1", "2026-10-26T08:00:00Z"),
pod("new-ready", "web", m.clone(), true, "n1", "2026-10-26T09:30:00Z"),
pod("unscheduled", "web", m.clone(), true, "", "2026-10-26T07:00:00Z"),
pod("not-ready", "web", m.clone(), false, "n1", "2026-10-26T06:00:00Z"),
];
let p = plan(&r, &pods).unwrap();
println!("t5 {p:?}");
assert_eq!(p, Plan::Delete(vec!["not-ready".into(), "unscheduled".into(), "new-ready".into()]));
}
#[test]
fn t6_terminating_and_finished_pods_do_not_count() {
let r = rs(2);
let m = mine(&r);
let mut gone = pod("gone", "web", m.clone(), true, "n1", "2026-10-26T09:00:00Z");
gone["metadata"]["deletionTimestamp"] = json!("2026-10-26T09:59:00Z");
let mut done = pod("done", "web", m.clone(), false, "n1", "2026-10-26T09:00:00Z");
done["status"]["phase"] = json!("Succeeded");
let live = pod("live", "web", m, true, "n1", "2026-10-26T09:00:00Z");
let p = plan(&r, &[gone, done, live]).unwrap();
println!("t6 {p:?}");
assert_eq!(p, Plan::Create(1));
}
#[test]
fn t7_empty_selector_matches_nothing_and_two_controllers_means_no_owner() {
let empty = json!({});
println!("t7 empty selector matches = {}", matches(&empty, &json!({"app":"web"})));
assert!(!matches(&empty, &json!({"app":"web"})));
let r = rs(1);
let two = json!([owner(&r), {"apiVersion":"apps/v1","kind":"ReplicaSet","name":"x","uid":"u2","controller":true}]);
let p = pod("contested", "web", two, true, "n1", "2026-10-26T09:00:00Z");
println!("t7 controller() = {:?}, owned = {}", controller(&p), owned(&p, &r));
assert!(controller(&p).is_none());
assert!(!owned(&p, &r));
}
#[test]
fn t8_out_of_range_replicas_is_refused_before_any_write() {
let e = plan(&rs(5000), &[]).unwrap_err();
println!("t8 {e}");
let e2 = plan(&rs(-1), &[]).unwrap_err();
assert_eq!(e, e2);
}
#[test]
fn t9_converged_owner_does_nothing() {
let r = rs(2);
let m = mine(&r);
let pods = vec![
pod("a", "web", m.clone(), true, "n1", "2026-10-26T09:00:00Z"),
pod("b", "web", m, true, "n1", "2026-10-26T09:01:00Z"),
];
let p = plan(&r, &pods).unwrap();
println!("t9 {p:?}");
assert_eq!(p, Plan::Nothing);
}Reading the output
The command was cargo test --offline -- --nocapture --test-threads=1 with the Compiling lines removed:
Locking 11 packages to highest compatible versions
Finished `test` profile [unoptimized + debuginfo] target(s) in 2.59s
Running unittests src/lib.rs (target/debug/deps/rs_reconcile_lab-db1fd400a83db888)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running tests/plan.rs (target/debug/deps/plan-bd1b793da5276907)
running 9 tests
test t1_orphan_with_matching_labels_is_adopted_and_the_pass_stops ... t1 Rewrite { pod: "stray", adopt: true }
ok
test t2_pod_controlled_by_someone_else_is_never_stolen ... t2 Create(1)
ok
test t3_owned_pod_whose_labels_drifted_is_released ... t3 Rewrite { pod: "drift", adopt: false }
ok
test t4_scale_up_is_capped_at_burst_per_pass ... t4 Create(8)
ok
test t5_scale_down_picks_unready_then_unscheduled_then_newest ... t5 Delete(["not-ready", "unscheduled", "new-ready"])
ok
test t6_terminating_and_finished_pods_do_not_count ... t6 Create(1)
ok
test t7_empty_selector_matches_nothing_and_two_controllers_means_no_owner ... t7 empty selector matches = false
t7 controller() = None, owned = false
ok
test t8_out_of_range_replicas_is_refused_before_any_write ... t8 replica count exceeds the current bounded controller range
ok
test t9_converged_owner_does_nothing ... t9 Nothing
ok
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests rs_reconcile_lab
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sAll nine tests passed on the first run. The running 0 tests blocks are the empty unit-test and doc-test targets.
t1. The ReplicaSet wants 3 Pods and sees one matching orphan. The plan is Rewrite { pod: "stray", adopt: true }, not a create. The pass stops after the rewrite even though two Pods are still missing, and the next pass re-lists before it counts anything.
t2. A Pod named theirs matches the labels but its controller entry points at rs-uid-9. It is not ours and it is not an orphan, so it is skipped. The ReplicaSet wants 1 Pod, counts 0 of its own, and plans Create(1). A Pod with matching labels but another owner is neither claimed nor counted.
t3. The Pod drift is owned by us but now has app: api. The plan is Rewrite { pod: "drift", adopt: false }. The same variant covers adopt and release, and the adopt flag tells them apart.
t4. Asking for 20 with no Pods gives Create(8). The remaining 12 take two more passes, 8 and then 4.
t5. The ReplicaSet wants 1 of 4 owned Pods, so three go. Sorting by the tuple puts not-ready first, because false sorts before true. Among ready Pods, unscheduled comes next because an empty nodeName makes the second element false. Among ready, scheduled Pods the newest comes first, so new-ready (09:30) goes before old-ready (08:00). The kept Pod is old-ready.
t6. gone has a deletion timestamp and done has phase Succeeded. Neither is active, so neither counts. One live Pod against a target of 2 gives Create(1).
t7. An empty selector matches nothing, because the first condition in matches requires at least one label or expression. A Pod listing two controller owners makes controller() return None, so owned is false. It is also not an orphan, because orphan is true only when no entry is flagged controller. Nobody owns it and nobody adopts it.
t8. Both 5000 and -1 produce replica count exceeds the current bounded controller range, before any other decision is made. The bound is LIMIT, 4096.
t9. Two owned, matching, active Pods against a target of 2 gives Nothing.
Scale-down order: what this project does and what the docs describe
The upstream docs describe this order for choosing which Pods to remove: "Pending (and unschedulable) pods are scaled down first. If controller.kubernetes.io/pod-deletion-cost annotation is set, then the pod with the lower value will come first. Pods on nodes with more replicas come before pods on nodes with fewer replicas. If the pods' creation times differ, the pod that was created more recently comes before the older pod (the creation times are bucketed on an integer log scale). If all of the above match, then selection is random."
The h3s sort key is (ready, scheduled, newest creationTimestamp, uid). Unready Pods go first, then unscheduled ones, then the newest, then a uid comparison breaks ties. The key does not read pod-deletion-cost and does not count replicas per node. Its final tiebreak is deterministic where the docs say random. The creation timestamp is compared as a plain string, with no bucketing. That string comparison gives time order only because timestamps in the RFC 3339 form 2026-10-26T09:30:00Z share one width and one zone, so character order and time order agree. A timestamp written with an offset such as -04:00 would break that agreement.
t5 shows the ranks stacking. unscheduled was created at 07:00, earlier than old-ready at 08:00, and it still goes first, because being unscheduled outranks age in the tuple. Age only decides between Pods that tie on both readiness and scheduling.
The lesson takes no side between the two orders. The docs describe the upstream design. This project implements a smaller one that gives the same answer every time for the same input, which is what makes t5 testable.
The writes the lab did not run
Two helpers in mod.rs guard every write. Both are quoted source here, not run.
async fn live<K>(api: &Api<K>, object: &Value) -> Result<bool, Error>
where
K: Clone + Debug + DeserializeOwned + Resource<DynamicType = ()>,
{
let (_, name, uid) = identity(object)?;
Ok(api.get_opt(name).await?.is_some_and(|p| {
p.uid().as_deref() == Some(uid)
&& p.meta().deletion_timestamp.is_none()
&& p.meta().generation == object["metadata"]["generation"].as_i64()
}))
}
async fn delete<K>(api: &Api<K>, object: &Value) -> Result<bool, Error>
where
K: Clone + Debug + DeserializeOwned,
{
let dp = DeleteParams {
propagation_policy: Some(PropagationPolicy::Background),
preconditions: Some(Preconditions {
uid: Some(text(&object["metadata"], "uid")?.into()),
resource_version: Some(text(&object["metadata"], "resourceVersion")?.into()),
}),
..Default::default()
};
match api.delete(text(&object["metadata"], "name")?, &dp).await {
Ok(_) => Ok(true),
Err(kube::Error::Api(e)) if matches!(e.code, 404 | 409) => Ok(false),
Err(e) => Err(e.into()),
}
}live is called with the ReplicaSet before every adopt, release, create and delete. It fetches the current object by name and returns true only if three things hold: the uid is the same as in the snapshot, there is no deletion timestamp, and the generation equals the snapshot's. The uid check catches a ReplicaSet that was deleted and recreated under the same name. The generation is a number the apiserver bumps when the spec changes. A mismatch means someone edited the ReplicaSet after this pass read it. In that case the sync stops (return Ok(()) in the ownership loop, break in the create and delete loops) and the next pass starts from fresh data.
delete carries two preconditions, the Pod's uid and resourceVersion. A resourceVersion is a stamp the apiserver changes whenever an object is written. This is optimistic concurrency: act on the version you read, and let the server refuse the write if the object has changed since. A 404 means the Pod is already gone, and a 409 means the preconditions did not hold. Both map to Ok(false), "did not delete". The caller ignores the boolean, and the comment next to the call says "Status is refreshed from API below; conflicts cannot count as deletion." The re-list that follows is the source of truth, and the status step never assumes a delete succeeded.
Pitfalls as diagnostics
**t2 Create(1) when a matching Pod is sitting in the namespace.** The Pod is controlled by another owner, so the ReplicaSet does not count it. The fix is on the label side: make the two selectors disjoint, or accept that each owner keeps its own Pods.
**t7 controller() = None, owned = false.** The Pod lists two entries with controller: true. It is not owned, and it is not an orphan, so no ReplicaSet adopts it. The fix is to remove one of the two controller flags so one owner is clear, or to remove both so the Pod becomes an orphan that matching ReplicaSets can adopt.
**t1 Rewrite { pod: "stray", adopt: true } and no create, even though replicas are short.** This is by design. The comment says "Re-list next cycle before acting on adoption/release results." If the same Rewrite keeps appearing for the same Pod across passes, the write is not landing. The replace() helper was not part of the excerpts, so look there.
**t8 replica count exceeds the current bounded controller range.** The replicas value is outside 0 to 4096. Reading the code, a missing or non-integer value takes the same path, because as_i64() returns None for it. Set spec.replicas to an integer in range.
**t7 empty selector matches = false.** A selector with no matchLabels and no matchExpressions matches no Pod, so the ReplicaSet will create Pods and never count them. Add at least one label or expression.
**A Pod with pod-deletion-cost set is deleted anyway.** This one comes from reading the source; the lab did not set the annotation. The sort key is (ready, scheduled, newest creationTimestamp, uid) and nothing in it reads the annotation. Protecting a specific Pod from scale-down needs a different mechanism in this project, or a change to the key.
What was not run
The lab had no cluster and no apiserver. Nothing here exercised run_replicaset_controller, the Controller::new and owns wiring, the requeue actions, live, replace, delete with its preconditions, or the status writes. Those appear as quoted source, and the claims about them come from reading that source and the kube 2.0.1 docs. The project pins kube =2.0.1 with the runtime feature, and the lab did not compile h3s.
The lab's helpers are copied from h3s, minus the error type. plan is a re-expression of sync's branches and not sync itself. It leaves out minimum_ready, template_metadata, every live check and all I/O. A change in sync after commit 612b4d2 would not show up in these tests.
Takeaway
A ReplicaSet controller is a function from "what exists" to "one next write". It adopts matching orphans, releases owned Pods that stopped matching, and ignores Pods that belong to someone else or to two controllers at once. It does only one ownership rewrite per pass, then re-lists. Creates and deletes are capped at 8 per pass, and live, uid and resourceVersion checks keep a stale snapshot from writing to the wrong object. A reasonable local contribution to h3s would be a unit test that pins the scale-down order from t5 as a documented contract. Another would be a short note in the crate saying that its order differs from the pod-deletion-cost and per-node rules in the upstream docs. Nothing has been filed upstream.
Sources
- hedronetes-h3s @
612b4d2,crates/h3s-controllers/src/workload/replicaset.rsandworkload/mod.rs(read-only; quoted excerpts verbatim). - Kubernetes docs: ReplicaSet, fetched 2026-10-26.
- [kube 2.0.1
runtime::Controller(docs.rs)](https://docs.rs/kube/2.0.1/kube/runtime/struct.Controller.html). - Lab:
crate/in this bundle,crate/lab-test.log(the lab Mac, rustc 1.99.0, serde_json 1.0.154).
Related
- One Kubernetes-style resource in axum: resourceVersion and 409 Conflict
- kube-rs typed resources and the client boundary