Kubernetes Scheduling on AKS — the scheduler said no
A Pending whose owner is kube-scheduler. The system pool is already tainted. The request is the packing input.
<!-- hal:authoritative:yaml -->
§I — Frame
Three days ago this arc put a guest kernel under a Pod on GKE. Six days ago it put a disk under a Pod on EKS and watched the bind stop. Both lessons ended with a Pod that was Pending. Both Pendings had an owner.
Today the owner is the scheduler.
A Pod is admitted. The image is fine. The claim is Bound. kubectl describe pod still says Pending, and the event is 0/6 nodes are available: 3 node(s) had untolerated taint {CriticalAddonsOnly: true}, 3 Insufficient cpu. That is not CSI. That is not gVisor. That is kube-scheduler walking the node list and writing no.
Call the failure the scheduler said no. The API server accepted the object. The controllers that own disks and sandboxes have nothing to do. The node list is the whole vote, and every node voted against this spec.
AKS makes the vote visible as pools. One system pool, tainted so user Pods bounce. One or more user pools, where the work is supposed to land. Azure CNI caps pods per node. The cluster autoscaler adds a node only after the scheduler has already failed. None of that is in the Pod YAML as a pool name. It is in taints, labels, and requests.
§II — Foundations
A pool is a node group with a taint you chose
AKS does not schedule onto "the cluster." It schedules onto nodes, and nodes arrive in pools. The system pool runs CoreDNS, metrics-server, the CNI pods, and the other add-ons AKS will not let you delete. Microsoft taints that pool CriticalAddonsOnly=true:NoSchedule on the default configuration so a Deployment with no toleration cannot land there.
A user pool is a second VMSS with its own SKU, its own max-pods, and usually no such taint. That is Poulton's node isolation, implemented as a purchase order: ring-fence the privileged add-ons, put the app on a different subset of workers (The Kubernetes Book, Ch. 16, pp. 222-223). The taint is the fence. The toleration is the gate.
Bootcamp CKA Q10 is the exam-sized version of the same fence. Taint node01 PERMISSION=granted:NoSchedule. A normal Pod stays Pending. A Pod that names the matching toleration lands. AKS did that to the system pool before you arrived.
Requests are the packing input; limits are not
Kubernetes Up and Running splits the two numbers the scheduler actually reads. A request is the minimum the scheduler reserves on a node. A limit is the maximum the kubelet will let the container consume once it is running (K8sUR3, resource requests and limits, pp. 86-88). The scheduler packs requests. It does not pack limits.
A Pod that asks for 2 CPU on a user pool of 2-vCPU nodes, after the kube-reserved and the CNI DaemonSet have taken their cut, does not fit. The event is Insufficient cpu. Raising the limit does nothing. Lowering the request, or moving to a larger SKU, does.
Memory is harsher. A container over its memory request can still be killed when the node is tight, because the OS cannot steal allocated pages the way it can steal idle CPU (K8sUR3, p. 88). The scheduler still used the request to decide the node. The kill is a later owner.
Three No's, one Pending
| Event fragment | Owner | What actually failed |
|---|---|---|
untolerated taint {CriticalAddonsOnly: true} | Scheduler vs system pool | The Pod has no matching toleration. It is not allowed on the add-on nodes. |
Insufficient cpu or Insufficient memory | Scheduler vs allocatable | The request does not fit the leftover on any user node. |
didn't match Pod's node affinity | Scheduler vs labels | The Pod asked for a pool label the remaining nodes do not have. |
had volume node affinity conflict | CSI / PV (08-10) | Wrong lesson. The disk was zoned first. |
The last row is here so you do not treat every Pending as today's problem. kubectl describe pod names the owner in the event text. Read it before you resize a pool.
§III — Mechanism
What AKS already tainted
On a default AKS cluster the system pool nodes show:
Taints: CriticalAddonsOnly=true:NoScheduleNoSchedule means the scheduler will not place a new Pod that lacks the toleration. It does not evict a Pod that is already running. PreferNoSchedule is a preference. NoExecute evicts. Q10 uses NoSchedule. So does the system pool. Mixing the effects is how people "fix" a Pending by deleting a CoreDNS replica.
A user Deployment should not tolerate CriticalAddonsOnly. If it does, the next burst fills the system pool, the add-ons starve, and the cluster looks healthy in kubectl get nodes while DNS dies. The fence exists so that does not happen. Do not open it from the app side.
User pools are where the request is spent
Create a user pool with a SKU that can hold the request plus the DaemonSets you already run (Azure CNI, kube-proxy, any node-exporter). Azure CNI's maxPods is a second ceiling. A 30-pod cap on a node that still has CPU left will still refuse the 31st Pod. The event then looks like a taint problem and is not. kubectl describe node shows Allocatable pods and the count already bound.
The cluster autoscaler, if enabled on that pool, sees an unschedulable Pod and tries to add a node. It does not see a Pod that the scheduler already placed on the system pool because someone added the add-on toleration. It does not see a Pod whose nodeSelector names a pool that has autoscaler off. The Pending you wanted to autoscale must be unschedulable and eligible for a pool that can grow.
The four writes that make a user workload land
- A user pool exists, Ready, with allocatable leftover.
- The system pool stays tainted. App Pods do not tolerate it.
- The Pod's requests fit one user node after DaemonSets.
- Any
nodeSelector/ affinity you added matches a label that pool actually has (agentpool=user, or the SKU label you thought you set).
Miss 1: every node the scheduler can see is system. Miss 2: you scheduled onto CoreDNS's house. Miss 3: Insufficient cpu. Miss 4: affinity conflict on a label you invented in the Deployment and never put on the VMSS.
None of those four is a StorageClass. None of them is a RuntimeClass. The 08-10 and 08-13 Pendings had different owners. Today's describe text is the tell.
What the plan in ARM will not show
Terraform or Bicep that creates azurerm_kubernetes_cluster_node_pool will plan the VMSS. It will not plan the scheduler's next decision. A pool with enable_auto_scaling = true and a max-count of 3 can still leave a 4-CPU Pod Pending forever. The apply is clean. The world does not match the intent. That is the same class of hole the 08-15 TF lesson named for a hardcoded WI member, on a different cloud, for a different controller.
§IV — The module contract
If you wrap AKS pools in a module, the contract is the taint and the label, not the pool display name.
resource "azurerm_kubernetes_cluster_node_pool" "user" {
name = "user"
kubernetes_cluster_id = azurerm_kubernetes_cluster.this.id
vm_size = var.user_sku
min_count = var.user_min
max_count = var.user_max
enable_auto_scaling = true
node_labels = {
agentpool = "user"
}
}Do not put CriticalAddonsOnly on this pool. Do not remove it from the system pool in the same PR "to make the first deploy land." The first deploy lands by pointing the Deployment at agentpool=user or by leaving selectors empty so any untainted node is legal.
A second output should publish the taint map the day's Dev lesson will read. The fit-checker does not call Azure. It lists nodes and reads spec.taints and status.allocatable. If the module lies about the taint, the checker and the scheduler will disagree, and the checker is the one you will trust at 2 a.m.
§V — Failure modes the scheduler will show
The only Ready nodes are system. kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints. If every row has CriticalAddonsOnly, the user pool is NotReady, scaling from zero, or never created. Fix the pool. Do not tolerate the add-on taint on the app.
Requests copied from a laptop kind cluster. 100m CPU looked fine on six empty cores. On a 2-vCPU user node after CNI, it still fits. 2 CPU does not. Bootcamp Q4 is the drill: write the request you mean, then watch the scheduler, not the limit.
**NoExecute used as NoSchedule.** A taint you add during an incident with NoExecute evicts running Pods. The Pending you wanted becomes a delete. Q10's type is NoSchedule for a reason.
Autoscaler on the wrong pool. The unschedulable Pod selects agentpool=gpu. The autoscaler is on user. The GPU pool stays at min-count. The event stays Insufficient.
§VI — What this rung adds
08-10 taught the Pending whose owner is a volume. 08-13 taught the RuntimeClass whose owner is a pool that can honor a guest kernel. This rung teaches the Pending whose owner is kube-scheduler, on AKS, where the first no is a taint Microsoft already placed and the second no is a request you typed.
The Dev lesson writes a client that predicts the no without calling bind. The Cert lesson is the CKA 15% that names requests, taints, PriorityClass, and HPA as four different answers to the same refused Pod.
Related
- Archmagus-Stack/Polyglot-Dev/Python/2026-08-16-python-and-the-kubernetes-scheduler-surface-node-taints-tolerations-and-a-fit-checker-that-does-not-bind/lesson
- Archmagus-Stack/Cert-Prep/CNCF/2026-08-16-cka-workloads-and-scheduling-requests-limits-taints-priorityclass-and-the-pod-the-scheduler-refused/lesson
- Archmagus-Stack/01-Earth-DevOps/Synthesis-Lessons/2026-08-10-kubernetes-storage-on-eks-the-ebs-csi-driver-storageclass-volumebindingmode-and-where-a-pending-pod-actually-stops/lesson
- Archmagus-Stack/01-Earth-DevOps/Synthesis-Lessons/2026-08-13-kubernetes-runtime-isolation-on-gke-gke-sandbox-gvisor-runtimeclass-and-the-host-kernel-boundary/lesson