K8s Troubleshooting — Pending Pods

K8s Troubleshooting handbook



Note, full K8s troubleshooting mind map is available at: “K8s Troubleshooting MindMap

What is “Pending Pods” Error?

In K8s, when a pod is in a “Pending” state, it means that the pod hasn’t been successfully scheduled and started on a worker node. There can be various reasons why a pod remains in this state, and it’s often crucial to diagnose the root cause to ensure proper cluster operation.

For example, the error message may look like:

Name:         my-pending-pod
Namespace: default
Status: Pending
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 30s (x8 over 3m) default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) had taint {key=value:NoSchedule}, that the pod didn't tolerate.

In this example, the error message indicates that the pod couldn’t be scheduled because:

  1. One node doesn’t have enough CPU.
  2. Two nodes have a taint that the pod doesn’t tolerate.

To solve such issues, you may need to adjust pod specifications, node taints/tolerations, ensure proper resource availability, or check other configurations that might be preventing the pod from being scheduled.

Possible Root Causes

Some common reasons for “Pending” pods include:

  • Resource Constraints: If there’s insufficient CPU, memory, or another resource on the available nodes, the pod won’t be scheduled.
  • Taints and Tolerations: Nodes might have taints, and unless a pod has a matching toleration, it won’t be scheduled on that node.
  • Storage Issues: A pod might be waiting for a Persistent Volume to be attached.
  • Node Selector & Affinity: If the pod has specific node selectors or affinity rules, and no nodes match those rules, it will stay in the pending state.
  • Network Issues: If there’s a configuration issue or other problems with the network, it might prevent the pod from starting.