Flexible Resource Allocation: Kubernetes v1.36 Makes Job Resource Updates Possible in Beta

With the release of Kubernetes v1.36, a key feature that originally debuted as alpha in v1.35 has now graduated to beta: the ability to dynamically modify container resource requests and limits within the pod template of a suspended Job. This advancement empowers queue controllers and cluster administrators to fine-tune CPU, memory, GPU, and extended resource specifications on a Job while it remains suspended—before any pods are created or resumed. Let’s dive into the details through a Q&A format.

What exactly is the mutable pod resources feature for suspended Jobs?

In Kubernetes v1.36, the mutable pod resources feature allows you to change the resources.requests and resources.limits fields in the pod template of a suspended Job. Previously, these values were immutable after the Job was created. Now, while a Job is suspended (i.e., spec.suspend: true), you can adjust CPU, memory, GPU, and extended resource allocations without needing to delete and re‑create the entire Job. This change is enabled at the API server level by relaxing the immutability constraint specifically for suspended Jobs. It works with existing batch/v1 Job resources—no new API types or custom resources are required. The feature is in beta starting from v1.36, meaning it’s enabled by default for all clusters that meet the version requirement.

Why was this feature necessary?

Batch processing and machine learning workloads often have resource needs that aren’t known precisely at Job creation time. Optimal allocation depends on real‑time cluster capacity, queue priorities, and the availability of specialized hardware like GPUs or TPUs. Before this feature, if a queue controller (such as Kueue) determined that a suspended Job should run with different resources, the only option was to delete and recreate the Job. That approach threw away important metadata, status history, and any tracking labels. This feature eliminates that cumbersome workaround. It also enables graceful degradation: a CronJob can let a specific instance proceed slowly with reduced resources instead of failing entirely when the cluster is under heavy load. Overall, it makes scheduling more flexible and efficient without sacrificing Job identity or lifecycle history.

Can you walk through a real‑world example?

Consider a machine learning training Job that initially requests 4 GPUs, 8 CPUs, and 32Gi of memory. A queue controller sees that only 2 GPUs are available in the cluster. With the mutable pod resources feature, the controller can update the Job’s pod template while it’s still suspended, reducing the GPU request to 2, CPU to 4, and memory to 16Gi. Here’s a simplified view of the adjustment:

spec:
  suspend: true
  template:
    spec:
      containers:
      - name: trainer
        resources:
          requests:
            cpu: "4"          # was 8
            memory: "16Gi"    # was 32Gi
            example-hardware-vendor.com/gpu: "2"  # was 4
          limits:
            cpu: "4"
            memory: "16Gi"
            example-hardware-vendor.com/gpu: "2"

After updating, the controller sets spec.suspend to false, and new Pods are created with the adjusted resource specifications. The Job retains its original name, labels, and annotations—ensuring continuity for monitoring and auditing.

How does this feature work under the hood?

The magic happens in the Kubernetes API server. For a suspended Job (where spec.suspend is true), the validation webhook that normally enforces immutability of the pod template resource fields is bypassed. No new CRDs or API versions are introduced; the change is purely behavioral. The Job object itself remains unchanged in structure. When the Job is resumed (by setting suspend to false), the pod template with the updated resources is used to create new Pods. It’s important to note that the feature only applies while the Job is suspended. Once resumed, any further changes to pod resources in that Job are blocked—immutability is re‑enforced. This design ensures that active Jobs maintain a consistent resource footprint.

What are the main benefits and are there any limitations?

Benefits:

• No Job deletion: Adjust resources without losing metadata, labels, status, or history.
• Graceful scaling: Queue controllers can fit Jobs into available capacity without failing them.
• CronJob resilience: A single CronJob instance can run with reduced resources instead of failing entirely.
• Better cluster utilization: Resources can be dynamically assigned based on real‑time conditions.

Limitations:

• The feature is only available for suspended Jobs. Active Jobs cannot be modified.
• Only resource requests and limits can be changed; other pod template fields (like image or command) remain immutable.
• In beta, the feature is enabled by default, but if you’re on an older cluster, you may need to enable the MutablePodResources feature gate.

How does this compare to previous workarounds?

Before Kubernetes v1.35, if you needed to change resource allocations for a Job you had two unattractive options:

1. Delete and recreate the entire Job — losing all associated metadata, status history, and labels.
2. Manually adjust the cluster’s resource quotas or node availability, which is impractical for automated queues.

Queue controllers like Kueue had to cancel pending Jobs and create new ones with different resource specs, leading to wasted administrative overhead and potential loss of job identity. The new feature eliminates these cumbersome steps by allowing in‑place updates on the suspended Job object. This approach preserves the Job’s lifecycle, simplifies automation, and reduces operational friction—making Kubernetes more suitable for dynamic, resource‑sensitive workloads like AI/ML training and high‑throughput batch processing.

What’s the current status of this feature and how do I enable it?

Status: As of Kubernetes v1.36, the Mutable Pod Resources for Suspended Jobs feature has been promoted to beta. It is enabled by default in all clusters running v1.36 or later. If you’re on an earlier version (v1.35), the feature is alpha and you must explicitly enable the MutablePodResources feature gate. No manual configuration is needed for v1.36+.

To verify: Check your cluster version with kubectl version and then test by creating a suspended Job, updating its resources via kubectl patch job <name> --type=json ..., and resuming it. The API server should accept the patch while the Job is suspended.