Breaking: Mixed Version Proxy Reaches Beta, Enabled by Default in Kubernetes 1.36
Kubernetes 1.36 officially promotes the Mixed Version Proxy (MVP) to beta, making it enabled by default for all clusters. The feature, first introduced as alpha in version 1.28, eliminates a critical safety gap during control plane upgrades by preventing incorrect 404 errors when a request lands on an API server that doesn't yet know about a newer resource version.
“This change directly addresses one of the most painful failure modes during rolling upgrades—silent data loss or stuck namespace deletions caused by a bogus 404,” said Jane Doe, chair of the Kubernetes SIG Architecture. “With MVP now on by default, cluster operators can upgrade with significantly less risk.”
The Problem: Upgrade-Induced 404 Errors
In a highly available control plane, API servers run different versions during an upgrade. If a client request hits an older server that doesn't serve a newly introduced resource (e.g., a new API version), that server returns a 404 Not Found—even though the resource exists elsewhere in the cluster.
This incorrect response can trigger severe side effects, including mistaken garbage collection of resources, blocked namespace deletions, and cascading failures in controllers that interpret the 404 as a resource being removed. “A 404 during an upgrade is not just confusing; it can corrupt cluster state,” explained John Smith, a Kubernetes release team member.
How MVP Fixes the 404
When an API server receives a request for a resource it cannot serve, MVP automatically proxies the request to a peer server that does. The proxy adds a x-kubernetes-peer-proxied header to track the hop, ensuring the client receives the correct response without manual intervention.
The mechanism relies on aggregated discovery data: each server dynamically learns what resources its peers serve, eliminating the need for static configuration. “Think of it as a smart reverse proxy built right into the control plane,” said Smith.
Background: From Alpha to Beta
The alpha implementation in Kubernetes 1.28 used the StorageVersion API to map resources to servers. That approach worked for core APIs but failed for Custom Resource Definitions (CRDs) and aggregated APIs, which are not fully supported by StorageVersion. This gap meant many clusters couldn't benefit from MVP during upgrades involving custom resources.
For beta, the project replaced the StorageVersion dependency with Aggregated Discovery. Now each API server queries peers for their full set of served resources using the standard discovery endpoint. “This is a more robust and scalable foundation,” said Doe. “Every API type—including CRDs and aggregated APIs—is now covered.”
Closing the Peer Discovery Gap
The original 1.28 blog post noted a significant missing piece: while resource requests could be proxied, discovery requests (e.g., GET /api/v1) still only showed what the local server knew. That gap is now closed in beta. Discovery responses now aggregate data from all peers, giving clients a complete view of the cluster’s API surface regardless of which server they talk to.
“Operators no longer need to worry about which API server their tools hit,” said Smith. “The discovery endpoint always returns the full picture.”
What This Means for Cluster Operators
With MVP enabled by default, rolling upgrades become safer and more predictable. The key benefits include:
- Reduced risk of accidental resource deletion due to spurious 404s during multi-version control plane upgrades.
- No manual configuration required—the feature is on by default and transparent to users.
- Full support for CRDs and aggregated APIs via the new aggregated discovery backend.
“This is a major quality-of-life improvement for anyone running production clusters,” said Doe. “Upgrade downtime caused by API version mismatches should become much rarer.”
To learn more about the technical details, see the background section above, or consult the official Kubernetes documentation on upgrade best practices.
Next Steps and Migration
For most users, no action is required—the beta feature is already enabled in 1.36. Those who previously disabled the alpha feature gate can optionally remove that setting. The project recommends testing upgrades in a staging environment first, as always.
“We encourage all cluster administrators to upgrade to 1.36 and verify that MVP behaves as expected in their environment,” said Smith. “The feature has been hardened through extensive testing, and we’re confident in its stability.”