KEP-555 Reading Notes: The Design Principles and Context of Server Side Apply

Server Side Apply (SSA) has been GA since Kubernetes v1.22 (August 2021), and the Kubernetes blog even published a post encouraging everyone to adopt it.

In October 2022, Argo CD v2.5.0 announced SSA support. In November 2025, Helm 4.0.0 was released with SSA enabled by default for new releases. So what exactly is “SSA”?

Let’s dive into “KEP-555: Server Side Apply” and explore its design principles and context!

Client Side Apply

#

Before introducing SSA, let’s understand how kubectl apply originally worked — Client Side Apply (CSA). CSA performs a Three-Way Merge Patch on the client side, then produces a Strategic Merge Patch to send to the apiserver.

The three ways refer to:

1. Desired Configuration — the YAML being applied this time
2. Last Applied Configuration
3. Live Object — the current state of the object on the apiserver

But how does K8s know the content of the last apply? Remember that kubectl.kubernetes.io/last-applied-configuration annotation that shows up in .metadata.annotations whenever you kubectl edit? That’s the content from the last apply.

flowchart TD
    subgraph inputs["Three Inputs of Three-Way Merge Patch"]
        A["① Desired Configuration
The YAML being applied"]
        B["② Last Applied Configuration
kubectl.kubernetes.io/last-applied-configuration
annotation storing the last apply content"]
        C["③ Live Object
Current state on apiserver
(including server defaults)"]
    end

    subgraph diff["kubectl-side computation"]
        D["Three-Way Merge Patch
CreateThreeWayMergePatch()"]
        E["Strategic Merge Patch
JSON patch payload"]
    end

    A -- "① vs ②
What you changed" --> D
    B --> D
    C -- "② vs ③
What others changed" --> D
    D --> E
    E -- "PATCH request
Content-Type:
strategic-merge-patch+json" --> F["apiserver"]
    F -- "Write back ② annotation
to Live Object" --> B

Three inputs, compute a patch, then send it off — the flow seems intuitive, but the real problems are huge!

What’s Wrong with CSA?

#

1. Merge Logic Depends on the Client Version

#

The merge computation logic lives in kubectl — the patch is computed first and then sent to the server.

But merging relies on field types and schemas, and the server knows these best. Having the client replicate logic that only the server truly has is error-prone and divergence-prone, and any client in a different language has to reimplement it from scratch.

Furthermore, for maintainers, ensuring the client’s merge logic is compatible with ±1 server versions imposes an enormous maintenance burden.

2. The Organically Grown Strategic Merge Patch

#

Kubernetes currently provides three patch types:

• JSON Patch (RFC 6902)
• JSON Merge Patch (RFC 7386)
• Strategic Merge Patch

The default, grandly named “Strategic” Merge Patch, is not an RFC and has no version control.

Additionally, Custom Resources (CR) cannot use Strategic Merge Patch. The merge rules are embedded in Go struct tags (patchStrategy and patchMergeKey). Custom Resource Definitions (CRDs) are not Go structs — they are registered OpenAPI v3 schemas. They ultimately fall back to JSON Merge Patch.

3. Apply Results Are Unpredictable

#

Think one-on-one, single-person apply is foolproof? Even a single person repeatedly applying can get unexpected results; once multiple people co-manage resources, it’s a complete loss of control.

Alice and Bob co-manage a Deployment — Alice only cares about resources, Bob only cares about the image version. Under CSA, both must write the complete manifest; there’s no way to declare “just the fields I’m responsible for,” so conflicts arise.

The only difference between their manifests is the image — Alice has 1.24, Bob wants to upgrade to 1.25, everything else (including resources) is identical.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: web
 5spec:
 6  replicas: 3
 7  selector:
 8    matchLabels:
 9      app: web
10  template:
11    metadata:
12      labels:
13        app: web
14    spec:
15      containers:
16      - name: nginx
17        image: nginx:1.24
18        resources:
19          requests:
20            memory: "256Mi"
21          limits:
22            memory: "512Mi"

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: web
 5spec:
 6  replicas: 3
 7  selector:
 8    matchLabels:
 9      app: web
10  template:
11    metadata:
12      labels:
13        app: web
14    spec:
15      containers:
16      - name: nginx
17        image: nginx:1.25
18        resources:
19          requests:
20            memory: "256Mi"
21          limits:
22            memory: "512Mi"

Alice applies first, and kubectl.kubernetes.io/last-applied-configuration will contain Alice’s version (image still at 1.24).

Then Bob upgrades image to 1.25 and applies — the annotation now contains Bob’s content.

Here’s the problem: suppose Alice wants to tweak the resources she’s responsible for, so she re-applies her manifest (which still has image 1.24). kubectl will compare it against Bob’s content in the annotation via Three-Way Merge Patch — she only intended to change resources, but Bob’s image upgrade from 1.25 gets silently reverted to 1.24, with no warning whatsoever.

The KEP’s Motivation section even listed it like this:

User does POST, then changes something and applies: surprise!
User does an apply, then kubectl edit, then applies again: surprise!
User does GET, edits locally, then apply: surprise!
User tweaks some annotations, then applies: surprise!
Alice applies something, then Bob applies something: surprise!

Five scenarios, all “surprises”!

What Does This KEP Aim to Solve? (Goals / Non-Goals)

#

From the examples above, you should be able to feel the motivation behind this KEP. Its Goals include (the original is lengthy; this is a condensed summary):

• Handle changes from other users, systems, defaulting (including Admission Webhooks), and object schema evolution more robustly.
• Integrators only need to learn one API concept, rather than learning each API object individually.
• Users can more intuitively understand what the system will do with their config changes, and the system should tell users why conflicts occur whenever possible.
• Apply can be invoked from non-Go languages or non-kubectl clients (e.g., curl).

Some default behaviors are confusing — imagePullPolicy defaults to IfNotPresent, but if the image tag is latest, imagePullPolicy defaults to Always.

This kind of “I can’t figure out what the system will do” confusion is not in scope for this KEP (Non-Goals):

• Multi-object apply will not be changed: it remains client side for now
• Some sources of user confusion will not be addressed:
  • Changing the name field makes a new object rather than renaming an existing object
  • Changing fields that can’t really be changed (e.g., Service type).

This KEP only addresses problems caused by CSA’s Three-Way Merge Patch + Strategic Merge Patch, reducing inconsistencies between what users apply and the final result.

Moving to server-side processing sounds good, but how exactly can it solve the problems above?

How Does SSA Work? (Proposal & Implementation Details)

#

The KEP’s proposal section briefly lists several changes:

• Apply will be moved to the control plane.

This is straightforward to understand, though the original design document and the final implementation differ. I’ll briefly mention the original design in the Afterword, but the examples and schemas in the document are correct.

• Apply is invoked by sending a certain Content-Type with the verb PATCH.

A new Content-Type is used with PATCH — this became application/apply-patch+yaml.

• Instead of using a kubectl.kubernetes.io/last-applied-configuration annotation, the control plane will track a “manager” for every field.

No more relying on kubectl.kubernetes.io/last-applied-configuration — the control plane tracks a manager for every field. But what about other modifications like kubectl edit? Let’s look at the next statement:

• Apply is for users and/or ci/cd systems. We modify the POST, PUT (and non-apply PATCH) verbs so that when controllers or other systems make changes to an object, they are made “managers” of the fields they change.

Other verbs need to be modified so we can know which controller or system changed which fields. How to solve this? This led to FieldManager, with all the information stored in .metadata.managedFields.

FieldManager

#

Suppose we have a ConfigMap like this:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: test-cm
5  namespace: default
6  labels:
7    test-label: test
8data:
9  key: some value

Apply it with SSA, then list it:

# Server Side Apply ConfigMap
kubectl apply -f testcm.yaml --server-side

# Print the ConfigMap & Managed Fields
kubectl get cm test-cm -o yaml --show-managed-fields

By default, managedFields is not printed — remember to add --show-managed-fields:

 1apiVersion: v1
 2data:
 3  key: some value
 4kind: ConfigMap
 5metadata:
 6  creationTimestamp: "2026-06-25T15:19:09Z"
 7  labels:
 8    test-label: test
 9  managedFields:
10  - apiVersion: v1
11    fieldsType: FieldsV1
12    fieldsV1:
13      f:data:
14        f:key: {}
15      f:metadata:
16        f:labels:
17          f:test-label: {}
18    manager: kubectl
19    operation: Apply
20    time: "2026-06-25T15:19:09Z"
21  name: test-cm
22  namespace: default
23  resourceVersion: "1530170"
24  uid: 41a6b943-c67c-48e8-9ac2-07323f211cd9

Every field has a manager, indicating who manages it. When an object has multiple managers, it looks like this:

 1apiVersion: v1
 2kind: ConfigMap
 3metadata:
 4  name: test-cm
 5  namespace: default
 6  labels:
 7    test-label: test
 8  managedFields:
 9  - manager: kubectl
10    operation: Apply
11    time: '2019-03-30T15:00:00.000Z'
12    apiVersion: v1
13    fieldsType: FieldsV1
14    fieldsV1:
15      f:metadata:
16        f:labels:
17          f:test-label: {}
18  - manager: kube-controller-manager
19    operation: Update
20    apiVersion: v1
21    time: '2019-03-30T16:00:00.000Z'
22    fieldsType: FieldsV1
23    fieldsV1:
24      f:data:
25        f:key: {}
26data:
27  key: new value

In the above case, kube-controller-manager manages .data.key, and kubectl manages .metadata.labels.test-label.

For example, when HPA scales a Deployment, kube-controller-manager modifies the replicas count. At that point, kube-controller-manager takes over managing the replicas field, while users can still update the image version.

Status Wiping

#

However, when SSA was first released in Alpha, if a user’s applied content included .status, the FieldManager would assume the user intended to take over .status. When a controller tried to update the status, a conflict would occur. (issue#75564)

Should the FieldManager just ignore .status entirely? .status is a field like any other and should rightfully have a manager. The KEP states:

Additionally ignoring status as a whole is not enough, as it should be possible to own status (and other fields) in some occasions.

So how should the status issue be fixed? We can look at PR#99661’s implementation:

1// ResetFieldsStrategy is an optional interface that a storage object can
2// implement if it wishes to provide the fields reset by its strategies.
3type ResetFieldsStrategy interface {
4 GetResetFields() map[fieldpath.APIVersion]*fieldpath.Set
5}

A new ResetFieldsStrategy was added, allowing the FieldManager to reset fields as appropriate.

For example, with Deployments, updating the main content wipes .status; updating the status wipes .spec and .metadata. The implementation looks like this:

 1// GetResetFields returns the set of fields that get reset by the strategy
 2// and should not be modified by the user.
 3func (deploymentStrategy) GetResetFields() map[fieldpath.APIVersion]*fieldpath.Set {
 4 fields := map[fieldpath.APIVersion]*fieldpath.Set{
 5  "apps/v1": fieldpath.NewSet(
 6   fieldpath.MakePathOrDie("status"),
 7  ),
 8 }
 9
10 return fields
11}
12
13// GetResetFields returns the set of fields that get reset by the strategy
14// and should not be modified by the user.
15func (deploymentStatusStrategy) GetResetFields() map[fieldpath.APIVersion]*fieldpath.Set {
16 return map[fieldpath.APIVersion]*fieldpath.Set{
17  "apps/v1": fieldpath.NewSet(
18   fieldpath.MakePathOrDie("spec"),
19   fieldpath.MakePathOrDie("metadata", "labels"),
20  ),
21 }
22}

Back to the next change in the KEP proposal:

• Dry-run will be implemented on control plane verbs (POST, PUT, PATCH).
  • Admission webhooks will have their API appended accordingly.

Previously, client-side --dry-run would not go through Admission Webhooks, so you couldn’t get the real result. With server-side --dry-run, it runs through the full Admission Webhook modification and validation pipeline.

• The things our “Go IDL” describes are formalized: structured merge and diff
• Existing Go IDL files will be fixed (e.g., by fixing the directives)

Kubernetes’ Go IDL (Interface Definition Language) content is formalized, with a new repo created as structured-merge-diff. Existing Go IDL markers are fixed and supplemented.

Go IDL

#

Go IDL plays a crucial role here, governing the entire SSA merge logic. The data types it handles include List, Map, and Struct.

Lists have three processing modes:

• Atomic lists: The default mode — only one manager can manage it, and the entire list gets replaced.
• Sets: Each element appears at most once, and each element has its own manager.
• Associative lists: A list that can be used as a dictionary based on designated fields. Kubernetes commonly uses name as the key, and each element has its own manager.

Examples of each:

• Pod’s .spec.containers[].args is an Atomic list (you wouldn’t want to see two people’s args merged together).
• .metadata.finalizers is a Set — when a resource is deleted, each Finalizer has its own manager, and each controller removes its owned strings after completing its process.
• Pod’s .spec.containers[] is an Associative list — merging is handled based on each element’s name.

For Structs and Maps, there are two processing modes:

• Granular: The default mode — each field underneath has its own manager.
• Atomic: The element has only one manager. If any field underneath changes, the entire thing gets replaced.

Examples of each:

• .metadata.labels can have many Labels. For example, Argo CD can manage app.kubernetes.io/instance. Each Label can be independently managed by different controllers, making it Granular.
• Deployment’s .spec.selector contains matchLabels and matchExpressions — these two structs must work together and are tightly coupled, making it Atomic.

How Does SSA Solve the Earlier Problems?

#

Now that we’ve seen how SSA works, let’s map back to each of the earlier problems:

1. Merge logic depends on the client version

Merge logic is moved to the server, handled by the entirely new structured-merge-diff, no longer dependent on client-side implementation.

1. The organically grown Strategic Merge Patch

Go IDL clearly marks data types and merge logic, enabling CRDs to use the same system. Lists are no longer limited to just the Atomic option.

1. Apply results are unpredictable

FieldManager is introduced — every field has its own manager, and multi-person collaboration no longer results in silent overwrites.

Although object storage size increases by approximately 60%, for the sake of stability and eliminating ambiguity — and considering that most users are IT or SRE professionals — this trade-off is acceptable.

Summary

#

SSA not only resolves the uncertainty caused by CSA’s complex merge rules and the field ownership problem, but also allows Custom Resources to define merge rules instead of being limited to the default JSON Merge Patch.

Here’s a comparison table for reference:

Afterword

#

When reading the original design document, the initial approach was to use Three-Way Merge to infer the owner of each field. But looking at the discussion records, there were concerns it would become the next SMP — the approach was later replaced by the familiar managedFields.

Although this KEP is marked as implementation complete, the original design document included Union comparisons and discriminators. Yet looking back at KEP-555, there’s no trace of Union at all.

I’ll leave this as food for thought: Why was Union dropped? Is Union not important? Where did Union go?

SSA’s design cannot be fully covered in a single article. A dedicated Working Group Apply was created to handle this, and the shared documents contain many designs not mentioned in this article. I’ll introduce them when I get the chance.

References

#

• KEP-555: Server Side Apply
• Server Side Apply Is Great And You Should Be Using It | Kubernetes Blog
• Server-Side Apply GA | Kubernetes Blog
• Field Merge Semantics | SIG CLI
• structured-merge-diff | kubernetes-sigs
• Working Group Apply | Kubernetes Community
• kubernetes/kubernetes#75564 - SSA status conflict issue
• kubernetes/kubernetes#99661 - ResetFieldsStrategy PR