GitOps Policy-as-Code: Securing Kubernetes with Argo CD and Kyverno

What this stack does: Deploys Kyverno as a policy engine into a Kubernetes cluster managed by Argo CD, enforcing security rules at the admission controller level with full GitOps audit trails and progressive Audit → Enforce rollout.

• Blocks, mutates, or generates resources based on YAML-defined policies
• Policies live in Git alongside infrastructure — versioned, reviewable, testable
• Sync-wave ordering ensures Kyverno is running before policies are applied
• Audit mode lets teams observe violations before any blocking begins
• Supports multi-cluster policy standardization from a single Git root app

Purpose

Argo CD manages declarative cluster state from Git but provides no built-in policy enforcement. Kyverno fills that gap: a CNCF-graduated policy engine that intercepts Kubernetes resource requests at the admission layer and evaluates them against rules defined as standard Kubernetes YAML. Together, the two tools close the loop between deployment automation and security governance — every resource that reaches the cluster was both managed by Git and validated by policy.

Prerequisites

• Argo CD installed with a root app watching an infra-services/ directory
• Git repository to hold infrastructure manifests and Kyverno policy definitions
• Kubernetes cluster with sufficient RBAC permissions for webhook installation
• Helm 3 available in the Argo CD deployment environment

Steps

1. Add Kyverno application manifest to infra-services/

Create infra-services/kyverno.yaml as an Argo CD Application pointing to your Kyverno Helm chart wrapper. Include these critical annotations:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true

Set syncPolicy.syncOptions to include ServerSideApply=true — Kyverno’s CRDs are large enough to exceed the standard annotation size limit, and server-side apply is required to avoid reconciliation failures.

2. Create the Kyverno Helm chart wrapper

In kyverno/, create a Chart.yaml that wraps the upstream Kyverno chart (v3.7.1 as of this writing) as a dependency. This standardizes the installation across clusters without forking the upstream chart.

# kyverno/Chart.yaml
dependencies:
  - name: kyverno
    version: "3.7.1"
    repository: "https://kyverno.github.io/kyverno/"

3. Add the policy deployment application

Create infra-services/kyverno-policies.yaml — a second Argo CD Application pointing to the kyverno-policies/ chart with sync-wave "2". This ensures policies deploy only after Kyverno’s admission webhooks are active.

4. Configure policy behavior in kyverno-policies/values.yaml

policyType: ValidatingPolicy
podSecurityStandard: baseline
validationFailureAction: Audit
podSecuritySeverity: Medium

Start with Audit mode. Violations are logged but not blocked — giving teams time to identify existing non-compliant workloads before enforcement creates outages.

5. Add custom policies to kyverno-policies/templates/

Any YAML file placed in templates/ is automatically included when Helm renders the chart. Example: a restrict-external-ips policy that blocks externalIPs on Service resources (mitigating CVE-2020-8554):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-external-ips
spec:
  validationFailureAction: Audit
  rules:
    - name: check-ips
      match:
        resources:
          kinds: [Service]
      validate:
        message: "externalIPs are not allowed."
        pattern:
          spec:
            X(externalIPs): "null"

6. Switch to Enforce after audit period

Once the audit period confirms no legitimate workloads are violating the policy, update validationFailureAction: Enforce. Commit to Git, Argo CD syncs the change, Kyverno begins blocking non-compliant resources. Applications deploying violating resources will appear Degraded in Argo CD with Kyverno validation messages in sync errors.

7. Monitor policy reports

kubectl get policyreport -A
kubectl describe clusterpolicyreport

Reports surface resource name, policy rule, severity, and violation message — providing the audit trail necessary for compliance reviews.

The Trap

If Kyverno gets stuck showing OutOfSync in Argo CD after installation, the likely cause is missing compare-options annotations. Without ServerSideDiff=true,IncludeMutationWebhook=true, Argo CD cannot accurately track Kyverno’s mutation webhook behavior and will continuously flag the Application as out-of-sync. Add the annotations to the Kyverno Application manifest and force a sync — this resolves the issue cleanly.

An additional gotcha: if you plan to use Kyverno’s cleanup/deletion policies to remove stale resources on a schedule, you must explicitly grant RBAC permissions to the cleanup controller in values.yaml. This permission is not included by default.

Cited References

• GitOps Policy-as-Code: Securing Kubernetes with Argo CD and Kyverno — CNCF Blog
• Kyverno Documentation — kyverno.io
• Argo CD Documentation — argo-cd.readthedocs.io
• AWS DevOps Agent for Autonomous Incident Response

Photo by Markus Spiske on Pexels

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.