Admission Webhooks Have a Bypass Problem — Runtime Verification Fixes It

Admission webhooks like Kyverno and OPA Gatekeeper verify container images at the Kubernetes API layer — which means anything that never touches the API layer sails through unverified.

How It Works

Static pods managed directly by kubelet bypass admission entirely. Direct kubelet API access circumvents webhook checks the same way. A webhook outage forces a binary choice between locking down the cluster or silently letting everything through. Misconfigured namespace selectors can skip verification without anyone noticing, and webhook manipulation attacks can confuse the validation logic itself. None of these are exotic edge cases — they’re structural consequences of enforcing policy at a layer that not every container creation path actually passes through.

The fix described in this approach is to move enforcement one layer lower: to the container runtime itself, using the Node Resource Interface (NRI), a plugin API supported by CRI-O 1.28+ and containerd 1.7+. On every CreateContainer call, a Supply Chain NRI Plugin extracts the image reference and digest from runtime annotations, fetches supply chain attestations from the OCI registry, verifies them against a per-namespace policy, and synchronously allows or rejects the container. Because this hook sits inside the runtime rather than the API server, it applies regardless of how the container was scheduled — static pod, direct kubelet call, or normal deployment all pass through the same checkpoint.

Three attestation types get verified: SLSA Provenance (proving the image came from a trusted builder and build environment), VEX — Vulnerability Exploitability eXchange — (determining whether a known CVE is actually exploitable in this context, with an affected status blocking the container), and VSA, a Verification Summary Attestation that’s a trusted verifier’s pre-computed result, used to short-circuit the SLSA and VEX checks and cut down on per-node registry traffic.

What This Means for You

The honest tradeoff here is availability versus security when registries are unreachable. The fetch_failure_policy setting controls what happens: the default is “warn” — fail-open, meaning containers pass but the gap gets logged — while setting it to “deny” blocks new containers during any registry outage, trading uptime for certainty. Neither default is free; picking one is a real security-versus-availability decision every adopting team has to make deliberately, not inherit by accident.

There’s also a boundary worth being clear-eyed about: this approach assumes node integrity. An attacker with root access on a node can kill the plugin process, replace policy files on disk, or disable NRI outright — runtime verification protects against unauthorized bypass of the intended path, not against a fully compromised node. That’s the same class of limitation cloud-native GitOps supply chain approaches run into: enforcement is only as strong as the layer beneath it that nobody’s separately attacking.

Deployment is designed for gradual rollout rather than a flag-day cutover: it ships as a static binary, container image, or DEB/RPM package, supports DaemonSet, systemd, or pre-installed deployment modes, and defaults to verification = "disabled" requiring explicit opt-in. The suggested rollout is three phases — observe, then tighten enforcement per-namespace, then enforce fully — with policy separated from operational config (JSON namespace files versus TOML) and hot-reload via SIGHUP so policy changes don’t require pod restarts.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.