Recap

As shown in the following diagram the Kubernetes Policy Controller (and therefore OPA) is called by the Kubernetes API Server in the following situations:

• For every authorized request to the Kubernetes API Server, i.e. almost every request
• Potentially for every create/update/delete on resources, depending on how the MutatingWebhook is configured

So obviously there’s a lot of potential to introduce latency when this is done wrong. The following sections show how we optimized the performance in our clusters.

Baseline cluster

In our baseline cluster a lot of the requests to the Kubernetes API Server originate from an operator which manages deployments of Helm Charts via CRDs. So there’s a steady stream of requests like creating Namespaces, updating Services/Deployments/Statefulsets and of course modifying the CRDs itself. I’m reproducing this use case via Vegeta and the following mixture of requests:

• 75% authorization only requests, e.g.: list Pods, get Services and watch Namespaces
• 25% admission requests, e.g.: update CRDs, delete Deployments and update Services (admission requests are also authorized before they are sent to the MutatingWebhook)

Open Policy Agent is run with GO_MAX_PROCS=2. Vegeta is used with the following settings:

• 60 s duration
• 20 concurrent connections
• 75 requests per second

The baseline latencies measured when calling OPA directly from Vegeta are:

• Mean: 7.24 s
• Quantile 50, 95, 99, max: 4.79 s, 24.01 s, 29.13 s, 31.54 s
• About 13% of the requests got a timeout

Only send requests to the MutatingWebhook which should actually be validated

Obviously OPA isn’t running very well. The first problem we identified is that it’s not really feasible to validate every object via the MutatingWebhook.

So we restricted our MutatingWebhook to validate only the objects which are actually mentioned in our OPA policies:

We don’t really want to validate deletions of objects, so now we’re only validating CREATE and UPDATE requests. All other listed resources are actually validated through our policies. A huge gain in performance was that we’re no longer validating resources like ConfigMaps. Usually a lot of configuration is stored via ConfigMaps and this comes at a huge performance penalty when Open Policy Agent parses all these resources. Especially, because OPA builds an Abstract Syntax Tree for every query.

So after changing the MutatingWebhookConfiguration we’re now getting less and smaller requests. The latencies with these optimizations are now:

• Mean: 3.22 s
• Quantile 50, 95, 99, max: 1.04 s, 11.29 s, 13.32 s, 16.19 s
• We’re still getting timeouts, but only for about 0.23% of the requests

Don’t send the old resource on Update AdmissionRequests

The next optimization addresses Kubernetes Policy Controller. It sends requests like the following to OPA:

The resources are basically sent twice to the MutatingWebhook on every update request. The old version in the oldObject field and the new version as object. Because our policies only apply on the new resource anyway, we removed the oldObject from the requests. This led to another performance improvement as we reduced the query size significantly:

• Mean: 790 ms
• Quantile 50, 95, 99, max: 13 ms, 3.21 s, 3.92 s, 4.78 s
• No timeouts

Go Garbage Collection

To find out why certain requests are slow, the tracing pprof endpoint was added to OPA and we took a look at the trace via Go Execution Tracer.

It’s noticeable that there are a lot of Garbage Collections which only free up ~2–5 MB of memory. Those were mostly triggered from inside the AST parser. Of course this costs a lot of CPU time. We also added some tracing regions. The most important code regions are validating and executing the query. The histograms provided some insights where the time is actually spent and lost.

When drilling into the slow runs of validate query it’s obvious that most time is spent waiting for scheduling:

The conclusion was that all this is probably caused by the very frequent GC runs. To reduce the amount of GC cycles we set GOGC=1000. The following is the trace when running OPA with the new GOGC setting:

The obvious difference is that we now have a lot less GC cycles and they free up ~30–40 MB of memory. I think it’s important to hit the sweet spot between too less GC cycles which leads to increased memory usage and too much GC cycles which slows down the application. If the GOGC value is set too high and the GC is therefore run less frequently, the GC duration also increases because there is more memory to free up. This may lead to latency spikes when the GC is eventually run. The time spent to validate and execute the queries are also improved:

Be careful, although GOGC=1000 seems like a good value for us, this depends highly on your workload/queries.

The latency improved to:

• Mean: 15 ms
• Quantile 50, 95, 99, max: 13 ms, 26 ms, 41 ms, 175 ms

Optimize Authorization query format

Although we optimized the GC, we still had a few requests which took far too long. We took a closer look at authorization queries, because they represent by far the majority of our requests to OPA:

This format was originally chosen to have a consistent query format with the admission requests. Obviously most of the fields like resourceAttributes or user and group are required to make useful authorization decision. But there is also a lot of overhead like the deep nested structure or the status and metadata fields. So we stripped the format to the bare minimum:

OPA should now be able to parse this a lot faster then the previous format.

Notable differences are:

• The GC cycles are less frequent than before because we consume less memory. Time between GC cycles increased from 250 ms to 400 ms.
• Goroutines aren’t waiting as long as before until they get scheduled. Peak runnable Goroutines dropped from 68 to 3 because most of our requests can be handled a lot faster.

If we take a look at the histograms we see a huge improvement:

The latencies improved to:

• Mean: 9 ms
• Quantile 50, 95, 99, max: 6 ms, 24 ms, 42 ms, 90 ms

Future optimization: Server-side Apply

Currently, kubectl apply implements the logic that resources are only applied if there is a difference between the current resources and the new resource version. Because we’re using client-go directly in our operator and haven’t implemented optimizations for this, a lot of resources are applied even when there are no changes. All these requests have to be validated by our MutatingWebhook. As soon as server-side apply is implemented in Kubernetes, we expect that a lot of these requests aren’t sent to the MutatingWebhook anymore and that this helps drive down the latencies even more.

Conclusion

Finally we got the performance we require to putting OPA-based authorization into production. The following table shows an overview over the optimization steps:

Depending on how you use Kubernetes, this optimizations may provide a huge performance improvement regarding Kubernetes API Server latencies.

Thanks to my team for bearing with me during a few days of performance debugging :). A big shout-out to all contributors of Go execution tracer, it’s an invaluable tool to investigate Go performance. Thanks to all contributors of Open Policy Agent and Kubernetes Policy Controller/Gatekeeper.

Optimizing Open Policy Agent-based Kubernetes Authorization via Go Execution Tracer was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

We are a team providing managed Kubernetes clusters to our company-internal customers. To provide a near upstream Kubernetes experience, we want to grant our customers cluster-admin-like access. But to ensure baseline security and stability, we don’t want to grant full cluster-admin privileges. For example:

• We want to allow full access to any namespace except `kube-system`, because our infrastructure (e.g. monitoring & logging) is deployed there.
• We want to enforce a PodSecurityPolicy which doesn’t allow running containers as `root` user or the direct mount of `hostPath` volumes.

Our first implementation was implemented via Kubernetes RBAC and a custom operator. The basic idea was to grant all necessary rights via RBAC RoleBindings. So we gave our customers the ClusterRole `admin` for every namespace except `kube-system` (via the operator). Every time we found that something wasn’t working as expected we added additional rights, either via a per-namespace Role or via a ClusterRole. This lead to a lot of individual rules for specific use-cases and wasn’t really maintainable in the long term. Especially as our user base continues to grow it’s not really feasible to adjust the Roles whenever somebody detects an edge cases which doesn’t work with our configuration.

So instead of configuring authorization based on a whitelist we switched to a blacklist-based model. What we actually wanted was to give our customers cluster-admin access and only restrict some specific rights. Therefore an implementation based on a blacklist via Open Policy Agent was a natural fit.

Whitelist vs. Blacklist-based Authorization

Most requirements regarding Authorization can be implemented by simply using the RBAC authorization module via Roles and RoleBindings, which are explained in Using RBAC Authorization. But RBAC is by design limited to whitelisting, i.e. for every requests it’s checked if one of the Roles and RoleBindings apply and in that case the request is approved. Requests are only denied if there is no match, there is no way to deny requests explicitly. At first this doesn’t sound like a big limitation, but some specific use cases require more flexibility. For example:

• A user should be able to create/update/delete pods in all namespaces except `kube-system`. The only way to implement this via RBAC is to assign the rights on a per-namespaces basis, e.g. by deploying a ClusterRole and a per-namespace RoleBinding. If the namespaces change over time you have to either deploy this RoleBindings manually or run an operator for this.
• A Kubernetes cluster is provided with pre-installed StorageClasses. A user should be able to create/update/delete custom StorageClasses, but he shouldn’t be able to modify the pre-installed ones. If this would be implemented via RBAC, the user must have the right to create StorageClasses and as soon as he creates a StorageClass additional rights must be assigned to update and delete this StorageClass. As above, this could be implemented via an operator.

When you have lot of this use cases, you’ll get a lot of custom logic implemented via operators. Sooner or later this doesn’t scale, because with a lof of operators and accompanying RBAC Roles it gets really hard to understand what rights a user actually has. We will show that both cases can be implemented easier via Open Policy Agent.

Webhook Authorization Module vs. ValidatingWebhook & MutatingWebhook

Some advanced use cases can also be implemented via Dynamic Admission Control, i.e. ValidatingWebhook or MutatingWebhook. There are also blog posts which dive into how Open Policy Agent can be used for this: Policy Enabled Kubernetes with Open Policy Agent and Kubernetes Compliance with Open Policy Agent. Dynamic Admission Control has the limitation that the webhooks are only called for create, update and delete events on Kubernetes resources. So it’s for example impossible to deny get requests. But they also have advantages compared to the Webhook authorization module because they can deny requests based on the content of a Kubernetes resource. These are informations the Webhook authorization module has no access to. For reference, the Webhook authorization module decides based on SubjectAccessReviews, whereas the ValidatingWebhook and MutatingWebhook decide based on AdmissionReviews. In our implementation we’ve integrated OPA via authorization module and via MutatingWebhook.

Architecture

This section shows on a conceptual level how Kubernetes is integrated with Open Policy Agent. Because the Open Policy Agent itself doesn’t implement the REST interface required by Kubernetes, the Kubernetes Policy Controller translates Kubernetes SubjectAccessReviews and AdmissionReviews into Open Policy Agent queries.

For every request the Kubernetes API Server receives, the following sequence is executed:

1. The request is authenticated.
2. Based on the user information extracted by the authentication the request is authorized:
   1. First the Webhook is called. In our case, the Webhook can either deny the request or forward it to RBAC. The Kubernetes Webhook can also allow requests, but that’s not implemented in Kubernetes Policy Controller.
   2. Second the RBAC module is executed. If RBAC doesn’t allow the request, the request is denied.
3. If the request leads to a change in the persistence, e.g. create/update/delete a resource, Admission Controllers are executed (MutatingWebhook is only one of them). Read more about Admission Controllers here.

So depending on what exactly we want to deny, we now can implement authorization or admission OPA policies. Further information how this scenario is configured can be found here open-policy-agent/kubernetes-policy-controller (Authorization scenario).

Examples

This section shows how this setup is used to implement the use cases described above.

Create/update/delete pods in every namespace except kube-system

The basic idea is to grant create/update/delete rights on pods cluster-wide via RBAC and then OPA policies are used to deny access to pods in kube-system. First of all, we grant the group `user` the rights to create/update/delete pods:

Now every user in the group `user` is allowed to create/update/delete pods cluster-wide. To restrict these rights via OPA the following policy is deployed:

Notes:

• We have to exclude `system:kube-controller-manager` and `system:kube-scheduler`, because both the Kubernetes controller manager as well as the scheduler have to be able to access pods.
• By simply removing `resource.spec.resourceAttributes.resource = “pods”` from the policy, we could restrict access to all namespaced resources in `kube-system`.
• We have to be careful to deny the correct verbs. A simple `delete` in RBAC allows `delete` and `deletecollection` (cf. Authorization Overview).
• Open Policy Agent makes it also very easy to write unit tests for all our policies. For more information, see How Do I Test Policies?.

Create/update/delete on specific StorageClasses

In this example, we want to grant the user create/update/delete rights to all StorageClasses except `ceph`. As in the first example, we have to grant the user access via RBAC:

Now we’re denying access to the StorageClass `ceph` via OPA. So we’re deploying the following policy:

An unit test for this policy can be implemented like this:

Conclusion

In summary, OPA allows far more flexible policies compared to the built-in RBAC authorization, especially if no additional operator is used. In my opinion it would be nice to have direct integration for OPA as Authorization Module and Admission Controller, but in the meantime Kubernetes Policy Controller bridges the gap between Kubernetes and OPA. Some inspirations what can be implemented:

• Deny access to specific CustomResourceDefinitions, e.g. `calico`
• Deny access to specific ClusterRoles, e.g. `cluster-admin`, `admin`, `edit`, `view`
• Allow port-forward only to some specific pods in `kube-system`
• Create a mapping which PodSecurityPolicies can be used in which namespaces
• Allow access to ValidatingWebhookConfigurations except some pre-installed ones

What do you think about Open Policy Agent as policy engine for Kubernetes? What are your use-cases and are they already covered by RBAC? If not, what would you like to implement via the Open Policy Agent?

If you’re planning to use policy enforcement via Open Policy Agent in production, I’ll recommend to also read Optimizing Open Policy Agent-based Kubernetes Authorization via Go Execution Tracer.

If there are any further questions, just write a comment or contact me via @sbueringer.

Thanks to Mario Constanti and Christian Schlotter for their help during the implementation and reviewing this Blogpost :). Thanks to all contributors of Open Policy Agent and Kubernetes Policy Controller especially to Nikhil Bhatia and Torin Sandall for their support for our implementation of the authorization module integration of Open Policy Agent.

Kubernetes Authorization via Open Policy Agent was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I was at the Open Source Summit this week in Edinburgh. This blog post summarizes my personal highlights:

Linux:

• An introduction to control groups, Michael Kerrisk [Slides]
• What’s new in control groups v2, Michael Kerrisk [Slides]
• Understanding user namespaces, Michael Kerrisk [Slides]
• Using seccomp to limit the kernel attack surface, Michael Kerrisk [Slides][Video]
• Improve Linux User-Space Core Libraries with Restartable Sequences, Mathieu Desnoyers
• MM 101: Introduction to Linux Memory Management, Christoph Lameter
• Fun with Dynamic Kernel Tracing Events, Steven Rostedt [Slides]
• Modern Strace, Dmitry Levin
• Super Fast Packet Filtering with eBPF and XDP, Helen Tabunshchyk [Slides]
• Kernel Analysis using eBPF, Daniel Thompson [Video]

Kubernetes:

• Data mobility for Kubernetes Persistent Volumes
• Thanos — High Availability and Long Term Storage of Prometheus Metrics, Bartek Plotka [Slides]
• What are my microservices doing?, Juraci Paixão Kröhling [Slides]
• Full Stack Observability with Elastic: Logs, Metrics and Traces, Carlos Pérez-Aradros [Slides]

Misc:

• Hashicorp Terraform: Deep Dive with no Fear, Victor Turbinsky