Restricting Flux permissions

Steve Wade

Published in

ITNEXT

4 min readJul 6, 2020

Problem statement

The problem statement for this piece of work was as follows:

As a platform engineer
I want to lock down flux permissions to “just enough”
So that we keep the cluster as secure as possible

What is Flux?

Flux is a tool that automatically ensures that the state of a cluster matches the config in git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means you don’t need a separate CD tool.

It monitors all relevant image repositories, detects new images, triggers deployments, and updates the desired running configuration based on that (and a configurable policy).

For more information see https://github.com/fluxcd/flux

The Problem

By default, the Flux helm chart sets the RBAC (Role-Based Access Control) permissions to a cluster role with the ability to do anything (see below)

rules:
  - apiGroups:
      - '*'
    resources:
      - '*'
    verbs:
      - '*'
  - nonResourceURLs:
      - '*'
    verbs:
      - '*'

Flux out the box has no understanding of what it's going to be reconciling, so this default position makes sense. Yet, from a security standpoint, this is far from ideal.

The requirements

The requirements are as follows:

A default set of RBAC permissions required for all fluxes

Out of the box, each flux instance will need a default set of permissions no matter what its reconciling, we need to define these.

2. A set of RBAC permissions to allow for automatic image promotion

If you are using automated image promotion using flux annotations on HelmRelease resources we need a set of permissions to make this possible.

3. A set of RBAC permissions for each instance of flux

Finally, we need to have the permissions required for the specific instance of flux we are running. This will be different depending upon what flux is reconciling. For example, a specific instance may only need to be reconciling SealedSecret resources into a set of namespaces and nothing else.

The implementation

For simplicity, I am going to talk about the changes required when using the upstream helm chart available here.

Take control of RBAC

The first thing we need to do is take complete control of the RBAC which the flux instance uses. This can be made possible by setting the following values.

rbac:
  create: false
  
serviceAccount:
  create: false
  name: flux-engineering

Constructing our ClusterRoleBinding & ServiceAccount

We now need a default ClusterRoleBinding and corresponding ServiceAccount (with the name specified in your helm chart values). An example of this can be seen below:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: flux-engineering
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flux-engineering
subjects:
  - kind: ServiceAccount
    name: flux-engineering
    namespace: flux-engineering----apiVersion: v1
kind: ServiceAccount
metadata:
  name: flux-engineering
  namespace: flux-engineering

Constructing our ClusterRole

When constructing the ClusterRole we need to start by adding in the default role-based permissions the Flux chart provides. These are the below

# Default RBAC permissions- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["list", "watch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list"]

The next section to add is the permissions required to complete automated image promotion, the resources listed are dependent upon what resources flux needs to look at, an example of these can be seen below:

# RBAC required for automatic image promotion- apiGroups: ["apps"]
  resources: ["deployments", "daemonset", "statefulset"]
  verbs: ["get", "list", "watch"]

- apiGroups: ["batch"]
  resources: ["cronjob"]
  verbs: ["get", "list", "watch"]

If you’re using imagePullSecrets you will need to provide flux with the ability to get and list secrets, see below:

# RBAC required to pull images from a private registry- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get", "list", "watch"]

Now we need to provide the permissions required to reconcile the specific resources in the repository, for example, this could just be HelmRelease resources. This can be seen below:

# Specific RBAC permissions for this flux instance- apiGroups: ["helm.fluxcd.io"]
  resources: ["helmreleases"]
  verbs: ["*"]

Additional ClusterRoleBinding for access to the Discovery API

Finally, we need to create a new ClusterRoleBinding which provides the flux service account with thesystem:discovery ClusterRole permissions. This is a default ClusterRole that allows read-only access to API discovery endpoints needed to discover and negotiate an API level. The ClusterRoleBinding can be seen below:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: flux-engineering-discovery-api
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:discovery
subjects:
  - kind: ServiceAccount
    name: flux-engineering
    namespace: flux-engineering

This ClusterRoleBinding is required for all flux instances no matter what the instance is reconciling.
Flux needs access to the Kubernetes Discovery API as it uses it for enumerating all the things it might need to garbage collect.

Shoutouts

I would finally like to shout out Michael Bridgen and Stefan Prodan from WeaveWorks. All three of us went on a journey to learn how to lock Flux down.