An Overview of the Kubernetes APIs
Kubernetes API changes are coming up again in 1.19 and 1.20 and I wanted to write a quick blog post to highlight what this means and show how at Mettle we have implemented a way to deal with the changes before they happen.
Back in October last year, there were a number of “Significant Changes to the Kubernetes API” which occurred in the Kubernetes 1.16 release.
The Kubernetes community dropped the ball communicating these changes and missed an opportunity to describe and discuss the problems that these changes will create. There were a few blog posts floating around but nothing front and center for the community to leverage to explain the deprecations in a concise manner.
The average user doesn’t pay attention to these blog posts, and there are a lot of old Helm charts out in the wild still, so I’m confident that these and incoming changes will create headaches and table flips when people start upgrading.
As an example, if you have an old API defined and running in a pre 1.16 cluster, and upgrade without fixing the API version first, APPS IN YOUR CLUSTER WILL BREAK. The good news is that new clusters won’t allow the old API versions, making errors easier to see and deal with.
A useful resource for learning about how Kubernetes deprecations work is the API deprecation documentation, highlighted in the deprecation post, but not widely shared.
Testing for deprecated APIs
With that mini-rant out of the way, there is a simple but effective way to test your existing configurations for API compatibility.
Conftest is a tool that helps write tests against structured configuration data, using the Rego language using Open Policy Agent (OPA). Conftest works with many file types including JSON, TOML, and HCL, which makes it a great choice for testing a variety of different configurations, but is especially useful for testing Kubernetes YAML configurations.
We bundle Conftest inside our kubernetes-toolkit container, the contents of that can be seen in my other blog post https://itnext.io/increasing-kubeval-linting-speeds-9607d1141c6a.
Then I wrote some Rego policies for current and future Kubernetes API deprecations which can be found at https://github.com/swade1987/deprek8ion
Here’s what a FAIL condition might look like according to what is defined in the rego policy file for an outdated API version …
The Rego policy is what actually defines the behavior that Conftest will display and as you can see, it found an issue with the Deployment object defined in the manifest. Below is the Rego policy that causes Conftest to spit out the failure message. The syntax is clean and easy to follow, so writing and adjusting policies are easy.
Once you know what is wrong with the configuration, you can fix up the existing deprecated API objects. Again, attempting to create objects using deprecated APIs in 1.16 will be rejected by Kubernetes, so you will only need to deal with converting existing objects in old clusters being upgraded. From the above error, we know the object type (Deployment) and the version (extensions/v1beta1). With this information, we can update the API version and re-run Conftest to confirm compliance.
How we use this at Mettle?
At Mettle, we are leveraging GitOps so we run these policy checks as part of our PR approval process for any repository which contains Kubernetes manifest files. An example of this can be seen below:
This has saved us from deploying deprecated API versions into our clusters during an upgrade process and also warns us of future deprecations coming down the pipe.
OPA and testing configurations using tools like Conftest and Rego policies is a great way to harden and help standardize configurations. Taken a step further, these configuration testing tools can be extended to test all sorts of other things.
Conftest looks especially promising because of the number of file types that it understands. There is a lot of potential here for doing things like unit testing Kubernetes configuration files and other things like Terraform configs.
My commitment to the community
My commitment is to keep https://github.com/swade1987/deprek8ion up to date with future API deprecations so that it can be used by the community to keep everyone ahead of the deprecation curve.