Kubernetes

Kubernetes is a cluster orchestration system for containerized workloads. Out of the box, it is configured with YAML files, making it a natural fit for Jsonnet.

Using Jsonnet With Kubernetes

Jsonnet's output JSON can be consumed by kubecfg which treats it as if it were YAML. This approach also works for other configuration formats that extend JSON, such as Javascript or HashiCorp's Configuration Language (HCL).

If we configure only a single Kubernetes object in each run of Jsonnet, we miss out on the ability to remove duplication not just within an object, but across sets of objects with commonalities. Fortunately, there are three ways to configure sets of Kubernetes objects: Use YAML stream output, multi-file output, or a single kubectl list object. This latter option is provided by Kubernetes without requiring any special support from Jsonnet. It allows us to group several Kubernetes objects into a single object. This is the most popular option because it does not require intermediate files, and other tools don't always reliably support YAML streams.

The Kubernetes community has been very active with Jsonnet, and the following resources may be helpful:

  • A simple example from the Jsonnet repo.
  • Kubecfg (whose name may sound familiar to current or ex-Googlers) is an unopiniated tool for evaluating Jsonnet and pushing the results to Kubernetes. It comes with a useful template library. See this blog post.
  • Ksonnet, by Heptio is a fork of kubecfg which uses a Jsonnet library that is automatically generated from Kubernetes API specifications. This forms a framework in which Kubernetes objects can be built using high-level composable descriptions.
  • Kapitan by Deepmind is another tool for driving Kubernetes with Jsonnet and textual templating.
  • Kpm is a similar deployment tool (now deprecated) by CoreOS that uses a mixture of Jsonnet and Jinja templating.
  • Box have blogged and spoken about how their internal Kubernetes-based infrastructure uses Jsonnet.
  • Databricks have written about how they manage infrastructure on Kubernetes using Jsonnet. They also published a style guide, which may be informative.

Conversion Tool

As a convenience, the following tool allows you to convert YAML Kubernetes examples into basic Jsonnet files. It simply converts the YAML to JSON and then runs the Jsonnet formatter on that JSON to make it look a bit cleaner. Sorry, the original file's comments are lost during the YAML parsing. This tool just helps you get started. You will want to further clean up the code to remove duplication.

output.jsonnet

Kubernetes Collections

In some places, the Kubernetes API objects use lists of named objects or (name, value) pairs rather than using objects to map names to values. This makes those objects hard to reference and extend in Jsonnet. For example, the containers part of the Pod spec looks like this:

local PodSpec = {
  containers: [
    {
      name: 'foo',
      env: [
        { name: 'var1', value: 'somevalue' },
        { name: 'var2', value: 'somevalue' },
      ],
    },
    {
      name: 'bar',
      env: [
        { name: 'var2', value: 'somevalue' },
        { name: 'var3', value: 'somevalue' },
      ],
    },
  ],
};

If we wanted to override this to modify the var3 environment variable of bar, we could use array indexes to reference the particular container and environment variable, but the resulting code is brittle and ugly:

PodSpec {
  containers: [
    super.containers[0],
    super.containers[1] {
      env: [
        super.env[0],
        super.env[1] { value: 'othervalue' },
      ],
    },
  ],
}

A better solution is to use an object to represent the mapping, converting to the Kubernetes representation post-hoc. We can now create a new Pod spec with the same override, specified much more easily and safely:

output.json

Such a pattern also makes referencing easier. Now, you can use the following code to access the same environment variable:

PodSpec.containersObj.bar.envObj.var3