Jsonnet - Kubernetes

Jsonnet's output JSON can be consumed by kubecfg which treats it as if it were YAML. This approach also works for other configuration formats that extend JSON, such as Javascript or HashiCorp's Configuration Language (HCL).

If we configure only a single Kubernetes object in each run of Jsonnet, we miss out on the ability to remove duplication not just within an object, but across sets of objects with commonalities. Fortunately, there are three ways to configure sets of Kubernetes objects: Use YAML stream output, multi-file output, or a single kubectl list object. This latter option is provided by Kubernetes without requiring any special support from Jsonnet. It allows us to group several Kubernetes objects into a single object. This is the most popular option because it does not require intermediate files, and other tools don't always reliably support YAML streams.

The Kubernetes community has been very active with Jsonnet, and the following resources may be helpful:

A simple example from the Jsonnet repo.
Kubecfg (whose name may sound familiar to current or ex-Googlers) is an unopiniated tool for evaluating Jsonnet and pushing the results to Kubernetes. It comes with a useful template library. See this blog post.
Tanka, by Grafana Labs is the spiritual successor of Ksonnet. It extends jsonnet with native functions to import Kustomize manifests and Helm charts, making it an allround tool to work with Kubernetes.
jsonnet-libs/k8s generator produces over 30 Jsonnet Kubernetes libraries and counting, with most notably the k8s-libsonnet library as the succesor of ksonnet-lib.
Kapitan by Deepmind is another tool for driving Kubernetes with Jsonnet and textual templating.
Box have blogged and spoken about how their internal Kubernetes-based infrastructure uses Jsonnet.
Databricks have written about how they manage infrastructure on Kubernetes using Jsonnet. They also published a style guide, which may be informative.

As a convenience, the following tool allows you to convert YAML Kubernetes examples into basic Jsonnet files. It simply converts the YAML to JSON and then runs the Jsonnet formatter on that JSON to make it look a bit cleaner. Sorry, the original file's comments are lost during the YAML parsing. This tool just helps you get started. You will want to further clean up the code to remove duplication.

➡

output.jsonnet

        {
          kind: 'ReplicationController',
          apiVersion: 'v1',
          metadata: {
            name: 'spark-master-controller',
          },
          spec: {
            replicas: 1,
            selector: {
              component: 'spark-master',
            },
            template: {
              metadata: {
                labels: {
                  component: 'spark-master',
                },
              },
              spec: {
                containers: [
                  {
                    name: 'spark-master',
                    image: 'k8s.gcr.io/spark:1.5.2_v1',
                    command: [
                      '/start-master',
                    ],
                    ports: [
                      {
                        containerPort: 7077,
                      },
                      {
                        containerPort: 8080,
                      },
                    ],
                    resources: {
                      requests: {
                        cpu: '100m',
                      },
                    },
                  },
                ],
              },
            },
          },
        }

Kubernetes Collections

In some places, the Kubernetes API objects use lists of named objects or (name, value) pairs rather than using objects to map names to values. This makes those objects hard to reference and extend in Jsonnet. For example, the containers part of the Pod spec looks like this:

local PodSpec = {
  containers: [
    {
      name: 'foo',
      env: [
        { name: 'var1', value: 'somevalue' },
        { name: 'var2', value: 'somevalue' },
      ],
    },
    {
      name: 'bar',
      env: [
        { name: 'var2', value: 'somevalue' },
        { name: 'var3', value: 'somevalue' },
      ],
    },
  ],
};

If we wanted to override this to modify the var3 environment variable of bar, we could use array indexes to reference the particular container and environment variable, but the resulting code is brittle and ugly:

PodSpec {
  containers: [
    super.containers[0],
    super.containers[1] {
      env: [
        super.env[0],
        super.env[1] { value: 'othervalue' },
      ],
    },
  ],
}

A better solution is to use an object to represent the mapping, converting to the Kubernetes representation post-hoc. We can now create a new Pod spec with the same override, specified much more easily and safely:

➡

output.json

        {
          "containers": [
            {
              "env": [
                {
                  "name": "var2",
                  "value": "somevalue"
                },
                {
                  "name": "var3",
                  "value": "othervalue"
                }
              ],
              "name": "bar"
            },
            {
              "env": [
                {
                  "name": "var1",
                  "value": "somevalue"
                },
                {
                  "name": "var2",
                  "value": "somevalue"
                }
              ],
              "name": "foo"
            }
          ]
        }

Kubernetes

Using Jsonnet With Kubernetes

Conversion Tool

Kubernetes Collections