Configuring services with a CMS and a Kubernetes Operator

source: https://drwn.io/

Autonomy for scale

A principle that has helped us scale loveholidays is to create an environment with autonomous teams. In our recent blog, we discussed how autonomy already transforms our business — by empowering people to do the right thing, our business grows every day.

To ensure autonomy for all teams, our systems need to be configured by business operators and market teams rather than by engineers. This puts control in the hands of domain experts and allows engineers to focus on innovation, reducing the time they spend making and deploying configuration changes, which we consider toil.

For example, our yielding service, which applies dynamic pricing to our holiday packages, is governed by configurable pricing rules. It is more efficient for a revenue management expert to have the ability to apply the configuration, and see it immediately updated on the website, as opposed to asking engineers to change a configuration that potentially requires a deployment. A user interface (e.g. a web form) for managing the configuration is often a more desirable experience.

CMS for configuration

For managing configuration, we wanted something that is easy to set up and comes with authentication, authorisation, audit logs and an accessible UI. A while ago, we invested in a new content management system, Sanity, leveraging our principle of “technology is a means to an end”. Sanity has been a critical part of our platform evolution, and we quickly realised it could be used for a lot more than just customer-facing content. With rich APIs that query and detect changes, Sanity can also be put to good use for anything configuration related. So, for example, when someone updates a configuration, we can fetch the latest (or be told about the latest) changes through listeners.

configuration in Sanity

Applications need reliable access to configuration

The configuration is available to the application via Sanity’s APIs, though these might become unavailable for unexpected reasons. The system should be resilient despite that and operate with the last successfully retrieved configuration.

Also, by having configuration that is outside of the project itself, local development can be cumbersome, so we need to ensure that it is easy. That means not having to set up network or database access in order to run an application on a workstation in development mode. Therefore, it is desirable for the application to be able to see its configuration as a file on disk.

We wanted to keep things simple, avoid a single point of failure with a third party and not have vendor lock-in. Therefore, we needed to come up with a simple solution around Sanity configuration (or any service configuration) that we could run locally, listen to changes as they happen and work in mock CI environments. An abstraction was needed over the delivery of the configuration.

Enter configuration delivery with Kubernetes ConfigMaps

Most of our services are orchestrated by Kubernetes and, luckily, delivering configuration to containers is already solved — we can attach ConfigMap resources to the file system. ConfigMaps are key-value pairs that are always available when Kubernetes is available, thus making application startup (either due to scaling, deployment or node rebalancing) consistently possible. Also, should the contents change, the files will be updated without application restart (provided the whole map is mounted).

Applications see the configuration as regular files, which can be parsed and watched for changes. The delivery of the configuration is not a concern for the application. Reading a file is trivial in most programming languages, so applications focus on the business logic, thus keeping all parts of our system as simple as possible. This complies with “Invest in simplicity”, another of our engineering principles. By avoiding tight coupling, we leave the window open to experiment with different libraries, languages and approaches in the future. Local development becomes straightforward because accessing configuration just means local file access.

To make Sanity content available as ConfigMaps, which can then be mounted on a container, we created a Kubernetes Operator. This way, our developers only have to specify a custom kubernetes resource (CRD), like the one below, and the content will appear in a ConfigMap of the same name as the resource.

apiVersion: configuration.loveholidays.com/v1alpha1
kind: SanityContent
metadata:
  name: primer-configuration
  namespace: web
spec:
  # GROQ is Sanity's query language.
  groq: >
    *[_type == 'primerConfiguration' && !(_id in path("drafts.**"))]

After mounting the resulting ConfigMap on the container, we get our freshest content as files:

$ ls -1 /srv/configuration/primer/
primerConfigurationGB
primerConfigurationIE
sha256

dependencies

Summary

Our vision is to make all our systems, both present and future, configurable by domain experts. We invested in decoupling the delivery of such configuration, thus making it reusable, simple and resilient. By using existing solutions like Sanity and ConfigMaps, we did not reinvent the wheel. The solution is simple enough to encourage adoption by our engineers, empowering more business users to operate our systems. We got authentication, authorisation and audit logs for free with Sanity, and avoided tightly coupling our platform to one technology. This is in line with our engineering principles: “Invest in simplicity”, “Think big, deliver iteratively”, and “Focus on differentiation”.