Autoscaling

The default setup now consists of only one Coffee Machine. This is, quite obviously, a serious bottleneck.

In this section, we’ll scale the Café by increasing the number of Coffee Machines based on the number of jobs currently queuing. We already exposed the number of open jobs as a metric of the Servitør. To use it for scaling the Coffee deployment, we need to provide this as a custom metric of the namespace via the Kubernetes API.

Warning

This section requires the implementation of the DNS resolution exercise for the Barista.

Kubernetes Custom Metric

As we will be using a Prometheus Metric, we’ll use the Prometheus Adapter to expose it to Kubernetes.

We’ll use the Prometheus Community Helm Chart, so all we need to do is provide a rule to turn the existing metric into a namespace metric. We’ll take the easy way by making use of the fact that there is only a single metric by the name.

The corresponding config file is

prometheus:
  url: "http://prometheus-server"
  port: "80"
  rules:
  - seriesQuery: 'job_queue_length'
    name: 'job_queue_length'
    metricsQuery: '<<.Series>>'

and we apply it with

helm install adapter -n obs prometheus-community/prometheus-adapter -f adapter-config.yaml

After a few minutes, the new metric will become available

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespace/*/job_queue_length" | jq

Horizontal Pod Autoscaler

We define a Horizontal Pod Autoscaler for the Coffee Machine as

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: schildcafe-coffee-scaler
  namespace: schildcafe
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: schildcafe-coffee
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      metric:
        name: job_queue_length
      describedObject:
        kind: Namespace
        name: schildcafe
      target:
        type: Value
        averageValue: "1"

This targets our earlier defined Deployment schildcafe-coffee and scales it between 1 and 10 based on the Prometheus metric.

To watch what’s happening run

kubectl get hpa -n schildcafe --watch