2.1   Resources¶

Kubernetes is built on resources. These are abstract descriptions of persistent entities that should be configured and run in a particular Kubernetes cluster. Resources are defined in YAML files which are submitted to the Kubernetes cluster.

For example, Kubernetes uses resources to define what services should be running, how traffic should be routed, and how persistent storage should be provisioned.

Kubernetes users are able to provide custom resource types; these are used in the Alert Distribution System to describe the desired Kafka configuration.

In addition, resources are configured to reside within a particular namespace. These namespaces act as boundaries for authorization, as well as providing naming uniqueness for resources. Somewhat unconventionally, all of the alert stream resources are in one namespace, “alert-stream-broker”, which is explained in 5.1   Single Namespace.

2.2   Operators¶

Kubernetes operators are programs that run within the Kubernetes cluster, and take actions when resources are created, modified, or deleted. There are many default operators, and others that are installed to the cluster explicitly. The Alert Distribution System uses two such custom operators: Strimzi [8] and Strimzi Registry Operator [7].

2.3   Helm Charts¶

Helm [4] is a project which provides tools for templating the YAML resource definitions used in Kubernetes. The templates are called _Charts_, and provide a flexible way to represent common or repeated configuration.

For the Rubin IDF, all charts are defined in the lsst-sqre/charts repository.

Charts are reified into infrastructure using a set of values which populate the templates. For the Rubin IDF, the values to be used are defined in the lsst-sqre/phalanx repository.

These two repositories are part of the Phalanx system, described in SQR-056 [11] and at phalanx.lsst.io.

2.4   Argo¶

While Helm can be run as a tool on the command line, the Science Platform convention is to run it through Argo CD [1]. Argo CD is a platform for coordinating changes to a Kubernetes cluster, and it is able to run Helm directly. It is configured through a set of conventions in the lsst-sqre/phalanx repository.

2.5   Terraform¶

Terraform [9] is a tool for provisioning infrastructure through code. On the Rubin Science Platform, Terraform is used to provision non-Kubernetes resources, such as Google Cloud Storage buckets and account permissions for applications running inside Kubernetes to access Google Cloud Platform APIs.

Terraform source code resides in the lsst/idf_deploy repository.

3.1   Strimzi Operator¶

Strimzi [8] is a third-party software system for managing a Kafka cluster on Kubernetes. It is used in the Alert Distribution System as an abstraction layer around the details of configuring Kafka on individual Kubernetes Pods and Nodes.

Strimzi works through Custom Resource Definitions, or “CRDs”, which are installed once for the entire Kubernetes cluster across all namespaces. This installation is performed automatically by Argo CD when installing the Strimzi Helm chart, as configured in Phalanx as the ‘strimzi’ service.

The Strimzi Operator is a long-running application on Kubernetes which does all the work of actually starting and stopping Kubernetes Pods which run Kafka. It also sets up Kubernetes Secrets which are used for authentication to connect to the Kafka broker, and can install ingresses for providing external access to the Kafka broker.

The Alert Distribution System generally uses the default settings for the Strimzi Operator. There are only two settings which are explicitly enabled:

watchNamespaces:
  - "alert-stream-broker"
logLevel: "INFO"

watchNamespaces is a list of Kubernetes namespaces to be watched for Strimzi Custom Resources by the Strimzi Operator. In our case, this is configured to watch for any resources created in the alert-stream-broker namespace, since that namespace holds all the resources used to define the Alert Distribution System. All resources go in one namespace; this is explained further in 5.1   Single Namespace.

logLevel is set explicitly to INFO to enable logging by the Strimzi Operator itself. Note that this configures the Operator, not the Kafka broker or anything else. This can be set to DEBUG to help with debugging thorny internal issues.

3.2   Kakfa Cluster¶

The Kafka Cluster is at the heart of the Alert Distribution System, and is defined in terms of custom Strimzi resources. These resources are defined with Helm templates in the alert-stream-broker chart.

The chart has the following subresources:

1. A Kafka resource which defines the cluster’s size, listeners, and core configuration, including that of the ZooKeeper nodes, in kafka.yaml.
2. A Certificate resource used to provision a TLS certificate for the Kafka cluster’s external address, defined in certs.yaml.
3. A list of KafkaUsers used to create client identities that can access the Kafka Cluster, defined in users.yaml and superusers.yaml.
4. A VaultSecret used to store superuser credentials in Vault, which provides gated human access to the credential values through 1Password; see the Phalanx Documentation on VaultSecrets for more details. This is defined in vault_secret.yaml.

These will each now be explained in further detail.

# Addresses based on the state as of 2021-12-02; these were assigned by
# Google and now we're pinning them.
externalListener:
  bootstrap:
    ip: 35.188.169.31
    host: alert-stream-int.lsst.cloud
  brokers:
    - ip: 35.239.64.164
      host: alert-stream-int-broker-0.lsst.cloud
    - ip: 34.122.165.155
      host: alert-stream-int-broker-1.lsst.cloud
    - ip: 35.238.120.127
      host: alert-stream-int-broker-2.lsst.cloud

config:
  offsets.topic.replication.factor: 3
  transaction.state.log.replication.factor: 3
  transaction.state.log.min.isr: 2
  log.message.format.version: {{ .Values.kafka.logMessageFormatVersion }}
  inter.broker.protocol.version: {{ .Values.kafka.interBrokerProtocolVersion }}
  ssl.client.auth: required
  {{- range $key, $value := .Values.kafka.config }}
  {{ $key }}: {{ $value }}
  {{- end }}

storage:
  type: jbod
  volumes:
    # Note that storage is configured per replica. If there are 3 replicas,
    # and 2 volumes in this array, each replica will get 2
    # PersistentVolumeClaims for the configured size, for a total of 6
    # volumes.
  - id: 0
    type: persistent-claim
    size: {{ .Values.kafka.storage.size }}
    class: {{ .Values.kafka.storage.storageClassName }}
    deleteClaim: false

users:
  # A user for development purposes by the Rubin team, with access to all
  # topics in readonly mode.
  - username: "rubin-devel-idfint"
    readonlyTopics: ["*"]
    groups: ["rubin-devel-idfint"]

  # A user used by the Rubin team but with similar access to the community
  # broker users.
  - username: "rubin-communitybroker-idfint"
    readonlyTopics: ["alerts-simulated"]
    groups: ["rubin-communitybroker-idfint"]

  # The actual community broker users
  - username: "alerce-idfint"
    readonlyTopics: ["alerts-simulated"]
    groups: ["alerce-idfint"]

  - username: "ampel-idfint"
    readonlyTopics: ["alerts-simulated"]
    groups: ["ampel-idfint"]

  - username: "antares-idfint"
    readonlyTopics: ["alerts-simulated"]
    groups: ["antares-idfint"]

 # ... truncated

3.3   Strimzi Registry Operator¶

The Strimzi Registry Operator [7] is a Kubernetes Operator which defines a custom resource, StrimziSchemaRegistry, and which creates and manages a deployment of Confluent Schema Registry in response to instances of that resource. The Operator is an application written and maintained by Rubin’s SQuaRE team in the lsst-sqre/strimzi-registry-operator repository.

The Strimzi Registry Operator’s primary value to the Alert Distribution System is that it coordinates and synchornizes credentials used to access the Strimzi-managed Kafka cluster. The Operator is responsible for updating the deployed Schema Registry instance any time credentials are updated or changed, which can happen as a downstream consequence of changes to the Kafka cluster’s configuration.

The Operator has an associated Helm chart in the strimzi-registry-operator chart directory. This chart contains custom resource definitions, or CRDs. These CRDs must be installed cluster-wide at a consistent version, and so the first installation of this chart through Argo is particularly important.

The Operator chart has almost no configuration. The only options are to configure the Docker repository and tag which identifies a Docker container that runs the Strimzi Registry Operator application. This Docker container is automatically built in the lsst-sqre/strimzi-registry-operator repository’s continuous integration system and is published to the lsstsqre/strimzi-registry-operator repository on Docker Hub.

containers:
   - name: operator
     image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
     imagePullPolicy: Always
     env:
     - name: SSR_CLUSTER_NAME
       value: "{{ .Values.clusterName }}"
     - name: SSR_NAMESPACE
       value: "{{ .Values.watchNamespace }}"
     command: ["kopf"]
     args: ["run",  "--standalone",  "-m",  "strimziregistryoperator.handlers",  "--namespace",  "{{ .Values.watchNamespace }}",  "--verbose"]

3.4   Schema Registry¶

The Schema Registry runs an instance of Confluent Schema Registry [2] which is a service that provides access to Avro schema definition documents. These Avro schemas are used by clients consuming alert data. The schemas provide instructions to Avro libraries on how to parse binary serialized alert data into in-memory structures, such as dictionaries in Python.

Confluent Schema Registry uses a Kafka topic as its backing data store. The Registry itself is a lightweight HTTP API fronting this data in Kafka.

The Schema Registry is currently running at https://alert-schemas-int.lsst.cloud/. For example, to retrieve schema ID 1, you can issue an HTTP GET to https://alert-schemas-int.lsst.cloud/schemas/ids/1.

The Schema Registry for the Alert Distribution System is implemented with a Helm chart in the charts repository, alert-stream-schema-registry. This chart defines five resources:

1. A StrimziSchemaRegistry instance which is used by the Strimzi Registry Controller, creating a Deployment of the Schema Registry, in schema-registry-server.yaml.
2. A KafkaTopic used to store schema data inside the Kafka cluster, in schema-registry-topic.yaml.
3. A KafkaUser identity used by the Schema Registry instance to connect to the Kafka cluster, in schema-registry-user.yaml.
4. An Nginx Ingress which provides read-only access to the Schema Registry from over the public internet in ingress.yaml.
5. A Job which synchronizes the latest version of the alert packet schema into the Schema Registry, in sync-schema-job.yaml.

These will each be described in detail now.

apiVersion: roundtable.lsst.codes/v1beta1
kind: StrimziSchemaRegistry
metadata:
  name: {{ .Values.name }}
spec:
  strimzi-version: {{ .Values.strimziAPIVersion }}
  listener: internal

nginx.ingress.kubernetes.io/configuration-snippet: |
  # Forbid everything except GET since this should be a read-only ingress
  # to the schema registry.
  limit_except GET {
    deny all;
  }

annotations:
  kubernetes.io/ingress.class: "nginx"
  cert-manager.io/cluster-issuer: cert-issuer-letsencrypt-dns

spec:
  tls:
  - hosts: [{{ .Values.hostname | quote }}]
    secretName: "{{ .Values.name }}-tls"

3.5   Alert Stream Simulator¶

The Alert Stream Simulator is a subsystem which publishes static sample alerts into the Alert Distribution System’s Kafka broker. DMTN-149 [13] describes the design of the Alert Stream Simulator, but in the context of using it as a standalone tool for community brokers, and emphasizes use of Docker Compose. The Alert Distribution System deploys this software on Kubernetes instead. That deployment requires a few additional components which will be described in this section.

The simulator’s software (particularly the rubin-alert-sim program) is in the lsst-dm/alert-stream-simulator repository. All of the Kubernetes deployment configuration for the simulator resides in the alert-stream-simulator Helm chart.

3.6   Alert Database¶

The Alert Database is responsible for storing an archival copy of all data published to the alert stream. Once Rubin is producing real data, this will be kept to maintain a durable history of what was sent to community brokers. For now, the Database merely stores the test alert data that has been published using the Alert Stream Simulator component of the Alert Distribution System.

The Alert Database’s design is described in DMTN-183 [14]. The implementation follows that design document fairly closely, using Google Cloud Storage Buckets for the “object store” mentioned in the DMTN-183.

As explained in DMTN-183, there are two software components to the Alert Database.

An ingester consumes data from the published alert stream and copies it (along with any schemas referenced) into the backing object store. The ingester is implemented in the lsst-dm/alert_database_ingester repository.

A server presents an HTTP API for accessing the ingested data over the internet. The server is implemented in the lsst-dm/alert_database_server repository.

Both of these components are deployed in a single helm chart, alert-database. This chart has the following templates:

1. A Deployment and ServiceAccount for the ingester.
2. A Deployment and ServiceAccount for the server.
3. A KafkaUser for the ingester.
4. A Service used to provide internal access to the server’s Deployment.
5. An Ingress used to provide external access to the Service.

In addition, there are several components which are based on Google Cloud Platform rather than in Kubernetes, so they are configured with Terraform in the environments/deployments/science-platform/alertdb module:

6. Cloud Storage buckets for storing alert packets and schemas.
7. Cloud Service Accounts for reading and writing to the buckets.

These will now be described in detail (albeit in a somewhat scrambled order).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.ingester.serviceAccountName }}
  annotations:
    # The following annotation connects the Kubernetes ServiceAccount to a GCP
    # IAM Service Account, granting access to resources on GCP, via the
    # "Workload Identity" framework.
    #
    # https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
    iam.gke.io/gcp-service-account: "{{ .Values.ingester.gcp.serviceAccountName }}@{{ .Values.ingester.gcp.projectID }}.iam.gserviceaccount.com"

ingress:
  enabled: true
  host: "data-int.lsst.cloud"
  gafaelfawrAuthQuery: "scope=read:alertdb"

config:
  loglevel: "DEBUG"
  host: "data-int.lsst.cloud"
  databaseUrl: "postgresql://gafaelfawr@localhost/gafaelfawr"

  github:
    clientId: "0c4cc7eaffc0f89b9ace"

  # Allow access by GitHub team.
  groupMapping:
    "admin:provision":
      - "lsst-sqre-square"
    "exec:admin":
      - "lsst-sqre-square"
    "exec:notebook":
      - "lsst-ops-panda"
      - "lsst-sqre-square"
      - "lsst-sqre-friends"
    "exec:portal":
      - "lsst-ops-panda"
      - "lsst-sqre-square"
      - "lsst-sqre-friends"
    "read:alertdb":
      - "lsst-sqre-square"
      - "lsst-sqre-friends"
    "read:image":
      - "lsst-ops-panda"
      - "lsst-sqre-square"
      - "lsst-sqre-friends"
    "read:tap":
      - "lsst-ops-panda"
      - "lsst-sqre-square"
      - "lsst-sqre-friends"

4.1   Kafka Clients¶

The Kafka system provides the stream of alert packet data in a Kafka topic.

Each alert is delivered as a separate Kafka message, encoded in Confluent Wire Format [3]. That is, the Kafka message starts with a zero byte, then a 4-byte little-endian integer which represents that schema ID, and then the alert data in binary-encoded Avro format.

The Schema ID can be provided to the Schema Registry to retrieve an Avro schema document which can be used to deserialize the binary-encoded Avro data into an alert packet.

Messages are retained in the simulated alert topic for 7 days, as configured in 3.5.2.3   KafkaTopic for replay.

Clients connect to the alert stream by accessing the bootstrap URL of the Kafka cluster, alert-stream-int.lsst.cloud:9094. They must provide their username and password under SCRAM-SHA-512 authentication, and must use a consumer group ID which is prefixed with their username (see also: kafka-users).

The name of the alert stream topic is ‘alerts-simulated’.

Detailed walkthroughs of connecting to the Kafka endpoint of the alert stream are provided in Alert Stream Integration Endpoint Examples. The examples in following that link have very thoroughly commented example scripts which explain every detail needed to connect.

4.2   Schema Registry Clients¶

The schema registry provides read-only access to the Avro schemas used to encode alert packets. It uses the API described in its own documentation [6]; only the GET endpoints are accessible over the internet.

The registry runs at https://alert-schemas-int.lsst.cloud/. Users are expected to use a client library (probably as part of their Kafka client library) to connect. Detailed examples are available in the Alert Stream Integration Endpoint Examples.

{
   "schema": "<schema-document-as-a-string>",
}

{
  "type": "record",
   "namespace": "com.example",
   "name": "FullName",
   "fields": [
     { "name": "first", "type": "string" },
     { "name": "last", "type": "string" }
   ]
}

{
   "schema": "{\"type\":\"record\",\"namespace\":\"com.example\",\"name\":\"FullName\",\"fields\":[{\"name\":\"first\",\"type\":\"string\"},{\"name\":\"last\",\"type\":\"string\"}]}"
}

4.3   Alert Database Clients¶

The Alert Database provides access to all published alerts, as well as the schemas used to encode them, over an HTTP interface.

The alert messages are stored exactly as they were sent in Kafka - that is, in Confluent Wire Format. Schemas are indexed by their schema ID.

Clients must provide credentials which will be accepted by Gafaelfawr to access the alert database. This requires a “token” that will be included in requests.

To generate a token, navigate to the Gafaelfawr token generation page, https://data-int.lsst.cloud/auth/tokens/. Click on “Create Token” and choose the read:alertdb scope.

Store the token value securely somewhere. This token will be used in HTTP client requests in an Authorization header in the format Authorization: Bearer <token>.

With this header set, GET requests can be made to “https://data-int.lsst.cloud/alertdb/v1/schemas/{id}” to get a schema by ID, or “https://data-int.lsst.cloud/alertdb/v1/alerts/{id}” to get an alert by ID.

For example, with Python’s Requests library:

import requests
import os

token = os.environ["SECRET_TOKEN_VALUE"]

def get_schema(id):
    response = requests.get(
        f"https://data-int.lsst.cloud/alertdb/v1/schemas/{id}",
        headers={"Authorization": f"Bearer {token}"},
    )
    response.raise_for_status()
    return response.content


def get_raw_alert(alert_id):
    response = requests.get(
        f"https://data-int.lsst.cloud/alertdb/v1/alerts/{alert_id}",
        headers={"Authorization": f"Bearer {token}"},
    )
    response.raise_for_status()
    return response.content

5.1   Single Namespace¶

All Strimzi and Kubernetes resources reside in the same namespace, with the exception of the Strimzi Operator and Strimzi Registry Operator. This is done because it’s the simplest way to allow internal authentication to the Kafka cluster using Kubernetes Secrets.

The Strimzi Operator creates Kubernetes Secrets for each KafkaUser associated with a Kafka cluster that it manages. These Secrets hold all of the data required for a Kafka client to connect to the broker: TLS certificates, usernames, passwords - anything needed for a particular authentication mechanism.

The Secrets are created automatically, and will be updated or rotated automatically if the Kafka Cluster is changed. In addition, they can be securely passed in to application code using Kubernetes’ primitives for secret management, which gives us confidence that access is safe. This hands-off system greatly simplifies the coordination processes that would be required if credentials were manually managed without Strimzi.

However, they come with a downside, which is that Secrets cannot be accessed across namespace boundaries; they must be resident in a single namespace and can only be used from there. Strimzi chooses to create them in the same namespace as that of the Kafka resources.

Since we want to also use the Secrets for access from applications, this means that the applications need to all reside in the same namespace as the Kafka resource - effectively requiring that everything be in one namespace if it needs to access Kafka internally.

This isn’t particularly consequential in practice, although it has a few downsides:

1. All applications need to be bundled together into one Phalanx service, resulting in a cluttered view with many, many resources in Argo CD’s UI. This view can be hard to browse.
2. Applications may have access to more than is necessary, since Kubernetes Roles often grant access to resources within a namespace boundary. Bundling things into one namespace removes that protection. In practice, there aren’t any Kubernetes permissions granted to any of the applications, so this may be a moot point at this time, but things may change as the system evolves.

As an alternative, the Kubernetes Secrets could be reflected into multiple namespaces using a custom Operator. However, this would come at the cost of extra cluster-wide complexity. If multiple systems on the cluster would take advantage of such an operator, it might be worthwhile overall.

5.2   Using Strimzi¶

All Kafka broker configuration, topic configuration (with one exception - see 3.5.2.1   The load-data job), and user configuration is handled through Strimzi resources.

This means that there is yet another layer of configuration indirection. Instead, the system could have been built from “bare” Kubernetes Deployments, ConfigMaps, and so on.

But this would be very, very complex, and lifecycle management is particularly tricky. For example, when user credentials are rotated, the Kafka broker needs to be informed, and in some cases it needs to be restarted; this restart process needs to be done gradually, rolled out one-by-one across the cluster to avoid having complete downtime. Then, the credentials need to be bundled into Secrets to be passed to applications, and those applications likely would need to be restrated as well. Strimzi handles all of this complexity without any extra effort from Rubin developers.

Internal networking complexity gets even harder, as Kafka requires several internal communication channels for management of the cluster. Strimzi handles this as well - and it’s a particularly difficult thing to debug.

Overall, while Strimzi adds additional abstraction and configuration to learn, it seems to have been clearly successful in managing the overall complexity of the system. We probably would have had to replicate a great deal of its functionality to build the Alert Distribution System from lower-level components.