[Merged by Bors] - Docs new landing page

CHANGELOG.md

@@ -14,10 +14,12 @@ All notable changes to this project will be documented in this file.
 - operator-rs: 0.30.1 -> 0.33.0 ([#545]).
 - Bumped stackable versions to "23.4.0-rc1" ([#545]).
 - Bumped kafka stackable versions to "23.4.0-rc2" ([#547]).
+- Updated landing page and restructured usage guide ([#573]).
 
 [#545]: https://github.com/stackabletech/kafka-operator/pull/545
 [#547]: https://github.com/stackabletech/kafka-operator/pull/547
 [#557]: https://github.com/stackabletech/kafka-operator/pull/557
+[#573]: https://github.com/stackabletech/kafka-operator/pull/573
 
 ## [23.1.0] - 2023-01-23
 

docs/modules/kafka/pages/getting_started/first_steps.adoc

@@ -30,7 +30,7 @@ include::example$getting_started/getting_started.sh[tag=install-zookeeper]
 
 Create a file `kafka-znode.yaml` with the following content:
 
-[source,bash]
+[source,yaml]
 ----
 include::example$getting_started/kafka-znode.yaml[]
 ----

docs/modules/kafka/pages/index.adoc

@@ -1,18 +1,50 @@
 = Stackable Operator for Apache Kafka
+:description: The Stackable Operator for Apache Superset is a Kubernetes operator that can manage Apache Kafka clusters. Learn about its features, resources, dependencies and demos, and see the list of supported Kafka versions.
+:keywords: Stackable Operator, Apache Kafka, Kubernetes, operator, SQL, engineer, broker, big data, CRD, StatefulSet, ConfigMap, Service, Druid, ZooKeeper, NiFi, S3, demo, version
 
-This is an operator for Kubernetes that can manage https://kafka.apache.org/[Apache Kafka] clusters.
+The Stackable Operator for Apache Kafka is an operator that can deploy and manage https://kafka.apache.org/[Apache Kafka] clusters on Kubernetes. 
+// what is Kafka?
+Apache Kafka is a distributed streaming platform designed to handle large volumes of data in real-time. It is commonly used for real-time data processing, data ingestion, event streaming, and messaging between applications.
+
+== Getting started
+
+Follow the xref:kafka:getting_started/index.adoc[] which will guide you through installing The Stackable Kafka and ZooKeeper Operators, setting up ZooKeeper and Kafka and testing your Kafka using kcat.
+
+== Resources
+
+The _KafkaCluster_ custom resource contains your Kafka cluster configuration. It defines a single `broker` xref:concepts:roles-and-role-groups.adoc[role].
+
+image::kafka_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the operator.]
+
+For every xref:concepts:roles-and-role-groups.adoc#_role_groups[role group] in the `broker` role the Operator creates a StatefulSet. Multiple Services are created - one at role level, one per role group as well as one for every individual Pod - to allow access to the entire Kafka cluster, parts of it or just individual brokers.
+
+For every StatefulSet (role group) a ConfigMap is deployed containing a `log4j.properties` file for xref:usage-guide/logging.adoc[logging] configuration and a `server.properties` file containing the whole Kafka configuration which is derived from the KafkaCluster resource.
+
+The Operator creates a xref:concepts:service_discovery.adoc[] for the whole KafkaCluster which references the Service for the whole cluster. Other operators use this ConfigMap to connect to a Kafka cluster simply by name and it can also be used by custom third party applications to find the connection endpoint.
+
+== Dependencies
+
+Kafka requires xref:zookeeper:index.adoc[Apache ZooKeeper] for coordination purposes (it will not be needed in the future as it will be replaced with a https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum[built-in solution]).
+
+== Connections to other products
+
+Since Kafka often takes on a bridging role, many other products connect to it. In the <<demos, demos>> below you will find example data pipelines that use xref:nifi:index.adoc[Apache NiFi with the Stackable Operator] to write to Kafka and xref:nifi:index.adoc[Apache Druid with the Stackable Operator] to read from Kafka. But you can also connect using xref:spark-k8s:index.adoc[Apache Spark] or with a custom Job written in various languages.
+
+== [[demos]]Demos
+
+xref:stackablectl::index.adoc[] supports installing xref:stackablectl::demos/index.adoc[] with a single command. The demos are complete data piplines which showcase multiple components of the Stackable platform working together and which you can try out interactively. Both demos below inject data into Kafka using NiFi and read from the Kafka topics using Druid.
+
+=== Waterlevel Demo
+
+The xref:stackablectl::demos/nifi-kafka-druid-water-level-data.adoc[] demo uses data from https://www.pegelonline.wsv.de/webservice/ueberblick[PEGELONLINE] to visualize water levels in rivers and coastal regions of Germany from historic and real time data. 
+
+=== Earthquake Demo
+
+The xref:stackablectl::demos/nifi-kafka-druid-earthquake-data.adoc[] demo ingests https://earthquake.usgs.gov/[earthquake data] into a similar pipeline as is used in the waterlevel demo.
 
-WARNING: This operator only works with images from the https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable%2Fkafka[Stackable] repository
 
 == Supported Versions
 
 The Stackable Operator for Apache Kafka currently supports the following versions of Kafka:
 
 include::partial$supported-versions.adoc[]
-
-== Getting the Docker image
-
-[source]
-----
-docker pull docker.stackable.tech/stackable/kafka:<version>
-----

docs/modules/kafka/pages/usage-guide/configuration-environment-overrides.adoc

@@ -0,0 +1,63 @@
+= Configuration & Environment Overrides
+
+The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
+
+IMPORTANT: Overriding certain properties which are set by operator (such as the ports) can interfere with the operator and can lead to problems.
+
+== Configuration Properties
+
+For a role or role group, at the same level of `config`, you can specify: `configOverrides` for the `server.properties`. For example, if you want to set the `auto.create.topics.enable` to disable automatic topic creation, it can be configured in the `KafkaCluster` resource like so:
+
+[source,yaml]
+----
+brokers:
+  roleGroups:
+    default:
+      configOverrides:
+        server.properties:
+          auto.create.topics.enable: "false"
+      replicas: 1
+----
+
+Just as for the `config`, it is possible to specify this at role level as well:
+
+[source,yaml]
+----
+brokers:
+  configOverrides:
+    server.properties:
+      auto.create.topics.enable: "false"
+  roleGroups:
+    default:
+      replicas: 1
+----
+
+All override property values must be strings.
+
+For a full list of configuration options we refer to the Apache Kafka https://kafka.apache.org/documentation/#configuration[Configuration Reference].
+
+== Environment Variables
+
+In a similar fashion, environment variables can be (over)written. For example per role group:
+
+[source,yaml]
+----
+servers:
+  roleGroups:
+    default:
+      envOverrides:
+        MY_ENV_VAR: "MY_VALUE"
+      replicas: 1
+----
+
+or per role:
+
+[source,yaml]
+----
+servers:
+  envOverrides:
+    MY_ENV_VAR: "MY_VALUE"
+  roleGroups:
+    default:
+      replicas: 1
+----

docs/modules/kafka/pages/discovery.adoc → docs/modules/kafka/pages/usage-guide/discovery.adoc

@@ -3,8 +3,9 @@
 :brokerPort: 9092
 
 = Discovery
+:page-aliases: discovery.adoc
 
-The Stackable Operator for Apache Kafka publishes a discovery https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#configmap-v1-core[`ConfigMap`], which exposes a client configuration bundle that allows access to the Apache Kafka cluster.
+The Stackable Operator for Apache Kafka publishes a xref:concepts:service_discovery.adoc[service discovery ConfigMap] which exposes a client configuration bundle that allows access to the Apache Kafka cluster.
 
 The bundle includes a thrift connection string to access the Kafka broker service. This string may be used by other operators or tools to configure their products with access to Kafka. This is limited to internal cluster access.
 
@@ -22,14 +23,14 @@ metadata:
 spec:
   [...]
 ----
-<1> The name of the Kafka cluster, which is also the name of the created discovery `ConfigMap`.
-<2> The namespace of the discovery `ConfigMap`.
+<1> The name of the Kafka cluster, which is also the name of the created discovery ConfigMap.
+<2> The namespace of the discovery ConfigMap.
 
-The resulting discovery `ConfigMap` is `{namespace}/{clusterName}`.
+The resulting discovery ConfigMap is `{namespace}/{clusterName}`.
 
 == Contents
 
-The `{namespace}/{clusterName}` discovery `ConfigMap` contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:
+The `{namespace}/{clusterName}` discovery ConfigMap contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:
 
 `KAFKA`::
 ====

docs/modules/kafka/pages/usage-guide/index.adoc

@@ -0,0 +1,2 @@
+= Usage guide
+:page-aliases: usage.adoc

docs/modules/kafka/pages/usage-guide/logging.adoc

@@ -0,0 +1,18 @@
+= Log aggregation
+
+The logs can be forwarded to a Vector log aggregator by providing a discovery
+ConfigMap for the aggregator and by enabling the log agent:
+
+[source,yaml]
+----
+spec:
+  clusterConfig:
+    vectorAggregatorConfigMapName: vector-aggregator-discovery
+  brokers:
+    config:
+      logging:
+        enableVectorAgent: true
+----
+
+Further information on how to configure logging, can be found in
+xref:home:concepts:logging.adoc[].

docs/modules/kafka/pages/usage-guide/monitoring.adoc

@@ -0,0 +1,4 @@
+= Monitoring
+
+The managed Kafka instances are automatically configured to export Prometheus metrics. See
+xref:home:operators:monitoring.adoc[] for more details.

docs/modules/kafka/pages/usage-guide/pod-placement.adoc

@@ -0,0 +1,22 @@
+= Pod Placement
+
+You can configure Pod placement for Kafka brokers as described in xref:concepts:pod_placement.adoc[].
+
+By default, the operator configures the following Pod placement constraints:
+
+[source,yaml]
+----
+affinity:
+  podAntiAffinity:
+    preferredDuringSchedulingIgnoredDuringExecution:
+    - podAffinityTerm:
+        labelSelector:
+          matchLabels:
+            app.kubernetes.io/component: broker
+            app.kubernetes.io/instance: cluster-name
+            app.kubernetes.io/name: kafka
+        topologyKey: kubernetes.io/hostname
+      weight: 70
+----
+
+In the example above `cluster-name` is the name of the Kafka custom resource that owns this Pod.

docs/modules/kafka/pages/usage-guide/security.adoc

@@ -0,0 +1,161 @@
+= Security
+
+== Encryption
+
+The internal and client communication can be encrypted TLS. This requires the xref:secret-operator::index.adoc[Secret Operator] to be present in order to provide certificates. The utilized certificates can be changed in a top-level config.
+
+[source,yaml]
+----
+---
+apiVersion: kafka.stackable.tech/v1alpha1
+kind: KafkaCluster
+metadata:
+  name: simple-kafka
+spec:
+  image:
+    productVersion: 3.3.1
+    stackableVersion: "23.4.0-rc2"
+  clusterConfig:
+    zookeeperConfigMapName: simple-kafka-znode
+    tls:
+      serverSecretClass: tls # <1>
+      internalSecretClass: kafka-internal-tls # <2>
+  brokers:
+    roleGroups:
+      default:
+        replicas: 3
+----
+<1> The `spec.clusterConfig.tls.serverSecretClass` refers to the client-to-server encryption. Defaults to the `tls` secret. Can be deactivated by setting `serverSecretClass` to `null`.
+<2> The `spec.clusterConfig.tls.internalSecretClass` refers to the broker-to-broker internal encryption. This must be explicitly set or defaults to `tls`. May be disabled by setting `internalSecretClass` to `null`.
+
+The `tls` secret is deployed from the xref:secret-operator::index.adoc[Secret Operator] and looks like this:
+
+[source,yaml]
+----
+---
+apiVersion: secrets.stackable.tech/v1alpha1
+kind: SecretClass
+metadata:
+  name: tls
+spec:
+  backend:
+    autoTls:
+      ca:
+        secret:
+          name: secret-provisioner-tls-ca
+          namespace: default
+        autoGenerate: true
+----
+
+You can create your own secrets and reference them e.g. in the `spec.clusterConfig.tls.serverSecretClass` or `spec.clusterConfig.tls.internalSecretClass` to use different certificates.
+
+== Authentication
+
+The internal or broker-to-broker communication is authenticated via TLS. In order to enforce TLS authentication for client-to-server communication, you can set an `AuthenticationClass` reference in the custom resource provided by the xref:commons-operator::index.adoc[Commons Operator].
+
+[source,yaml]
+----
+---
+apiVersion: authentication.stackable.tech/v1alpha1
+kind: AuthenticationClass
+metadata:
+  name: kafka-client-tls # <2>
+spec:
+  provider:
+    tls:
+      clientCertSecretClass: kafka-client-auth-secret # <3>
+---
+apiVersion: secrets.stackable.tech/v1alpha1
+kind: SecretClass
+metadata:
+  name: kafka-client-auth-secret # <4>
+spec:
+  backend:
+    autoTls:
+      ca:
+        secret:
+          name: secret-provisioner-tls-kafka-client-ca
+          namespace: default
+        autoGenerate: true
+---
+apiVersion: kafka.stackable.tech/v1alpha1
+kind: KafkaCluster
+metadata:
+  name: simple-kafka
+spec:
+  image:
+    productVersion: 3.3.1
+    stackableVersion: "23.4.0-rc2"
+  clusterConfig:
+    authentication:
+      - authenticationClass: kafka-client-tls # <1>
+    zookeeperConfigMapName: simple-kafka-znode
+  brokers:
+    roleGroups:
+      default:
+        replicas: 3
+----
+<1> The `clusterConfig.authentication.authenticationClass` can be set to use TLS for authentication. This is optional.
+<2> The referenced `AuthenticationClass` that references a `SecretClass` to provide certificates.
+<3> The reference to a `SecretClass`.
+<4> The `SecretClass` that is referenced by the `AuthenticationClass` in order to provide certificates.
+
+
+== [[authorization]]Authorization
+
+If you wish to include integration with xref:opa::index.adoc[Open Policy Agent] and already have an OPA cluster, then you can include an `opa` field pointing to the OPA cluster discovery `ConfigMap` and the required package. The package is optional and will default to the `metadata.name` field:
+
+[source,yaml]
+----
+---
+apiVersion: kafka.stackable.tech/v1alpha1
+kind: KafkaCluster
+metadata:
+  name: simple-kafka
+spec:
+  image:
+    productVersion: 3.3.1
+    stackableVersion: "23.4.0-rc2"
+  clusterConfig:
+    authorization:
+      opa:
+        configMapName: simple-opa
+        package: kafka
+    zookeeperConfigMapName: simple-kafka-znode
+  brokers:
+    roleGroups:
+      default:
+        replicas: 1
+----
+
+You can change some opa cache properties by overriding:
+
+[source,yaml]
+----
+---
+apiVersion: kafka.stackable.tech/v1alpha1
+kind: KafkaCluster
+metadata:
+  name: simple-kafka
+spec:
+  image:
+    productVersion: 3.3.1
+    stackableVersion: "23.4.0-rc2"
+  clusterConfig:
+    authorization:
+      opa:
+        configMapName: simple-opa
+        package: kafka
+    zookeeperConfigMapName: simple-kafka-znode
+  brokers:
+    configOverrides:
+      server.properties:
+        opa.authorizer.cache.initial.capacity: "100"
+        opa.authorizer.cache.maximum.size: "100"
+        opa.authorizer.cache.expire.after.seconds: "10"
+    roleGroups:
+      default:
+        replicas: 1
+----
+
+A full list of settings and their respective defaults can be found https://github.com/anderseknert/opa-kafka-plugin[here].