We use cookies and other similar technology to collect data to improve your experience on our site, as described in our Privacy Policy.

AppsCode
  • KubeDB

    Run Production-Grade Databases on Kubernetes

    arrow_forward
    Stash

    Backup and Recovery Solution for Kubernetes

    arrow_forward
    KubeVault

    Run Production-Grade Vault on Kubernetes

    arrow_forward
    Voyager

    Secure HAProxy Ingress Controller for Kubernetes

    arrow_forward
    ConfigSyncer

    Kubernetes Configuration Syncer

    arrow_forward
    Guard

    Kubernetes Authentication WebHook Server

    arrow_forward
    KubeDB

    KubeDB simplifies Provision, Upgrade, Scaling, Volume Expansion, Monitor, Backup, Restore for various Databases in Kubernetes on any Public & Private Cloud

    • task_altLower administrative burden
    • task_altNative Kubernetes Support
    • task_altPerformance
    • task_altAvailability and durability
    • task_altManageability
    • task_altCost-effectiveness
    • task_altSecurity
    Stash

    A complete Kubernetes native disaster recovery solution for backup and restore your volumes and databases in Kubernetes on any public and private clouds.

    • task_altDeclarative API
    • task_altBackup Kubernetes Volumes
    • task_altBackup Database
    • task_altMultiple Storage Support
    • task_altDeduplication
    • task_altData Encryption
    • task_altVolume Snapshot
    • task_altPolicy Based Backup
    KubeVault

    KubeVault is a Git-Ops ready, production-grade solution for deploying and configuring Hashicorp's Vault on Kubernetes.

    • task_altVault Kubernetes Deployment
    • task_altAuto Initialization & Unsealing
    • task_altVault Backup & Restore
    • task_altConsume KubeVault Secrets with CSI
    • task_altManage DB Users Privileges
    • task_altStorage Backend
    • task_altAuthentication Method
    • task_altDatabase Secret Engine
    Voyager

    Secure HAProxy Ingress Controller for Kubernetes

    • task_altHTTP & TCP
    • task_altSSL
    • task_altPlatform support
    • task_altHAProxy
    • task_altPrometheus
    • task_altLet's Encrypt
    configsyncer

    Kubernetes Configuration Syncer

    • task_altConfiguration Syncer
    Guard

    Kubernetes Authentication WebHook Server

    • task_altIdentity Providers
    • task_altCLI
    • task_altRBAC
  • RESOURCES
  • Webinar New
CONTACT SALES
KubeDB KubeDB
github-icon twitterx-icon
TRY NOW FREE
You are looking at the documentation of a prior release. To read the documentation of the latest release, please visit here.

New to KubeDB? Please start here.

Monitoring Apache Kafka with KubeDB

KubeDB has native support for monitoring via Prometheus. You can use builtin Prometheus scraper or Prometheus operator to monitor KubeDB managed databases. This tutorial will show you how database monitoring works with KubeDB and how to configure Database crd to enable monitoring.

Overview

KubeDB uses Prometheus exporter images to export Prometheus metrics for respective databases. As KubeDB supports Kafka versions in KRaft mode, and the officially recognized exporter image doesn’t expose metrics for them yet - KubeDB managed Kafka instances use JMX Exporter instead. This exporter is intended to be run as a Java Agent inside Kafka container, exposing a HTTP server and serving metrics of the local JVM. To Following diagram shows the logical flow of database monitoring with KubeDB.

  Database Monitoring Flow

When a user creates a Kafka crd with spec.monitor section configured, KubeDB operator provisions the respective Kafka cluster while running the exporter as a Java agent inside the kafka containers. It also creates a dedicated stats service with name {database-crd-name}-stats for monitoring. Prometheus server can scrape metrics using this stats service.

Configure Monitoring

In order to enable monitoring for a database, you have to configure spec.monitor section. KubeDB provides following options to configure spec.monitor section:

Field Type Uses
spec.monitor.agent Required Type of the monitoring agent that will be used to monitor this database. It can be prometheus.io/builtin or prometheus.io/operator.
spec.monitor.prometheus.exporter.port Optional Port number where the exporter side car will serve metrics.
spec.monitor.prometheus.exporter.args Optional Arguments to pass to the exporter sidecar.
spec.monitor.prometheus.exporter.env Optional List of environment variables to set in the exporter sidecar container.
spec.monitor.prometheus.exporter.resources Optional Resources required by exporter sidecar container.
spec.monitor.prometheus.exporter.securityContext Optional Security options the exporter should run with.
spec.monitor.prometheus.serviceMonitor.labels Optional Labels for ServiceMonitor crd.
spec.monitor.prometheus.serviceMonitor.interval Optional Interval at which metrics should be scraped.

Sample Configuration

A sample YAML for TLS secured Kafka crd with spec.monitor section configured to enable monitoring with Prometheus operator is shown below.

apiVersion: kubedb.com/v1alpha2
kind: Kafka
metadata:
  name: kafka
  namespace: demo
spec:
  enableSSL: true
  tls:
    issuerRef:
      apiGroup: cert-manager.io
      name: kafka-ca-issuer
      kind: Issuer
  replicas: 3
  version: 3.4.0
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: standard
  monitor:
    agent: prometheus.io/operator
    prometheus:
      exporter:
        port: 9091
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s
  storageType: Durable
  terminationPolicy: WipeOut

Let’s deploy the above example by the following command:

$ kubectl create -f https://github.com/kubedb/docs/raw/v2023.10.9/docs/examples/kafka/monitoring/kf-with-monitoring.yaml
kafka.kubedb.com/kafka created

Here, we have specified that we are going to monitor this server using Prometheus operator through spec.monitor.agent: prometheus.io/operator. KubeDB will create a ServiceMonitor crd in databases namespace and this ServiceMonitor will have release: prometheus label.

Next Steps

What's on this Page

Search
Improve This Page