Continuous Testing Kafka with Xinfra Monitor in Kubernetes

kafka, monitor

Monitors kafka as part of continuously cheking how the cluster performs. The challenge with this is to create script to produce and consume with some ludicrous amount of works. Instead of creating stuff in house, a mate refering a fancy tool called Kafka Monitor.

Kafka monitor, or now known as Xinfra Monitor is an open source software created by folks in Linkedin. This tools provide the produce, consume and offset test and measurement.
The fun part here, is Jolokia. So, Xinfra Monitor (Kafka Monitor), expose mbeans through Jolokia, so it can be accessed easily by any monitoring tools like Prometheus (plus Jolokia Exporter) or paid one like Datadog.

Kafka Xinfra Monitor Deployment #

In this case I would use kubernetes cluster to host the xinfra monitor, and also use strimzi as kafka and zookeeper operator. For the sake of simplicity, I will host everything on the kubernetes, minikube -in this example.

Prerequisites :

Nice to have locally :

Base assumption :

Deploy the Kafka Monitor / Xinfra Monitor #

Initial steps : #

Modify the Dockerfile #

If you can see. There is Dockerfile here.

FROM anapsix/alpine-java


WORKDIR /opt/kafka-monitor
ADD build/ build/
ADD bin/ bin/
ADD bin/ bin/
ADD config/ config/
ADD config/ config/
ADD docker/
ADD webapp/ webapp/

CMD ["/opt/kafka-monitor/"]

Side note : in the Dockerfile above, you will find out that the container will use the properties which can be configured in config/ With this config, you can set the configuration in granular way, also possible for you to monitor multiple kafka cluster - which I don’t really need in this case. Much better way to do this approach, is to convert the config file to kubernetes configmap.

In my case, I will not use the default Dockerfile approach, since I only want to monitor a single Kafka Cluster. So I modified the Dockerfile to this :

FROM anapsix/alpine-java

WORKDIR /opt/kafka-monitor
ADD build/ build/
ADD bin/ bin/
ADD bin/ bin/
ADD config/ config/

ENTRYPOINT ["/opt/kafka-monitor/bin/"]

Build Docker image #

Deploying the Kubernetes Objects #

I have create the kubernetes manifest here: xinfra-monitor-manifest.yaml

If you want to use exactly that manifest mentioned above, just do

kubectl apply -f -n kafka-monitor

If you need to change the kafka and zookeeper cluster, please change the argument lines here :

      - name: kafka-monitor
        image: robeevanjava/kafka-monitor:v2.5.10
        imagePullPolicy: Always
        env: []
        - --topic
        - kafkamonitor-topic-test
        - --broker-list
        - my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
        - --zookeeper
        - my-cluster-zookeeper-client.kafka.svc.cluster.local:2181

Change the topic, broker-list and zookeeper parameter accordingly.

In the manifest I only expose 1 port on the kubernetes service, and that is for Jolokia ref, a software that exposing mbeans to http. So it enable us to query the mbeans metric through http request, with json returning value.

To find the available metric, try to do these steps :

Now you have kafka monitor run on your kubernetes cluster.

Optional Steps #

Some of the organization uses prometheus as “default” observability toolings. We can use jolokia-exporter

You can clone the repo and build your very own Jolokia docker image and push it to your favourite container registry, but I will use the prebuilt one :)

I have kubernetes manifest here

So to apply it, run

kubectl apply -f -n kafka-monitor

Then you can just do either ServiceMonitor or write a prometheus scrape config like explained in the example here

PS : If the URL is broken, please refer to each linked gist file above :D