web-server/doc/en/user/source/community/monitor-kafka/usage.rst

.. _monitor_kafka_usage:

Usage of Monitoring Kafka extension
==========================================

To try out there is a ``docker-compose.yml`` file in the ``test/resources`` directory.
This will start up a Kafka broker and a schema registry instance.
To start it up you need docker and docker-compose installed then run::

    # start kafka
    cd src/community/monitor-kafka/src/test/resources
    docker-compose up -d

This will start up the Kafka broker and the schema registry.

Once Kafka is running start geoserver with monitor and monitor-kafka extension from ``src/web/app``::

     mvn jetty:run -P monitor,monitor-kafka

This will initialize the data directory in ``src/web/app/src/main/webapp/data`` and start up geoserver.
Stop the geoserver again.

With this you will get the default monitor config installed automatically in ``src/web/app/src/main/webapp/data/monitoring/``. Edit it to use the Kafka storage.

Then you need to configure the ``monitor-kafka`` extension for it with::

    storage=kafka
    kafka.bootstrap.servers=localhost:9092
    kafka.schema.registry.url=http://localhost:8081



Create your topic with::

    docker exec -ti broker kafka-topics --bootstrap-server localhost:9092 --create --topic geoserver-monitor --partitions 1 --replication-factor 1 --if-not-exists

Start geoserver again with::

     mvn jetty:run -P monitor,monitor-kafka


Check that the monitoring extension is actually enabled by looking for the following log line::

    INFO   [geoserver.monitor] - Kafka connection established and topic geoserver-monitor exists

Head over to http://localhost:8080/geoserver and hit some map, so you get some data into the topic.

If you want to consume the data you need a consumer which has the right Deserializer configured (``io.confluent.kafka.serializers.KafkaAvroDeserializer``).

The easiest way to do this is to enter the schema registry container with::


    docker exec -ti schema-registry bash
    # from within the container run:
      kafka-avro-console-consumer \
        --bootstrap-server broker:29092 \
        --topic geoserver-monitor \
        --from-beginning

   # or directly from the host machine so you can pipe it into jq or a file...
    docker exec -ti schema-registry kafka-avro-console-consumer \
        --bootstrap-server broker:29092 \
        --topic geoserver-monitor \
        --from-beginning | tee >(grep -E "^{" | jq) | grep -vE "^{"


Then you will see messages like this:

.. code-block:: json

    {
        "id": 2,
        "status": "FINISHED",
        "category": "OWS",
        "path": "/tiger/wms",
        "queryString": "SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=true&STYLES&LAYERS=tiger:giant_polygon&exceptions=application/vnd.ogc.se_inimage&SRS=EPSG:4326&WIDTH=768&HEIGHT=384&BBOX=3.7958354296875,-40.4131489453125,71.2627583203125,-6.6962260546875",
        "body": "",
        "bodyContentLength": 0,
        "bodyContentType": "",
        "httpMethod": "GET",
        "startTime": 1697172148809,
        "endTime": 1697172148902,
        "totalTime": 93,
        "remoteAddress": "[0:0:0:0:0:0:0:1]",
        "remoteHost": "0:0:0:0:0:0:0:1",
        "host": "localhost",
        "internalHost": "client",
        "remoteUser": "anonymous",
        "remoteUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "remoteCountry": "",
        "remoteCity": "",
        "remoteLat": 0.0,
        "remoteLon": 0.0,
        "service": "WMS",
        "operation": "GetMap",
        "owsVersion": "1.1.1",
        "subOperation": "",
        "resources": [
            "tiger:giant_polygon"
        ],
        "responseLength": 6594,
        "responseContentType": "image/png",
        "errorMessage": "",
        "responseStatus": 0,
        "httpReferer": "",
        "coordinateReferenceSystem": "EPSG:WGS 84",
        "minx": 3.7958354296875,
        "miny": -40.4131489453125,
        "maxx": 71.2627583203125,
        "maxy": -6.6962260546875,
        "cacheResult": "",
        "missReason": "",
        "resourceProcessingTimes": [],
        "labelingProcessingTime": 0
    }
    {
        "id": 3,
        "status": "FINISHED",
        "category": "OWS",
        "path": "/tiger/wms",
        "queryString": "SERVICE=WMS&VERSION=1.1.1&REQUEST=GetFeatureInfo&FORMAT=image/png&TRANSPARENT=true&QUERY_LAYERS=tiger:giant_polygon&STYLES&LAYERS=tiger:giant_polygon&exceptions=application/vnd.ogc.se_inimage&INFO_FORMAT=text/html&FEATURE_COUNT=50&X=50&Y=50&SRS=EPSG:4326&WIDTH=101&HEIGHT=101&BBOX=39.23891004908502,-32.22561743843973,48.11586317408502,-23.348664313439727",
        "body": "",
        "bodyContentLength": 0,
        "bodyContentType": "",
        "httpMethod": "GET",
        "startTime": 1697172178181,
        "endTime": 1697172178213,
        "totalTime": 32,
        "remoteAddress": "[0:0:0:0:0:0:0:1]",
        "remoteHost": "",
        "host": "localhost",
        "internalHost": "client",
        "remoteUser": "anonymous",
        "remoteUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "remoteCountry": "",
        "remoteCity": "",
        "remoteLat": 0.0,
        "remoteLon": 0.0,
        "service": "WMS",
        "operation": "GetFeatureInfo",
        "owsVersion": "1.1.1",
        "subOperation": "",
        "resources": [
            "giant_polygon"
        ],
        "responseLength": 803,
        "responseContentType": "text/html;charset=utf-8",
        "errorMessage": "",
        "responseStatus": 0,
        "httpReferer": "",
        "coordinateReferenceSystem": "EPSG:WGS 84",
        "minx": 43.63344129908502,
        "miny": -27.743195563439727,
        "maxx": 43.63344129908502,
        "maxy": -27.743195563439727,
        "cacheResult": "",
        "missReason": "",
        "resourceProcessingTimes": [],
        "labelingProcessingTime": 0
    }