Skip to main content
Version: 1.23

Kubernetes Fury Logging

Kubernetes Fury Logging provides a logging stack for the Kubernetes Fury Distribution (KFD).

If you are new to KFD please refer to the official documentation on how to get started with KFD.

Overview

Kubernetes Fury Logging uses a collection of open source tools to provide the most resilient and robust logging stack for the cluster.

The central piece of the stack is the CNCF certified open source search engine elasticsearch, combined with its analytics and visualization platform kibana. The logs are collected using a node-level data collection and enrichment agent fluentbit(deployed as Daemonsets), pushing it to the Elasticsearch via fluentd.

Following is a high level design diagram of the module:

logging module

All the components are deployed in the logging namespace in the cluster.

Packages

The following packages are included in the Fury Kubernetes Logging katalog:

PackageVersionDescription
cerebro0.9.4Web admin tool that helps you manage your Elasticsearch cluster via a graphical user interface
curator5.8.4Manages elasticserach indices and snapshots, alongwith configurations to set the retention log policies
curator-s35.8.4curator with S3 compliant bucket support
elasticsearch-single7.16.2Single node elasticsearch deployment
elasticsearch-triple7.16.2Three node high-availability elasticsearch deployment
fluentd1.14.2Data collector for unified logging that can store collected data in Elasticsearch
kibana7.16.2Analytics and visualization platform for Elasticsearch

Click on each package to see its full documentation.

Compatibility

Kubernetes VersionCompatibilityNotes
1.20.xNo known issues
1.21.xNo known issues
1.22.xNo known issues
1.23.x⚠️Conformance tests passed. Not officially supported.

Check the compatibility matrix for additional informations about previous releases of the modules.

Usage

Prerequisites

ToolVersionDescription
furyctl>=0.6.0The recommended tool to download and manage KFD modules and their packages. To learn more about furyctl read the official documentation.
kustomize>=3.5.0Packages are customized using kustomize. To learn how to create your customization layer with kustomize, please refer to the repository.

Deployment

  1. List the packages you want to deploy and their version in a Furyfile.yml
bases:
- name: logging/cerebro
version: "v1.10.3"
- name: logging/curator
version: "v1.10.3"
- name: logging/elasticsearch-single
version: "v1.10.3"
- name: logging/fluentd
version: "v1.10.3"
- name: logging/kibana
version: "v1.10.3"

See furyctl documentation for additional details about Furyfile.yml format.

  1. Execute furyctl vendor -H to download the packages

  2. Inspect the download packages under ./vendor/katalog/logging.

  3. Define a kustomization.yaml that includes the ./vendor/katalog/logging directory as resource.

resources:
- ./vendor/katalog/logging/cerebro
- ./vendor/katalog/logging/curator
- ./vendor/katalog/logging/elasticsearch-single
- ./vendor/katalog/logging/fluentd
- ./vendor/katalog/logging/kibana
  1. To deploy the packages to your cluster, execute:
kustomize build . | kubectl apply -f -

Common Customisations

Setup a high-availability three-node elasticsearch

Logging module offers an out of the box, highly-available setup for elasticsearch instead of a single node version. To set this up, in the Furyfile and kustomization, you can replace elasticsearch-single with elasticsearch-triple.

Setup curator with datastore in an s3 compliant bucket

To setup curator with s3 compliant bucket support, we provide curator-s3. To set it up, instead of curator, use curator-s3 in Furyfile and kustomization in the above defined deployment step.

To configure the s3 bucket, you will have to edit the file secret-es-backup.env with the following content, with the right configuration secrets:

AWS_ACCESS_KEY_ID=AAAA
AWS_REGION=eu-west-1
AWS_SECRET_ACCESS_KEY=myKey
S3_BUCKET_NAME=my-s3_bucket