Logging with Fluent Bit and Fluentd in Kubernetes, pt.3

Fluent Bit is a fast and lightweight log processor, stream processor and forwarder. It’s gained popularity as the younger sibling of Fluentd due to its tiny memory footprint(~650KB compared to Fluentd’s ~40MB), and zero dependencies - making it ideal for cloud and edge computing use cases.

This post is part 3 in a series of posts about logging using Fluent Bit and the Fluentd forwarder in Kubernetes, and it describes the steps to deploy Fluentd and Fluent Bit.

Other posts in this series:

  • Part 1 Motivation and Architecture Overview.

  • Part 2 Deploying a single-node Elasticsearch, along with Kibana and Cerebro.

  • Part 4 Viewing the End Result.

Deploying Fluentd

Fluentd is the log aggregator and processor stage before Elasticsearch, and we will deploy this now. We will use the Bitnami Fluentd Helm chart.

  1. Extend the Bitnami image by installing the rewrite_tag_filter plugin. We will push this up to docker hub as a custom image, to be used later.

    cat <<EOF | docker build -t ${CUSTOM_DOCKER_IMG} -
    FROM bitnami/fluentd:1.10.4-debian-10-r2
    LABEL maintainer "Bitnami <containers@bitnami.com>"
    ## Install custom Fluentd plugins
    RUN fluent-gem install 'fluent-plugin-rewrite-tag-filter'
    docker push ${CUSTOM_DOCKER_IMG}
  2. Add the Bitnami Helm repo.

    helm repo add bitnami https://charts.bitnami.com/bitnami
  3. Create a custom ConfigMap that can send output to Elasticsearch. Substitute ES_HOST=es.lab.example.com with your own Elasticsearch hostname.

    cat <<EOF | sed "s/<elasticsearch-host>/${ES_HOST}/" > fluentd-elasticsearch-output-configmap.yaml
    apiVersion: v1
    kind: ConfigMap
      name: fluentd-elasticsearch-output
      namespace: k8s-system-logging
      fluentd.conf: |
        # Prometheus Exporter Plugin
        # input plugin that exports metrics
          @type prometheus
          port 24231
        # input plugin that collects metrics from MonitorAgent
          @type prometheus_monitor
            host \${hostname}
        # input plugin that collects metrics for output plugin
          @type prometheus_output_monitor
            host \${hostname}
        # Ignore fluentd own events
        <match fluent.**>
          @type null
        # TCP input to receive logs from the forwarders
          @type forward
          port 24224
        # HTTP input for the liveness and readiness probes
          @type http
          port 9880
        # Throw the healthcheck to the standard output instead of forwarding it
        <match fluentd.healthcheck>
          @type stdout
        # rewrite tags based on which namespace the logs come from.
        <match kube.**>
          @type rewrite_tag_filter
            key     \$['kubernetes']['namespace_name']
            pattern /^(kube-system|kubeapps|k8s-system-[\S]+)$/
            tag     ops.\${tag}
            key     \$['kubernetes']['namespace_name']
            pattern /.+/
            tag     prod.\${tag}
        <match ops.kube.**>
          @type copy
          @id output_copy_ops
            @type elasticsearch_dynamic
            @id output_elasticsearch_ops
            host <elasticsearch-host>
            port 9200
            logstash_format true
            logstash_prefix kube-ops.\${record['kubernetes']['namespace_name']}
              @type file
              path /opt/bitnami/fluentd/logs/buffers/ops-logs.buffer
              flush_thread_count 2
              flush_interval 5s
        <match prod.kube.**>
          @type copy
          @id output_copy
            @type elasticsearch_dynamic
            @id output_elasticsearch
            host <elasticsearch-host>
            port 9200
            logstash_format true
            logstash_prefix kube.\${record['kubernetes']['namespace_name']}
              @type file
              path /opt/bitnami/fluentd/logs/buffers/logs.buffer
              flush_thread_count 2
              flush_interval 5s
    kubectl apply -f fluentd-elasticsearch-output-configmap.yaml
  4. Install the Fluentd Helm chart. Substitute the image.repository and image.tag values with the relevant values from step 1.

    helm install fluentd \
         --set image.repository=georgegoh/fluentd \
         --set image.tag=1.10.4-debian-10-r2-rewrite_tag_filter \
         --set forwarder.enabled=false \
         --set aggregator.enabled=true \
         --set aggregator.replicaCount=1 \
         --set aggregator.port=24224 \
         --set aggregator.configMap=fluentd-elasticsearch-output \
         bitnami/fluentd -n k8s-system-logging

    Consider using a Horizontal Pod Autoscaler for the fluentd StatefulSet to react to higher volumes of incoming logs.

    Now that we’ve completed setup of Fluentd, Elasticsearch and Kibana, it’s time to move on to Fluent Bit and complete the logging setup.

Deploy Fluent Bit

  1. Add the Fluent Helm repo.

    helm repo add fluent https://fluent.github.io/helm-charts
  2. Save the following in a file called values.yaml.

    on_minikube: false
        repository: fluent/fluent-bit
        tag: v1.3.7
      pullPolicy: Always
    # When enabled, exposes json and prometheus metrics on {{ .Release.Name }}-metrics service
      enabled: true
           k8s-app: fluent-bit
          'prometheus.io/path': "/api/v1/metrics/prometheus"
          'prometheus.io/port': "2020"
          'prometheus.io/scrape': "true"
        port: 2020
        type: ClusterIP
        enabled: false
        additionalLabels: {}
        # namespace: monitoring
        # interval: 30s
        # scrapeTimeout: 10s
      type: forward
        host: fluentd-0.lab.spodon.com
        port: 24224
        tls: "off"
        tls_verify: "on"
        tls_debug: 1
        shared_key: thisisunsafe
      enabled: true
      ## List the respective parsers in key: value format per entry
      ## Regex required fields are name and regex. JSON and Logfmt required field
      ## is name.
        - name: cri-mod
          regex: "^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$"
          timeFormat: "%Y-%m-%dT%H:%M:%S.%L%z"
          timeKey: time
        - name: catchall
          regex: "^(?<message>.*)$ }"
      logfmt: []
      json: []
    env: []
    podAnnotations: {}
    fullConfigMap: false
    existingConfigMap: ""
    rawConfig: |-
      @INCLUDE fluent-bit-service.conf
      @INCLUDE fluent-bit-input.conf
      @INCLUDE fluent-bit-filter.conf
      @INCLUDE fluent-bit-output.conf
      input: ""
      audit: ""
      filter: |
        Merge_Parser     catchall
        Keep_Log         Off
      output: ""
    extraPorts: []
    extraVolumes: []
    extraVolumeMounts: []
    resources: {}
    hostNetwork: false
    dnsPolicy: ClusterFirst
    tolerations: []
    nodeSelector: {}
    affinity: {}
      flush: 1
      logLevel: info
        memBufLimit: 5MB
        parser: cri-mod
        path: /var/log/containers/*.log
        ignore_older: ""
        enabled: false
            - docker.service
            - kubelet.service
            - node-problem-detector.service
        maxEntries: 1000
        readFromTail: true
        stripUnderscores: false
        tag: host.*
      enable: false
        memBufLimit: 35MB
        parser: docker
        tag: audit.*
        path: /var/log/kube-apiserver-audit.log
        bufferChunkSize: 2MB
        bufferMaxSize: 10MB
        skipLongLines: true
        key: kubernetes-audit
      kubeURL: https://kubernetes.default.svc:443
      kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubeTag: kube
      kubeTagPrefix: kube.var.log.containers.
      mergeJSONLog: true
      mergeLogKey: ""
      enableParser: true
      enableExclude: true
      useJournal: false
      create: true
      pspEnabled: false
      directory: /var/lib/fluent-bit
      # Specifies whether a ServiceAccount should be created
      create: true
      # Annotations to add to the service account
      annotations: {}
      # The name of the ServiceAccount to use.
      # If not set and create is true, a name is generated using the fullname template
    ## Specifies security settings for a container
    ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
    securityContext: {}
      # securityContext:
      #   privileged: true
    ## Specifies security settings for a pod
    ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
    podSecurityContext: {}
      # podSecurityContext:
      #   runAsUser: 1000
  3. Install the Fluent Bit helm chart, using values created in the previous step.

    helm install --name fluent-bit -f values.yaml stable/fluent-bit


In this post, I shared the steps for deploying Fluentd and Fluent Bit in a Forwarding pattern.

In Part 4 I’ll wrap up with creating indices in Kibana and viewing the results.