Logging with Fluent Bit and Fluentd in Kubernetes, pt.4

Fluent Bit is a fast and lightweight log processor, stream processor and forwarder. It’s gained popularity as the younger sibling of Fluentd due to its tiny memory footprint(~650KB compared to Fluentd’s ~40MB), and zero dependencies - making it ideal for cloud and edge computing use cases.

This post is part 4(and the final part) in a series of posts about logging using Fluent Bit and the Fluentd forwarder in Kubernetes, and it describes the steps to view the result of our logging pipeline.

Other posts in this series:

  • Part 1 Motivation and Architecture Overview.

  • Part 2 Deploying a single-node Elasticsearch, along with Kibana and Cerebro.

  • Part 3 Deploying Fluentd and Fluent Bit to work together.

Viewing the end result

We have completed the deployment of the logging stack and we can now view logs in Kibana by navigating to it’s location http://<kibana-host>:5601/ and clicking on the Discover icon.

Kibana UI

Creating separate views for applications and platform operations logs

  1. Create the kube.* index pattern. Click on the settings icon, then click on Index Patterns.

    Index pattern

  2. Click on Create index pattern.

    Create index pattern

  3. In the Index pattern field, type in kube.*. You should see some matches to the pattern you just entered. Click Next step.

    Define index pattern

  4. In the Time Filter field name field, select @timestamp, and then Create index pattern.

    Define time filter

  5. Repeat steps 1-4 for the index pattern kube-ops.*.

  6. Navigate back to the Discover view, and click on the dropdown list for index patterns. You should be able to see the new index patterns you just created(kube-ops. and kube.). Select kube-ops.*.

    View indices

  7. Notice that all logs displayed now are filtered to only come from the kube-system, kubeapps, and k8s-system-* namespaces.

Observe filtered logs

The kube-ops.* and kube.\* indices were created through the use of the Fluentd’s rewrite_tag_filter and routing capabilities. Now we can see the results in Elasticsearch and Kibana.

Cerebro

To view the health and status of Elasticsearch, navigate to the Cerebro UI http://<elasticsearch-hostname>:9000/. In the Node address text entry field, enter http://<elasticsearch-hostname>:9200. You should see a dashboard with green status and populated indices.

Cerebro UI

Summary

And that concludes this series on Logging with Fluent Bit and Fluentd in Kubernetes. I hope this has been useful if you’re just starting out building a Kubernetes logging pipeline.

There are many aspects which were not covered in this series, for example:

  • Security

  • Authentication and Authorization

  • Availability and Resiliency

  • Multi-cluster

  • Multi-site

These topics can get very deep, and I’m not able to cover all these aspects.

Nonetheless, I hope to this series has been useful wherever you found it and I welcome feedback!