Docker Container Log analysis with Elastic Stack

Logging is an essential component within any application. Logs enable you to analyze and sneak peak into what’s happening within your application code like a story. Software developers spend a large part of their day today life monitoring, troubleshooting and debugging applications, which can sometimes be a nightmare. Logging allows software developers to make this hectic process much more easier and smooth.

If you have containerized your application with a container platform like Docker, you may be familiar with docker logs which allows you to see the logs created within your application running inside your docker container. Why then think of Elastic Stack to analyze your logs. Well, there are mainly two burning problems here,

• Imagine you have tens, hundreds, or even thousands of containers generating logs, SSH in to all those servers and extracting logs won’t work well.
• Also containers are immutable and ephemeral, which means they have shorter life span, so once your containers are gone and replaced with new containers all of your application logs related to old containers are gone.

So the ultimate solution for this is to create a centralized logging component for collecting all of your container logs into a single place. This is where Elastic Stacks comes in.

Elastic Stack mainly consist of four major components,

• Beats, is the new member which made the ELK Stack, Elastic Stack, Beats is actually a light weight log data shippers which can push logs to ELK Stack. For this post I will be using Filebeats a member in Beats family which offers a lightweight way to collect and forward and centralize logs and files.
• Logstash, is a component which aggregate, modify, transfer logs form multiple input locations into Elasticsearch.
• Elasticsearch, a distributed, JSON-based search and analytics engine that stores and indexes data (log entries in this case) in a scalable and manageable way.
• Kibana, an enriched UI to analyze and easily access data in Elasticsearch.

In the original post I published on Medium.com about this, I show how above mentioned components can be used to implement a centralized log analyzer to collect and extract logs from Docker containers.

Visit my original article published on Medium.com about this topic by visiting the link - https://medium.com/@RavinduNOnline/docker-container-log-analysis-with-elastic-stack-53d5ec9e5953