Real Time Log Analytics Website
Real time application can be created using open source stack using technologies like nodejs, SocketIO, MiNiFi, NiFi etc.
The approach would have following steps:
1) Ship the logs from various source system to central server.
2) Central server can manipulate the logs before writing to Message Queue or write into message queue directly.
3) Web Application server reads from the message queue, transform or translate into required format like JSON and pushes that data onto front end HTML page.
4) HTML receives the data in the real time, uses charting techniques to create charts and display it on the page.
Overall Architecture
Below is how overall architecture would look like:
Let’s walk through all the steps one by one:
1) Ship the logs from various source systems to central server:
We are using MiNiFi and NiFi for shipping the logs from each application server. MiNiFi is a light weight component that can be installed on the application server to tail the log file and send the latest records to the server. Below is the link that will help you set up MiNiFi and NiFi.
https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html
2) Central server can manipulate the logs before writing to Message Queue or write into message queue directly.
Here we are not manipulating the logs, directly writing them to Kafka once received. Remember you need to set up Kafka before you can use this step. Setting up easy single node Kafka server along with zookeeper can be done by quick start guide in a link below:
https://kafka.apache.org/quickstart
Once you have Kafka set up and it is running. Pick up publish Kafka on Nifi Server and add the Kafka details to it in publish kafka properties.
This process will move the data real time to Kafka.
3) Web Application server reads from the message queue, transform or translate into required format like JSON and pushes that data onto front end HTML page
Here we are using NodeJS based webserver along with Express JS. The NodeJS has a package node-kafka that can be used to read from Kafka. Once you have read the record from Kafka you can transform aggregate in any way you want to. After record is converted into data that needs to be shown to end users, emit this data using socketio package. Socketio uses underlying websocket technology. Below are the links for socketio and kafka-node:
https://github.com/SOHU-Co/kafka-node
https://www.npmjs.com/package/socket.io
4) HTML receives the data in the real time, uses charting techniques to create charts and display it on the page.
Socketio client is used in html pages to receive the data in real time. If you are using JSON data D3JS can be used to create charts and animations based on data.
Once you stitch all these together you can have working real time dashboard of log analytics.
