Three-Tier App Deployment on Docker with Observability using Prometheus & Grafana

Prometheus and Grafana are powerful tools that work together to help you monitor and visualize your systems.

Prometheus is an open-source tool that collects data about your systems, like servers, apps, and databases, at regular intervals. It stores this data in a time-series database, which means it keeps track of data over time. You can then query and analyze this data to see how well your systems are performing.

Grafana is a platform that helps you create dashboards to display the data collected by Prometheus. These dashboards can include graphs, charts, and tables to help you easily understand the performance of your systems and spot any issues

Key benefits of using Prometheus and Grafana together

• Real-time monitoring: Prometheus collects data continuously, so you get real-time insights into how your systems are performing.
• Flexible querying: Prometheus lets you query and analyze your data in many different ways.
• Custom dashboards: Grafana helps you create dashboards that show the data in a way that makes sense for you.
• Alerting: Prometheus can send alerts when something goes wrong, helping you act quickly
• Scalability: Both Prometheus and Grafana can handle large systems, so they work well for big environments

Why Observability : Because of over application running in carton way

1 Step : Create Your Instance

Update your system : sudo apt-get update && sudo apt-get upgrade -y

Install Docker and Docker Compose : sudo apt-get install docker.io -y sudo apt-get install docker-compose-v2 -y

Clone This Repository

• Navigate to the observability directory.
• Start the stack with Docker Compose

• Verify the container is running docker ps
• Access the application via the browser at http://localhost:80

The prometheus service section in your docker-compose.yml file defines how the Prometheus container is configured to run. Here's a breakdown of the key parts of the Prometheus section and what the command does

Prometheus Service Configuration:

image: prom/prometheus:latest

Specifies the Docker image to use for Prometheus. It pulls the latest Prometheus image from Docker Hub.

container_name: prometheus

Sets the name of the container as prometheus to make it easier to reference when running Docker commands.

restart: unless-stopped

This policy tells Docker to automatically restart the container unless it is explicitly stopped. If the container stops unexpectedly, it will automatically restart.

networks: - fullstack-chatapp

Connects the Prometheus container to the fullstack-chatapp network defined elsewhere in the docker-compose.yml file. This allows Prometheus to communicate with other services (e.g., your backend or frontend).

volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml

This mounts the prometheus.yml configuration file from your local filesystem (relative to the directory where the docker-compose.yml file is located) to the container’s /etc/prometheus/prometheus.yml. The container uses this configuration file to scrape metrics from the defined targets.

command section:

This part specifies the custom arguments that will be passed when starting the Prometheus container. They control the configuration and behavior of Prometheus:

ports: - "9090:9090"

Exposes port 9090 on the host machine and maps it to port 9090 inside the container. This is the default port that Prometheus uses to serve its web interface

Code Reference : https://sulkurl.com/ftx

Add New yml file : prometheus.yml

global:
  scrape_interval: 1m

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:80']        

Your prometheus.yml configuration file looks almost correct for basic Prometheus setup. However, here are some key points and adjustments you might consider to ensure it works effectively

Issues and Recommendations

1 Indentation Issue :

• YAML syntax is strict about indentation. In your configuration, the - job_name: 'docker' section appears to be correctly aligned, but if there are hidden indentation errors
• Ensure consistent spacing (use 2 spaces per indentation level).

2 Target Addresses:

• The target localhost:80 is pointing to the local machine's port 80. Ensure the service you intend to monitor is accessible on this port.
• If you’re running Prometheus in a Docker container, replace localhost with the container name or service name as defined in your Docker Compose file (e.g., frontend)

3 Scrape Interval

• The scrape interval is set to 1m for both jobs. This is fine for most use cases, but you can adjust it based on how frequently you need updates.

4 Adding Labels :

Consider adding labels to your targets for better identification in Prometheus metrics

static_configs:

- targets: ['localhost:9090']

labels:

service: prometheus

5 Verify Prometheus Access

• After making these changes, ensure Prometheus can reach the specified targets by checking the Targets page in the Prometheus web interface (http://<host-ip>:9090/targets)

Corrected and Enhanced Configuration

6 Applying and Testing

• Save this file as prometheus.yml.
• Restart Prometheus to apply the new configuration
• docker-compose down
• docker-compose up -d
• Verify the configuration by visiting Prometheus's Targets page
• http://<host-ip>:9090/targets

Docker Compose Up

Prometheus metrics are a standardized way of collecting and storing data about system and application performance. These metrics are numerical data points collected over time, which are used for monitoring, alerting, and analysis

The error indicates that Prometheus is attempting to scrape metrics from http://localhost:80/metrics, but it cannot connect to this endpoint because no service is running or exposing /metrics on port 80 of the Prometheus container

1. Reasons for the "Down" State : No Service on Port 80 Prometheus is trying to access localhost:80 inside its container, but no service or application is running there to provide metrics.
2. Incorrect Target Address : localhost in the Prometheus container refers to the Prometheus container itself, not the host machine or other containers
3. Missing Metrics Endpoint : The application running on port 80 may not have an endpoint at /metrics or isn't instrumented to expose metrics in the Prometheus format.
4. Networking Configuration : The target address should use the container name or service name defined in your Docker Compose file to allow inter-container communication (e.g., frontend:80).

Solution: Use the Container Name

In Docker Compose, services in the same network can communicate using their service names. Replace localhost:80 with the correct service name from your docker-compose.yml file

Why Use cAdvisor?

cAdvisor (Container Advisor) is a tool from Google designed to collect, monitor, and export resource usage and performance metrics for containers. It works well with Prometheus for container-specific monitoring

1. Tracks CPU, memory, network, and disk usage per container.
2. Exposes metrics in the Prometheus format, making it easy to scrape and visualize in Grafana
3. Specifically designed to work with Docker and Kubernetes environments
4. Provides per-container metrics for detailed monitoring
5. Runs as a standalone container, exposing metrics via an HTTP endpoint (/metrics).

How to Use cAdvisor with Prometheus some Changes in Docker Compose

If you are actually using it then check that Jenkins is not already installed there.

Now My cAdvisor is running on port 8080

Now Click on Docker Containers

Now Move to solve this problem

• cAdvisor getting container Data & send data on port 8080
• Node Exporter get system like server information host information & send 9100

cAdvisor (Container Advisor)

• Purpose: cAdvisor is a tool for monitoring container performance. It collects metrics like CPU, memory, disk, and network usage for Docker containers.
• Port: It exposes metrics at port 8080 by default.

Metrics Provided:

• CPU usage, memory usage, network statistics, and disk I/O statistics.
• Metrics about container resource utilization (e.g., CPU time, memory usage).
• When running cAdvisor, it will expose a metrics endpoint on port 8080 (or any port you configure). This endpoint can be scraped by Prometheus to collect the performance data.

Node Exporter

• Purpose: Node Exporter is used for gathering system-level metrics about the host, such as CPU usage, memory usage, disk I/O, and network statistics. It provides overall host machine metrics.
• Port: It exposes metrics at port 9100 by default.
• Metrics Provided:CPU usage, memory stats, disk usage, network statistics, filesystem usage, etc.
• Node Exporter runs on the host and exposes a metrics endpoint on port 9100. It helps collect metrics related to the host system.

Now Add Node Exporter code on docker-compose.yml file

node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - "9100:9100"
    networks:
      - fullstack-chatapp        

Prometheus scrape job for collecting metrics from a Node Exporter instance

  - job_name: 'node'
    scrape_interval: 1m
    static_configs:
      - targets: ['node-exporter:9100']        

Run Promethous : docker-compose up --no-deps --build prometheus

again Docker Compose up

All Server Metrics

after the change all code need to check on localhost:9090/targets to check logs

sum by(mode)(irate(node_cpu_seconds_total{mode!="idle"}[1h]))

Integration of Grafana with Prometheus and other services

Grafana is widely used for visualizing metrics collected by Prometheus and other monitoring tools. Here's how to set up and integrate Grafana into your environment.

Step 1: Add Grafana to Your docker-compose.yml

version: '3.8'

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - fullstack-chatapp

networks:
  fullstack-chatapp:
    driver: bridge

volumes:
  grafana_data:        

Step 2: Start the Services

Run the following command to start Grafana along with other services

• docker-compose up -d

Step 3: Access Grafana

• Open a web browser and navigate to: http://localhost:3000
• Log in with the default credentials: Username: admin
• Password: admin (you can change this later)

Step 4: Add Prometheus as a Data Source

• After logging in to Grafana:
• Go to the "Configuration" → "Data Sources".
• Click on "Add data source".
• Select Prometheus.
• Enter the URL of your Prometheus instance (e.g., http://prometheus:9090 if using Docker Compose networks).
• Click Save & Test to verify the connection.

Step 5: Create Dashboards and Panels

• Go to "Dashboards" → "New Dashboard".
• Add a new panel.
• Use Prometheus queries (e.g., rate(node_cpu_seconds_total[1m]) * 100 for CPU usage) to visualize metrics.

Step 7: Additional Integrations

You can integrate Grafana with other monitoring tools like:

• cAdvisor: Visualize container metrics.
• Node Exporter: Track system-level metrics like CPU, memory, and disk.
• Redis Exporter: Monitor Redis performance.

Note if your Grafana dashboard not work then check your Prometheus is working or not if it not working then use the command

• docker compose down
• docker-compose up --no-deps --build Prometheus
• docker compose up

Click on New Connection : type Promethous select Prometheus

Grafana Templates Theme : link

Copy Dashboard ID