Integration of Prometheus and Grafana using Kubernetes.

Task :

Integrate Prometheus and Grafana and perform in following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services.
2. make their data to be remain persistent .
3. both of them should be exposed to outside world.

ConfigMap :

ConfigMap is used to store the configurations of the application running in Pod. For example, the httpd.conf for httpd. You could modify the httpd.conf file in Pod but everything will be gone once Pod is replace. So we can write the desired configurations in ConfigMap and tell Pod to use such configuration whenever Pod is replaced. Therefore Pod is always running with how we wanted it to be.

PVC :

when implementing Prometheus and Grafana on K8S. Everything will work fine but as we know that Pod is stateless and could be recycle anytime, therefore any settings configured on Prometheus and Grafana would be gone when Pod is replaced.

Therefore, to solve this problem, we need to use Persistent Volumes(PV), PersistentVolumeClaims(PVC) and ConfigMap.

PVs as named the provisioned storage space in cluster to store data. Data is stored in PV rather than Pod itself, so data is retain when Pod is replaced. PVC as named is used to claim the provisioned storage space for use.

Service :

This is used to expose the pods services with a port number so that it can be accessible from the outside world.

The default port range for kubernetes services is in between 30000-32767, you can select any between them or kubernetes will provide automatically any random port in that range.

Manifest file for Prometheus :

ConfigMap :

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:

        - targets: ['localhost:9090']

PVC :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pvc
  labels:
    app: prometheus
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:

      storage: 20Gi

Service :

To access the Prometheus WebUI use http://<minikube ip>:30001 .

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  ports:
    - port: 9090
      nodePort: 30001
  selector:
    app: prometheus
    tier: backend

  type: NodePort

Deployment :

Now we have to create the deployment for prometheus. It will pull the prometheus image from docker hub, configure it, then create and mount the persistent volume at the destination folder.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      tier: backend
  strategy:
    type: Recreate
  template:
    metadata:
      labels: 
        app: prometheus
        tier: backend
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus
        - name: data
          mountPath: /prometheus
        ports:
        - containerPort: 80
      securityContext:
        runAsUser: 0
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config
        - name: data
          persistentVolumeClaim:

            claimName: prometheus-pvc

Manifest file for Grafana :

PVC :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  labels:
    app: grafana
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:

      storage: 20Gi

Service :

we will be able to access the Grafana WebUI at http://<minikube ip>:30002 .

apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  ports:
    - port: 3000
      nodePort: 30002
  selector:
    app: grafana
    tier: frontend

  type: NodePort

Deployment :

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: grafana
  labels:
    app: grafana
    tier: frontend
spec:
  selector:
    matchLabels:
      app: grafana
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: grafana
        tier: frontend
    spec:
      containers:
      - image: grafana/grafana:latest
        name: grafana
        ports:
        - containerPort: 3000
          name: grafana
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:

          claimName: grafana-pvc

Kustomization.yml :

This file will bind the two resources (prometheus-deployment.yml and grafana-deployment.yml).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - prometheus-deployment.yaml

  - grafana-deployment.yaml

Now to deploy the complete setup the command to be used is...

kubectl create -k .

To view the deployments

To view the ConfigMaps...

To view the pods...

To view the PVCs...

kubectl get all

Prometheus WebUI :

to acces the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.

http://<minikube ip>:30001

and this will be the first page you will see

enter a query in the Expression text field and click the execute button to get the value or click on graph to get a graph for visualization.

Grafana WebUI :

to acces the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.

http://<minikube ip>:30002

when logged in for the first time the user and password will be admin, then after you can change the password of it.

Dashboard:

click on the Data Sources to add your data sources.

Set Prometheus as the data source.

you can add as many panel as you want with different queries.

As the pods have a persistent storage, even the pods crashes as your data is saved in a PVC your data will not be lost and replica set will launch a new pod and connect it with the PVC instantly.