Find the PersistentVolumeClaim utilization on AWS EBS

You are running your container workloads on AWS Elastic Kubernetes Services (EKS), and some of your containers have storage that are backed by PersistentVolumeClaim (PVC). For example, here is a PVC I have for Grafana:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: monitoring-grafana
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-gp2
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi        

What if I want to find out how much of those 5Gi are actually in use?

First I need to find the name of the PersistentVolume that is supporting the PVC:

kubectl describe pvc monitoring-grafana -n monitoring | grep Volume

Volume:        pvc-b5608574-b2e1-4e8c-9f83-f3b83fb75e40
VolumeMode:    Filesystem        

Then I need to find the AWS Elastic Block Storage (EBS) volume ID that backs the PersistentVolume:

kubectl describe pv pvc-b5608574-b2e1-4e8c-9f83-f3b83fb75e40 | grep VolumeHandle

VolumeHandle:      vol-05dcbb0027df6c381        

Login to the AWS Console and navigate to the EC2 Volume dashboard for the region where you have your EC2 instances and EBS volumes that powers your Kubernetes cluster.

Search for the volume you found in the prevous kubectl command. For instance in my case, I use the eu-west-1 region, and the url will look like this:

https://eu-west-1.console.aws.amazon.com/ec2/home?region=eu-west-1#Volumes:v=3;search=:vol-05dcbb0027df6c381

As you can see in the screendump, you will get information about both the attached EC2 instance and the attached storage device. The EC2 instance is one of the nodes in your cluster.

Copy the name of the storage device (in this case /dev/xvdaa), because you will need it later.

Click on the name of the attaced instance, and you will get to the EC2 instance dashboard.

In the Public IPv4 column you should see the public IP of the Kubernetes node, on which the EBS volume is attached. I have removed my public IP from the screendump.

Connect to the IP over SSH using the private key you chose when creating your Kubernetes cluster. Example:

ssh -i <ec2-private-key> ec2-user@<public-ip>        

Find the disk utilization of the EBS volume by running the df command on the attached volume you copied earlier:

sudo df -Th /dev/xvdaa

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme1n1   ext4  4.8G   13M  4.8G   1% /var/lib/kubelet/pods/52edc7ef-0f0e-46c6-8c0b-484b8e3a8f79/volumes/kubernetes.io~csi/pvc-b5608574-b2e1-4e8c-9f83-f3b83fb75e40/mount        

So in my case, I am only using ~ 13M of the ~ 5G reserved.

I also published this article on my blog earlier today: https://medium.com/@audun-nes/find-the-persistentvolumeclaim-utilization-on-aws-ebs-60a5f6b81926