Controlling The Storage In Hadoop Cluster By Data Node

!! Hello Folks !!

This is Chirag Goyal

🔰 Welcome you all to my article based on 𝗧𝗔𝗦𝗞 - 𝟰.𝟭 of 𝘼𝙍𝙏𝙃 - 𝙏𝙝𝙚 𝙎𝙘𝙝𝙤𝙤𝙡 𝙊𝙛 𝙏𝙚𝙘𝙝𝙣𝙤𝙡𝙤𝙜𝙞𝙚𝙨✨

!! TASK DESCRIPTION:

🤔 In Hadoop cluster , find how to contribute limited/specific amount of storage as a slave node to the cluster ?

Hint : Use Of Partitions

🎯 TASK COMPLETION:

🔰 START SLAVE NODE:

1) ON AWS Cloud , I have build the Hadoop cluster for this we have to start the Hadoop services as SLAVE Node and MASTER Node .

🔰 df –h

2) In windows we store any folder or directory inside one storage drive like for example C-drive .Normally by default the folder takes total size of that storage drive . In case of Hadoop cluster we give storage of Slave Node by some folder or directory which by default shares all the storage amount of the drive inside which the folder is present. Now we use this command as to check hard disk or volume size in linux.

df –h

here we have 10 Gib space for / drive and my slave node is mounted with / drive. So total storage of / folder has mounted to our data node drive that is /dn1.

🔰 hadoop dfsadmin –report

3) using this we can see our total space of slave has been transferred to master.

hadoop dfsadmin –report

🔰 Attatch EBS volume to slave node

4) Now we have to find some way by which the we can control or limit these Slave Node storage in Hadoop cluster . For these we have to use the concept of disk partitions .So , Lets create one EBS Volume of some size and attach it to the Slave Node . now we created 2gb ebs storage and attached with slave.

🔰 fdisk -l

5) We can see all the disk partitions present in our Node using Linux Command :

fdisk -l

fdisk device_name

6) We have attached the EBS Volume /dev/xvdf of 2 GB to the Slave Node . To create the partitions we have command as :

fdisk  device_name

Now we just have to specify the amount or size of Last Sector of primary partition . In my case I have created the partition of size 1 GB .

Let's check the partition is successfully created or not . We see that one partition named as /dev/xvdf1 is created of size 1 GB.

🔰 mkfs.ext4 device_name

7) Now we have to format the partition . To format the partition we have command as :

mkfs.ext4  device_name

🔰 mount device_name directory_name

8) Now After formatting the partition we have to mount our Slave Node directory to the new partition we have created . To mount the partition we have command as :

🔰 hadoop dfsadmin -report

9) We can see that now we have control or Limit on Slave Node Storage as /dn1 Slave Node Directory is now access the storage that we have provided to it .

✨ ⏩ Now We see that now the Slave Node not takes the total storage of EBS Volume of size 2 GB as we have control or provided it specific amount 1GB of storage using the concept of disk partitions.

Thanks to Vimal Daga sir