Gluster FS

Install And Peer GlusterFS

Operating System : ubuntu 20.04

Configure Firewall

For the Gluster to communicate within a cluster either the firewalls have to be turned off or enable communication for each server.

iptables -I INPUT -p all -s `<ip-address>` -j ACCEPT

Install software-properties-common:

apt install software-properties-common

Then add the community GlusterFS PPA:

add-apt-repository ppa:gluster/glusterfs-7

Update ubuntu repo

sudo apt update

Finally, install the packages:

enable glusterfs daemon on startup

systemctl enable glusterd.service

Check glusterfs daemon

systemctl status glusterd.service

Peer Gluster server node

sudo gluster peer probe {{ip_address}}

check peer status

sudo gluster peer status

Partition the disk

Assuming you have an empty disk at /dev/sdb: (You can check the partitions on your system using fdisk -l)

fdisk /dev/sdb

And then create a single XFS partition using fdisk

Format the partition

mkfs.xfs -i size=512 /dev/sdb1

Add an entry to /etc/fstab

echo "/dev/sdb1 /export/sdb1 xfs defaults 0 0"  >> /etc/fstab

advance configuration

Gluster FS Configuration

Configure the trusted pool

Remember that the trusted pool is the term used to define a cluster of nodes in Gluster.

gluster peer probe {{ip}}

check status

gluster peer status

Partition the disk

Assuming you have an empty disk at /dev/sdb: (You can check the partitions on your system using fdisk -l*)*

fdisk /dev/sdb

And then create a single XFS partition using fdisk

Format the partition

mkfs.xfs -i size=512 /dev/sdb1

Add an entry to /etc/fstab

echo “/dev/sdb1 /export/sdb1 xfs defaults 0 0” >> /etc/fstab

Mount the partition as a Gluster “brick”

mkdir -p /export/sdb1 && mount -a

Set up a Gluster volume

The most basic Gluster volume type is a “Distribute only” volume (also referred to as a “pure DHT” volume if you want to impress the folks at the water cooler). This type of volume simply distributes the data evenly across the available bricks in a volume. So, if I write 100 files, on average, fifty will end up on one server, and fifty will end up on another. This is faster than a “replicated” volume, but isn’t as popular since it doesn’t give you two of the most sought after features of Gluster — multiple copies of the data, and automatic failover if something goes wrong.

To set up a replicated volume:

gluster volume create gv0 replica 3 node01.mydomain.net:/export/sdb1/brick \\

node02.mydomain.net:/export/sdb1/brick \\

node03.mydomain.net:/export/sdb1/brick

Breaking this down into pieces:

• the first part says to create a gluster volume named gv0 (the name is arbitrary, gv0 was chosen simply because it’s less typing than gluster_volume_0).
• make the volume a replica volume
• keep a copy of the data on at least 3 bricks at any given time. Since we only have three bricks total, this means each server will house a copy of the data.
• we specify which nodes to use, and which bricks on those nodes. The order here is important when you have more bricks.
• the brick directory will be created by this command. If the directory already exists, you may get <brick> is already part of a volume errors.

It is possible (as of the most current release as of this writing, Gluster 3.3) to specify the bricks in such a way that you would make both copies of the data reside on a single node. This would make for an embarrassing explanation to your boss when your bulletproof, completely redundant, always on super cluster comes to a grinding halt when a single point of failure occurs.

Get Andre’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Now, we can check to make sure things are working as expected:

gluster volume info

And you should see results similar to the following:

Volume Name: gv0

Type: Replicate

Volume ID: 8bc3e96b-a1b6–457d-8f7a-a91d1d4dc019

Status: Created

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: node01.yourdomain.net:/export/sdb1/brick

Brick2: node02.yourdomain.net:/export/sdb1/brick

Brick3: node03.yourdomain.net:/export/sdb1/brick

This shows us essentially what we just specified during the volume creation. The one key output worth noticing is Status. A status of Created means that the volume has been created, but hasn’t yet been started, which would cause any attempt to mount the volume fail.

Now, we should start the volume before we try to mount it.

gluster volume start gv0

Types of Volumes

1.Distributed Glusterfs Volume — This is the type of volume which is created by default if no volume type is specified. Here, files are distributed across various bricks in the volume.

Create a Distributed Volume

gluster volume create NEW-VOLNAME [transport [tcp | rdma | tcp,rdma]] NEW-BRICK…

For example to create a distributed volume with four storage servers using TCP.

gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4

volume create: test-volume: success: please start the volume to access data

To display the volume info:

gluster volume info

Volume Name: test-volume

Type: Distribute

Status: Created

Number of Bricks: 4

Transport-type: tcp

Bricks:

Brick1: server1:/exp1

Brick2: server2:/exp2

Brick3: server3:/exp3

Brick4: server4:/exp4

Replicated Glusterfs Volume — In this volume we overcome the risk of data loss which is present in the distributed volume.

Create a Replicated Volume

gluster volume create NEW-VOLNAME [replica COUNT] [transport [tcp |rdma | tcp,rdma]] NEW-BRICK…

For example, to create a replicated volume with three storage servers:

gluster volume create test-volume replica 3 transport tcp \\

server1:/exp1 server2:/exp2 server3:/exp3

volume create: test-volume: success: please start the volume to access data

Distributed Replicated Glusterfs Volume — In this volume files are distributed across replicated sets of bricks.

Create the distributed replicated volume:

gluster volume create NEW-VOLNAME [replica COUNT] [transport [tcp | rdma | tcp,rdma]] NEW-BRICK…

For example, six node distributed replicated volume with a three-way mirror:

gluster volume create test-volume replica 3 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6

volume create: test-volume: success: please start the volume to access data

Dispersed Glusterfs Volume — Dispersed volumes are based on erasure codes. It stripes the encoded data of files, with some redundancy added, across multiple bricks in the volume. You can use dispersed volumes to have a configurable level of reliability with minimum space waste.

Create a dispersed volume:

gluster volume create test-volume [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport tcp | rdma | tcp,rdma] <NEW-BRICK>

For example, three node dispersed volume with level of redundancy 1, (2 + 1):

gluster volume create test-volume disperse 3 redundancy 1 server1:/exp1 server2:/exp2 server3:/exp3

volume create: test-volume: success: please start the volume to access data

Distributed Dispersed Glusterfs Volume — Distributed dispersed volumes are the equivalent to distributed replicated volumes, but using dispersed subvolumes instead of replicated ones.

Create a distributed dispersed volume:

gluster volume create [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport tcp | rdma | tcp,rdma] <NEW-BRICK>

For example, six node distributed dispersed volume with level of redundancy 1, 2 x (2 + 1) = 6:

gluster volume create test-volume disperse 3 redundancy 1 server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6

volume create: test-volume: success: please start the volume to access data