AlmaLinux8.6上用cephadm搭建v17.2.0 Quincy版本ceph集群

一、初始化节点  
1.参考资料
https://docs.ceph.com/en/quincy/
https://github.com/ceph/ceph
https://github.com/ceph/ceph-container/tree/master/src/daemon
https://docs.ceph.com/docs/master/cephadm/
https://docs.ceph.com/docs/master/cephadm/install/

2.主机环境信息

3.初始化配置各个节点为模板机

#安装工具软件
yum install vim net-tools wget lsof python3 -y

#关闭防火墙和selinux
systemctl stop firewalld && systemctl disable firewalld
sed -i '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0


#配置主机地址解析
if [ -z "`cat /etc/hosts | grep \`ip route ls | grep 192.168.3.0/24 | awk '{print $9}'\` | awk '{print $2}'`" ]; then
cat << EOF >> /etc/hosts
192.168.3.51 ceph01
192.168.3.52 ceph02
192.168.3.53 ceph03
192.168.3.54 ceph04
192.168.3.55 ceph05
192.168.3.56 ceph06
EOF
fi

#根据IP地址获取主机名并写入hostname
echo `cat /etc/hosts | grep \`ip route ls | grep 192.168.3.0/24 | awk '{print $9}'\` | awk '{print $2}'` >/etc/hostname

#重新登录终端立即生效
hostnamectl set-hostname `cat /etc/hosts | grep \`ip route ls | grep 192.168.3.0/24 | awk '{print $9}'\` | awk '{print $2}'`

# 内核参数设置：开启IP转发，允许iptables对bridge的数据进行处理
cat << EOF > /etc/sysctl.d/ceph.conf 
kernel.pid_max = 4194303
vm.swappiness = 0
EOF

# 立即生效
sysctl --system

#设置时区为东八区
rm -f /etc/localtime
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

#配置集群时间同步
yum install -y chrony
cat > /etc/chrony.conf << EOF
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
server ntp1.aliyun.com iburst
local stratum 10
allow 192.168.3.0/24
EOF
systemctl restart chronyd
systemctl enable chronyd

4.在所有节点安装配置docker-ce

#安装部署docker
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce
systemctl enable docker && systemctl start docker

二、初始化cephadm环境
1.获取quincy版本cephadm脚本

curl -O https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm

2.授予执行权限

[root@cephadm ~]# chmod +x cephadm 

3.基于发行版的名称配置ceph仓库

./cephadm add-repo --release quincy
./cephadm install

#查看安装情况
[root@cephadm ~]# which cephadm
/usr/sbin/cephadm

#安装ceph-common ceph-fuse
cephadm install ceph-common ceph-fuse

三、部署MON
1.在ceph01上执行部署命令

[root@ceph01 ~]# cephadm bootstrap --mon-ip 192.168.3.51

上述指令会为我们完成以下工作：

• 创建mon
• 创建ssh key并且添加到 /root/.ssh/authorized_keys 文件
• 将集群间通信的最小配置写入/etc/ceph/ceph.conf
• 将client.admin管理secret密钥的副本写入/etc/ceph/ceph.client.admin.keyring。
• 将公用密钥的副本写入/etc/ceph/ceph.pub

2.执行结果如下：

[root@ceph01 ~]# cephadm bootstrap --mon-ip 192.168.3.51
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 2765f0bc-e461-11ec-8881-525400062554
Verifying IP 192.168.3.51 port 3300 ...
Verifying IP 192.168.3.51 port 6789 ...
Mon IP `192.168.3.51` is in CIDR network `192.168.3.0/24`
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image quay.io/ceph/ceph:v17...
Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 192.168.3.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ceph01...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:

             URL: https://ceph01:8443/
            User: admin
        Password: e0jg4mikb4

Enabling client.admin keyring and conf on hosts with "admin" label
Enabling autotune for osd_memory_target
You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid 2765f0bc-e461-11ec-8881-525400062554 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/master/mgr/telemetry/

Bootstrap complete.

  
3.cephadm shell命令在装有所有Ceph软件包的容器中启动bash shell。默认情况下，如果在主机上的/etc/ceph中找到配置文件和keyring文件，它们将被传递到容器环境中，从而使Shell可以正常运行。 若是在MON主机上执行时，cephadm Shell将使用MON容器的配置，而不是使用默认配置。 如果给出了–mount ，则主机 （文件或目录）将出现在容器内的/mnt下

#为了方便使用，可以给cephadm shell换个名字
[root@cephadm ~]# alias ceph='cephadm shell -- ceph'

想要永久生效可以编辑/etc/.bashrc
[root@cephadm ~]# echo "alias ceph='cephadm shell -- ceph'" >>/root/.bashrc

4.通过https://192.168.3.51:8443/访问集群，修改默认密码

四、添加新的节点到集群
1.如果要添加新的节点到集群，要将ssh 公钥推送到新的节点authorized_keys文件中

[root@ceph01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph02
[root@ceph01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph03
[root@ceph01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph04
[root@ceph01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph05
[root@ceph01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph06

2.告诉Ce