centos7.7源码安装ceph最新版+rdma

最新推荐文章于 2026-05-08 16:14:38 发布

原创最新推荐文章于 2026-05-08 16:14:38 发布 · 1.5w 阅读

6 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#ceph #rdma

ceph 专栏收录该内容

19 篇文章

订阅专栏

本文详细介绍如何通过源码编译的方式，在Ceph集群中启用RDMA协议，以提升存储性能。包括环境准备、基本库及依赖库安装、编译配置、系统服务与命令预设等关键步骤。

需求：为提升ceph性能，采用infiniband 网卡 rdma协议，ceph-deploy安装方式是不支持rdma协议的，故只能采用源码安装，把rdma参数编译进去。

环境准备

IP	host	IOS	kernel
192.168.1.82	host82	centos7.7	4.19.6-1.el7.elrepo.x86_64
192.168.1.83	host83	centos7.7	4.19.6-1.el7.elrepo.x86_64
192.168.1.84	host84	centos7.7	4.19.6-1.el7.elrepo.x86_64

基本库准备

1. 安装CMake-v3.12.4(这一步貌似ceph源码包编译可以自动执行）

$: wget https://github.com/Kitware/CMake/archive/v3.12.4.tar.gz
$:tar zxf v3.12.4.tar.gz -C /usr/local/
$: cd /usr/local/CMake-3.12.4
$: ./bootstrap 
$: make && make install
$: cmake --version

2.安装依赖库

$:yum install -y libffi-devel boost cryptopp-devel fcgi-devel libatomic_op-devel libedit-devel boost-devel fuse-devel libtool libuuid-devel libblkid-devel keyutils-libs-devel cryptopp-devel fcgi-devel libcurl-devel expat-devel gperftools-devel libedit-devel libatomic_ops-devel snappy-devel libaio-devel xfsprogs-devel libudev-devel libcrypto++-dev libcrypto++-utils

$:yum install -y fuse fuse-devel

3.安装leveldb-1.20（centos7.7版本较低需要升级）

下载地址：

https://github.com/google/leveldb/releases（wget貌似下载失败，可用浏览器下载）
$: tar -zxvf leveldb-1.20.tar.gz -C /usr/local/
$: cd /usr/local/leveldb-1.20
$: make

需要替换原系统库
$:unalias cp
$:cd /usr/local/levedb-1.20/out-shared/
$:cp libleveldb.so* /usr/lib64/
$:cd /usr/lib64
$:rm libleveldb.so libleveldb.so.1
$:ln -s libleveldb.so.1.20 libleveldb.so
$:ln -s libleveldb.so.1.20 libleveldb.so.1
$: cd /usr/local/leveldb-1.20/include
$: cp -r leveldb/ /usr/include/

4.安装gcc-8.2.0（这一步貌似ceph git源码包编译也会自动执行）

$: wget https://mirrors.ustc.edu.cn/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.gz
$: tar zxf gcc-8.2.0.tar.gz
$: cd gcc-8.2.0
$: ./contrib/download_prerequisites
$: mkdir gcc-build-8.2.0
$: cd gcc-build-8.2.0
$: ../configure --enable-checking=release --enable-languages=c,c++ --disable-multilib
      参数解释：
      –enable-checking=release 增加一些检查，也可以–disable-checking生成的编译器在编译过程中不做额外检查
      –enable-languages=c,c++ 你要让你的gcc支持的编程语言
      –disable-multilib 取消多目标库编译(取消32位库编译)
$: make && make install

替换gcc的动态库

$: cp /usr/local/lib64/libstdc++.so.6.0.25 /usr/lib64
$: rm -rf /usr/lib64/libstdc++.so.6
$: ln -s /usr/lib64/libstdc++.so.6.0.25 /usr/lib64/libstdc++.so.6
$: rm -rf /usr/bin/gcc
$: rm -rf /usr/bin/g++
$: rm -rf /usr/bin/c++
$: rm -rf /usr/bin/cc
$:ln -s /usr/local/bin/gcc /usr/bin/gcc
$:ln -s /usr/local/bin/gcc /usr/bin/cc
$:ln -s /usr/local/bin/g++ /usr/bin/g++
$:ln -s /usr/local/bin/c++ /usr/bin/c++

ceph最新版源码编译

$:git clone https://github.com/ceph/ceph.git
$:cd ceph
$:./install-deps.sh

下面步骤的参数为必须

$:./do_cmake.sh  -DWITH_RDMA=ON  -DCMAKE_BUILD_TYPE=RelWithDebInfo

编译安装

 $: cd build
 $: make && make install DESTDIR=安装路径

检查

[root@node82 ~]# ceph -v
ceph version 15.0.0-7335-g310a2d3 (310a2d37be135b5865536cc02cbdb2e65f85953d) octopus (dev)

尝试使用ceph-deploy部署集群(实验单节点)

1.系统服务预准备

$:yum -y install ceph					##必要步骤，可参考官网yum源，最新版即可
$:cp /data/ceph/systemd/* /usr/lib/systemd/system/
$:sed -i '/Service]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/ceph-mon@.service
$:sed -i  's/PrivateDevices=yes/PrivateDevices=no/' /usr/lib/systemd/system/ceph-mon@.service
$:sed -i '/Service]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/ceph-mds@.service
$:sed -i  's/PrivateDevices=yes/PrivateDevices=no/' /usr/lib/systemd/system/ceph-mds@.service
$:sed -i '/Service]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/ceph-radosgw@.service
$:sed -i 's/PrivateDevices=yes/PrivateDevices=no/'  /usr/lib/systemd/system/ceph-radosgw@.service
$:sed -i '/Service]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/ceph-mgr@.service
$:sed -i '/Service]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/ceph-osd@.service
$:systemctl daemon-reload 
另外系统服务的命令统一改成ExecStart=/data/ceph/ceph/build/bin/ceph-* -f --cluster ${CLUSTER} --id %i --setuser root --setgroup root

2.系统命令预准备

$:cat /etc/profile

export PATH=/data/ceph/ceph/build/bin:$PATH
export PYTHONPATH=/data/ceph/ceph/src/pybind:/data/ceph/ceph/build/lib/cython_modules/lib.2:/data/ceph/ceph/src/python-common:$PYTHONPATH
export LD_LIBRARY_PATH=/data/ceph/ceph/build/lib:$LD_LIBRARY_PATH
alias cephfs-shell=/data/ceph/ceph/src/tools/cephfs/cephfs-shell

$:cat /etc/ld.so.conf

include ld.so.conf.d/*.conf
/data/ceph_install/lib
/data/ceph_install/lib64

$:ldconfig
注意：
ceph编译安装没有ceph-volume命令，故需要想办法拿到ceph-volume命令

$: mv /bin/ceph-volume /data/ceph/ceph/build/bin/
$: rm -rf /bin/ceph*
$: rm -rf /usr/sbin/ceph*
$: ln -s /data/ceph/ceph/build/bin/* /bin/
$: ln -s /data/ceph/ceph/build/bin/* /usr/sbin/

3.ceph-deploy 安装monitor

$:mkdir ceph-cluster&&cd ceph-cluster
$:ceph-deploy new node84

$:cat ceph.conf

[global]
fsid = 2e82e49c-57d1-4f20-bb24-244cab262383
mon_initial_members = node84
mon_host = 172.10.1.84
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public network = 172.10.0.0/16
cluster network = 172.10.0.0/16

ms_type=async+rdma
ms_async_rdma_device_name=mlx4_0
ms_async_rdma_port_num = 1
ms_cluster_type = async+rdma

开始生成monitor

$:ceph-deploy --overwrite-conf mon create-initial
$:ceph-deploy admin node84

检查：

$:ceph -s
  cluster:
    id:     2e82e49c-57d1-4f20-bb24-244cab262383
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum node84 (age 4h)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

4.ceph-deploy安装mgr

拷贝编译包到别的节点会出现很多Python module找不到,可如下执行

$:cd /data/ceph/ceph/
$:./install-deps.sh

采用系统服务起mgr组件还是会出问题，待解决
但是命令方式是没有问题的

$;/data/ceph/ceph/build/bin/ceph-mgr  -f --cluster ceph --id node83 --setuser root --setgroup root

5.别的组件

其他组件经测试是没有问题的，可以部署成功

[参考文档]

https://docs.ceph.com/docs/master/
https://github.com/ceph/ceph
https://www.hwchiu.com/ceph-with-rdma.html
https://community.mellanox.com/s/article/howto-configure-ceph-rdma--outdated-x
https://blog.csdn.net/smart9527_zc/article/details/84894749