本文共 3987 字，大约阅读时间需要 13 分钟。
一、基本环境介绍：
本文采用ceph-deploy工具进行ceph安装，ceph-deploy可以单独作为一个admin节点，也可以装在任意node节点上。
系统环境如下：
1、 系统采用redhat-6.5x86_64 basic_server安装，总共3个节点,ntp时间同步
2、 关闭selinux，用epel及ceph官方源，版本为0.86
3、 3个节点已做互信，并配置host，每个节点上有3个磁盘做osd
4、 升级内核为3.18，并从新内核启动。
二、安装步骤：
1、 每个node上设置iptables规则或关闭iptables(eth0指ceph网络所在的网卡名称)
| 1 2 3 | iptables -A INPUT -i eth0 -p tcp -s0.0.0.0 /0 --dport 6789 -j ACCEPT iptables -A INPUT -i eth0 -m multiport -ptcp -s 0.0.0.0 /0 --dports 6800:7300 -j ACCEPT service iptables save |
2、 格式化、挂载osd
| 1 2 3 4 5 | yum -y install xfsprogs mkfs.xfs /dev/sdb mkdir /osd {0..2} #blkid 查看sdb的UUID echo 'UUID=89048e27-ff01-4365-a103-22e95fb2cc93/osd0 xfs noatime,nobarrier,nodiratime 00' >> /etc/fstab |
一个磁盘对应一个osd，每个节点都创建osd0、osd1、osd2目录，对应的磁盘挂载相应目录即可。
3、安装ceph部署工具
| 1 2 3 | #mkdir ceph #最好创建一个目录，因为安装ceph过程中，会在安装目录生成一些文件 #cd ceph #yum -y install ceph-deploy |
4、创建mon
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ceph-deploy new node1 nod2 node3 #这个命令其实就是仅仅生成了ceph.conf和ceph.mon.keyring 文件 vim ceph.conf 追加以下内容(根据需求更改) debug_ms = 0 mon_clock_drift_allowed = 1 osd_pool_default_size = 2 #副本数量 osd_pool_default_min_size = 1 osd_pool_default_pg_num = 128 #pg数量 osd_pool_default_pgp_num = 128 osd_crush_chooseleaf_type = 0 debug_auth = 0 /0 debug_optracker = 0 /0 debug_monc = 0 /0 debug_crush = 0 /0 debug_buffer = 0 /0 debug_tp = 0 /0 debug_journaler = 0 /0 debug_journal = 0 /0 debug_lockdep = 0 /0 debug_objclass = 0 /0 debug_perfcounter = 0 /0 debug_timer = 0 /0 debug_filestore = 0 /0 debug_context = 0 /0 debug_finisher = 0 /0 debug_heartbeatmap = 0 /0 debug_asok = 0 /0 debug_throttle = 0 /0 debug_osd = 0 /0 debug_rgw = 0 /0 debug_mon = 0 /0 osd_max_backfills = 4 filestore_split_multiple = 8 filestore_fd_cache_size = 1024 filestore_queue_committing_max_bytes =1048576000 filestore_queue_max_ops = 500000 filestore_queue_max_bytes = 1048576000 filestore_queue_committing_max_ops = 500000 osd_max_pg_log_entries = 100000 osd_mon_heartbeat_interval = 30 #Performance tuning filestore osd_mount_options_xfs =rw,noatime,logbsize=256k,delaylog #osd_journal_size = 20480 日志大小，不指定，默认是5G osd_op_log_threshold = 50 osd_min_pg_log_entries = 30000 osd_recovery_op_priority = 1 osd_mkfs_options_xfs = -f -i size=2048 osd_mkfs_type = xfs osd_journal = /var/lib/ceph/osd/ $cluster-$ id /journal journal_queue_max_ops = 500000 journal_max_write_bytes = 1048576000 journal_max_write_entries = 100000 journal_queue_max_bytes = 1048576000 objecter_infilght_op_bytes = 1048576000 objecter_inflight_ops = 819200 ms_dispatch_throttle_bytes = 1048576000 sd_data = /var/lib/ceph/osd/ $cluster-$ id merge_threshold = 40 backfills = 1 mon_osd_min_down_reporters = 13 mon_osd_down_out_interval = 600 rbd_cache_max_dirty_object = 0 rbd_cache_target_dirty = 235544320 rbd_cache_writethrough_until_flush = false rbd_cache_size = 335544320 rbd_cache_max_dirty = 335544320 rbd_cache_max_dirty_age = 60 rbd_cache = false |
5、安装ceph
#On All nodes to Install
| 1 | yum -y install ceph |
admin节点上执行：
| 1 2 | ceph-deploy mon create node1 node2 node3 ceph-deploy gatherkeys node1 #从moniter节点获得keys，用来管理节点 |
6、创建激活osd
| 1 2 3 4 5 | ceph-deploy osd prepare node1: /osd0node1 : /osd1 node1: /osd2 node2: /osd0 node2: /osd1 node2: /osd2 node3: /osd0node3 : /osd1 node3: /osd2 ceph-deploy osd activate node1: /osd0node1 : /osd1 node1: /osd2 node2: /osd0 node2: /osd1 node2: /osd2 node3: /osd0node3 : /osd1 node3: /osd2 ceph-deploy admin node1 node2 node3 #从admin节点复制配置文件及key到node chmod +r /etc/ceph/ceph .client.admin.keyring (所有节点为该文件添加读权限) |
也可以创建相关的pool
| 1 2 3 4 | ceph osd pool create volumes 128 ceph osd pool create images 128 ceph osd pool create vms 128 ceph osd pool create backups 128 |
如果ceph安装失败，要清除ceph环境，执行以下命令
| 1 2 3 | ceph-deploy purge node1 node2 node3 (所有节点) ceph-deploy purgedata node1 node2 node3 (所有节点) ceph-deploy forgetkeys |