Ceph

ceph自動化部署

Google+ Pinterest LinkedIn Tumblr

Ceph當前的自動化部署有兩個,分別是ceph-deploy和ceph-ansible,語言都是python,對應的github庫地址為:

ceph-deploy: https://github.com/ceph/ceph-deploy

ceph-ansible: https://github.com/ceph/ceph-ansible

ceph-deploy在手動部署中用的比較多,要實現自動化部署,必須再寫指令碼執行;

ceph-ansible基於ansible,可以實現根據配置檔案的ceph自動化部署;

我在之前寫過一個基於ceph-deploy部署ceph的自動化指令碼,用起來比較方便熟練;

所以這裏我寫的自動化部署是基於自寫指令碼呼叫ceph-deploy的自動化部署;

二、指令碼

這裏有兩個檔案:

[email protected]:~/ceph-deploy# ls
ceph-auto-deploy.py     deploy.conf

  • ceph-auto-deploy.py: 執行自動化部署的python指令碼
  • deploy.conf:ceph deploy的配置檔案,裏面包括了ceph cluster的配置資訊,ceph叢集的引數優化資訊等;

deploy.conf

該檔案裡需要使用者配置的資訊如下:(在檔案底部)

##################
## host-specific
## Put your settings for each node here, these settings will be removed
## from the ultimate ceph.conf
[host-specific]
mon_hosts = ceph0, ceph1, ceph2
osd_hosts = ceph0, ceph1
rgw_hosts = ceph0
 
# Example configuration use file system path
#osd.ceph0.paths = /var/lib/ceph/osd/ceph-0, /var/lib/ceph/osd/ceph-1
#osd.ceph1.paths = /var/lib/ceph/osd/ceph-2, /var/lib/ceph/osd/ceph-3
osd.ceph0.devs = vdb:vde, vdc:vde, vdd:vde
osd.ceph1.devs = vdb, vdc, vdd
 
## for multi network settings
## public network 1;cluster network1 = Host1, Host2, Host3
## public network 2;cluster network2 = Host4, Host5
192.10.4.0/24;192.10.4.0/24 = ceph0, ceph1, ceph2

指定Ceph元件部署在哪些Host上:

  • mon_hosts = ceph0, ceph1, ceph2
    • Ceph Monitors部署ceph0, ceph1, ceph2上
  • osd_hosts = ceph0, ceph1, ceph2
    • Ceph OSDs部署ceph0, ceph1上
  • rgw_hosts = ceph0
    • Ceph RadosGW部署ceph0上(若不配置RadosGW,可註釋掉改行)

指定每個部署OSDs的host上,哪些磁碟來啟動osd:(於上述配置osd_hosts對應)

  • osd.ceph0.devs = vdb:vde, vdc:vde, vdd:vde
    

    在ceph0上,啟動 vdb, vdc, vde三個osd,vde作為這三個osd的journal盤

  • osd.ceph1.devs = vdb, vdc, vdd
    

    在ceph1上,啟動 vdb, vdc, vde三個osd,每個盤會自動格式化一個分割槽作為journal

注: 在deploy.conf裡可配置journal size

ceph-auto-deploy.py

該檔案是自動化部署ceph的執行指令碼,支援的命令有:

[email protected]:~/ceph-deploy# python ceph-auto-deploy.py --help
ceph-auto-deploy.py usage:
    python ceph-auto-deploy.py     run this script with default configuration
    -h, --help                          print help message
    -v, --version                       print script version
    -c [configure file]                 specify the configure file
    -p                                  just purge older ceph and ceph data
    -r                                  do ceph purge and ceph reinstall
    --just-deploy-osds                  just do deploy ceph osds
    --debug                             just print out the commands

常用的命令有:

  • python ceph-auto-deploy.py
    根據deploy.conf的配置,部署ceph叢集
  • python ceph-auto-deploy.py -p
    根據deploy.conf的配置,刪除舊ceph系統和其中ceph相關資料
  • python ceph-auto-deploy.py -r
    根據deploy.conf的配置,執行purge後,重新安裝ceph系統相關packages(不會部署ceph叢集)

三、依賴

執行上述python自動化部署ceph叢集,依賴ceph-deploy,整體架構如下:

ceph自動化部署

其中ceph-deploy所在節點也可以是ceph的node節點。

所以有如下兩個依賴:

  1. admin-node上安裝ceph-deploy
  2. admin-node到ceph nodes的無密碼登陸

四、部署步驟

這裏以三個虛擬機器自動化部署ceph叢集為例,說明該自動化指令碼的使用:

三個節點分別為:

nodeHostnameNetworkDevices
node1ceph0192.10.4.109vdb, vdc, vdd, vde
node2ceph1192.10.4.110vdb, vdc, vdd, vde
node3ceph2192.10.4.111vdb, vdc, vdd, vde

每個node節點的系統資訊為:

[email protected]:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

確定nodes節點角色

首先我們需要確定使用哪個node作為admin-node,然後確定ceph叢集nodes節點都部署哪些元件,包括Monitor、OSD、RadosGW、MDS等;

根據ceph叢集部署需求,我們可以做如下node節點角色規劃:

nodeHostnameRoleceph NetworkCeph ComponentOSD/Journal Devices
node1ceph0admin-node ceph-node192.10.4.109Monitor, OSDOSD: vdb, vdc, vdd Journal: vde
node2ceph1ceph-node192.10.4.110Monitor, OSDOSD: vdb, vdc, vdd Journal為OSD盤的一個分割槽
node3ceph2ceph-node192.10.4.111Monitor不配置OSD

注:上述規劃與第二章節中的 deploy.conf 配置保持一致

admin-node節點安裝ceph-deploy

如前文所述,需要在選擇的admin-node上安裝自動化部署依賴的ceph-deploy,步驟如下:

[email protected]:~# apt-get install -y ceph-deploy
[email protected]:~# ceph-deploy --help
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
                   [--overwrite-conf] [--cluster NAME] [--ceph-conf CEPH_CONF]
                   COMMAND ...
 
Easy Ceph deployment
 
    -^-
   /   /
   |O o|  ceph-deploy v1.5.32
   ).-.(
  '/|||/`
  | '|` |
    '|`
 
Full documentation can be found at: http://ceph.com/ceph-deploy/docs
 
...

獲取pkg源的ceph版本資訊

我們當前推薦安裝Ceph的Jewel版本(10.2.*),在Ubuntu系統上,我們可以通過下面命令獲取其對應的Ceph pkgs版本資訊:

[email protected]:~# apt-get update
...
[email protected]:~# apt-cache show ceph
Package: ceph
Architecture: amd64
Version: 10.2.9-0ubuntu0.16.04.1
Priority: optional
Section: admin
Origin: Ubuntu

注:若對應Ceph不是Jewel版本,需要更換pkg源

admin-node節點配置無祕訪問ceph nodes

自動化部署指令碼會通過ssh自動在ceph nodes節點執行命令,ceph-deploy也依賴ssh的無祕訪問nodes節點,所以這裏我們要先配置這些:

  1. /etc/hosts上新增ceph nodes節點的hostname與ip的對應關係
vim /etc/hosts
...
192.10.4.109 ceph0
192.10.4.110 ceph1
192.10.4.111 ceph2

  1. 生成admin-node節點的ssh key
[email protected]:~# ssh-keygen
...
[email protected]:~# cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqIu+xLrVo0e0++z3i/zdAbZqqqWeyKshA47oUJzQneqD9HP39AJ8btGS5Pow4I/V/1zGsRJ2iV4EDxMymw2wwJ9LyxLl81WsfhSsuOVo7uvhlu3PiU6xDpr8UK66Sv7lquQ67yx9UnH9Vra2TQFsWPwbiPZm+N+qdXuXHeX4RVpOKlKmtEWN40Q3AZt3sbFe5hfAfP08v8XM70znFIEHKbhlN4XpLmuQxZ+vdxAY7kOA1EUugSZpc5nfc61SnCEzhI0stW+ccDsD/vmZsnCtITitS0YqGYJYvGHziKWfmZAw3ZtQd9CQIlZHnmqyA7vF4eoFSoj6YhkI4ozHzxLqt [email protected]

  1. 在其他節點填入ceph0的public key
[email protected]:~# cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqIu+xLrVo0e0++z3i/zdAbZqqqWeyKshA47oUJzQneqD9HP39AJ8btGS5Pow4I/V/1zGsRJ2iV4EDxMymw2wwJ9LyxLl81WsfhSsuOVo7uvhlu3PiU6xDpr8UK66Sv7lquQ67yx9UnH9Vra2TQFsWPwbiPZm+N+qdXuXHeX4RVpOKlKmtEWN40Q3AZt3sbFe5hfAfP08v8XM70znFIEHKbhlN4XpLmuQxZ+vdxAY7kOA1EUugSZpc5nfc61SnCEzhI0stW+ccDsD/vmZsnCtITitS0YqGYJYvGHziKWfmZAw3ZtQd9CQIlZHnmqyA7vF4eoFSoj6YhkI4ozHzxLqt [email protected]

注:因為ceph0既是admin-node,也是ceph node,所以也需要配置本身的ssh無祕訪問

admin-node節點執行自動化部署

之前的準備工作做完後,就可以通過自動化指令碼部署Ceph叢集了

  1. 獲取ceph deploy自動化指令碼

GitLab地址: https://github.com/ictfox/ceph-deploy

[email protected]:~# cd ceph-deploy
[email protected]:~/ceph-deploy# ls
ceph-auto-deploy.py     deploy.conf

  1. 根據第二章節介紹,配置 deploy.conf 檔案
[email protected]:~/ceph-deploy# vim deploy.conf
##################
## host-specific
## Put your settings for each node here, these settings will be removed
## from the ultimate ceph.conf
[host-specific]
mon_hosts = ceph0, ceph1, ceph2
osd_hosts = ceph0, ceph1
rgw_hosts = ceph0
 
# Example configuration use file system path
#osd.ceph0.paths = /var/lib/ceph/osd/ceph-0, /var/lib/ceph/osd/ceph-1
#osd.ceph1.paths = /var/lib/ceph/osd/ceph-2, /var/lib/ceph/osd/ceph-3
osd.ceph0.devs = vdb:vde, vdc:vde, vdd:vde
osd.ceph1.devs = vdb, vdc, vdd
 
## for multi network settings
## public network 1;cluster network1 = Host1, Host2, Host3
## public network 2;cluster network2 = Host4, Host5
192.10.4.0/24;192.10.4.0/24 = ceph0, ceph1, ceph2

  1. 安裝Ceph對應packages
[email protected]:~# python ceph-auto-deploy.py -r
...

這個會花費些時間,但每個Ceph node上安裝ceph相關pkgs是並行的,時間不會太長;

  1. 執行命令,自動化部署Ceph叢集
[email protected]:~# python ceph-auto-deploy.py
...

這個會花費較長時間,因為ceph-deploy部署osd時是序列的,爲了使得每個Host上的osd id是連續的,方便之後ceph叢集使用。

中間也可能出錯,錯誤資訊會輸出到螢幕上,詳細錯誤資訊也會輸出到同目錄下的檔案:ceph-deploy-ceph.log 裡,方便出錯時候檢視定位問題。

Write A Comment