这是一篇 2019 年的生产环境实操记录:某云服务商要把它基于 CloudStack 的存量业务迁到 OpenStack 上统一运维。好在两边虚拟机的磁盘本来就落在同一套 Ceph 集群里,迁移可以只在 RBD 层做,数据不过网线。原文是当年的工作笔记,发布前修复了编辑器硬换行造成的断行、隐去了客户环境信息与口令,个别笔误按上下文修正,操作步骤与脚本逻辑保持原样。
| 角色 | 版本 / 规格 |
|---|---|
| 存量 Ceph | Ubuntu 14.04 · Ceph Jewel 10.2.11 · ceph-deploy 部署 · 5 monitor + 11 OSD 节点 |
| 新建 OpenStack | CentOS 7.4 · Queens · Kolla-Ansible 容器化 · 3 控制节点 + 1 计算节点起步 |
| 存储对接 | OpenStack 三组件(glance / cinder / nova)全部对接存量 Ceph,作为 client 端使用 |
| 迁移对象 | CloudStack 上存于 Ceph 的 KVM 虚机:系统盘 + 数据盘,按宿主机批量接管 |
整体思路三步走:先给 Ceph 分离出 cluster network(生产隔离);再部署 OpenStack 并把 glance / cinder / nova 三个后端全部指向这套存量 Ceph;最后写迁移脚本,把 CloudStack 虚机的 RBD 镜像原位「搬」进 OpenStack 的卷和虚机里。因为磁盘不挪窝,迁移窗口可以压到每台虚机几分钟。
操作系统 Ubuntu 14.04,Ceph Jewel 10.2.11,ceph-deploy 部署,5 个 monitor 节点、11 台 OSD 节点。
所有 Ceph 节点(mon / osd / client)的 /etc/ceph/ceph.conf 增加集群网络段:
vim /etc/ceph/ceph.conf
[default]
......
public network = 10.78.0.0/16
cluster network = 10.100.4.0/24
.......
mon 与 osd 节点全部重启 Ceph 服务生效:
restart ceph-all
# 或者分别重启
restart ceph-mon-all
restart ceph-osd-all
CentOS 7.4,OpenStack Queens,Kolla 容器化部署,使用外接 Ceph。controller / compute 节点全部作为 Ceph 的 client 端,不部署存储组件。
清掉默认源,换清华镜像(base / updates / extras / centosplus)加 EPEL,Ceph 客户端源用阿里云 Jewel 通道:
cd /etc/yum.repos.d/
rm -rf *
vim CentOS-Base.repo
[base]
name=CentOS-$releasever - Base
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
[updates]
name=CentOS-$releasever - Updates
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
[extras]
name=CentOS-$releasever - Extras
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
[centosplus]
name=CentOS-$releasever - Plus
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/centosplus/$basearch/
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
vim epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
vim ceph.repo
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/
gpgcheck=0
enabled=1
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/
gpgcheck=0
enabled=1
yum clean all
yum makecache fast
在部署节点配好 hosts 与免密,用 ceph-deploy 给 OpenStack 节点装客户端并下发配置:
vim /etc/hosts
# controller
10.78.0.11 controller1
10.78.0.12 controller2
10.78.0.13 controller3
# compute
10.78.0.14 compute01
.....
ssh-copy-id controller1
ssh-copy-id controller2
ssh-copy-id controller3
ssh-copy-id compute01
ceph-deploy install controller1 controller2 controller3 compute01
ceph-deploy admin controller1 controller2 controller3 compute01
# 验证
ceph -s
# 所有节点:关闭防火墙与 selinux
systemctl stop firewalld.service
systemctl disable firewalld.service
vim /etc/selinux/config
SELINUX=disabled
# 设置 hostname
hostnamectl set-hostname $HOSTNAME
# 关闭 NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager
reboot
# pip 走内网镜像
yum install python2-pip
cat > /etc/pip.conf <<'EOF'
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host = mirrors.aliyun.com
EOF
pip install -U pip
yum install -y python-devel libffi-devel gcc openssl-devel libselinux-python git
pip install -U 'ansible>=2.2.0'
# docker-ce
yum remove docker docker-common docker-selinux docker-engine
yum install -y yum-utils device-mapper-persistent-data lvm2
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum clean all
yum makecache fast
yum install docker-ce
# 配置 docker 镜像加速器
mkdir /etc/docker/
cat > /etc/docker/daemon.json <<'EOF'
{
"registry-mirrors": ["https://<你的加速器地址>"]
}
EOF
systemctl daemon-reload
systemctl start docker-ce
# 搭私有 docker registry(Kolla 镜像已提前编译好推进去)
docker run -d -v /opt/registry/:/var/lib/registry/ -p 4000:5000 --name=registry registry
# kolla-ansible 源码已上传到部署机 /root/ 目录
cd kolla-ansible
pip install -r test-requirements.txt -r requirements.txt
python setup.py install
cp -rv ./etc/kolla/ /etc/
mkdir /etc/kolla/config
kolla-genpwd
vim /etc/kolla/passwords.yml
keystone_admin_password: <自定义强口令>
ceph osd pool create volumes 128
ceph osd pool set volumes size 3
ceph osd pool create vms 128
ceph osd pool set vms size 3
ceph osd pool create images 64
ceph osd pool set images size 3
ceph osd pool create backups 64
ceph osd pool set backups size 3
vim /etc/kolla/globals.yml
enable_ceph: "no"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
cinder_backup_driver: "ceph"
nova_backend_ceph: "yes"
mkdir /etc/kolla/config/{glance,cinder/{cinder-volume,cinder-backup},nova}
vim /etc/kolla/config/glance/glance-api.conf
[glance_store]
stores = rbd
default_store = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
# 集群配置与 keyring 放进组件目录,Kolla 起容器时挂进去
cp /etc/ceph/ceph.conf /etc/kolla/config/glance/ceph.conf
ceph auth get-or-create client.glance mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' \
-O ceph.client.glance.keyring
cp ceph.client.glance.keyring /etc/kolla/config/glance/
vim /etc/kolla/config/cinder/cinder-volume.conf
[DEFAULT]
enabled_backends = rbd-1
[rbd-1]
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
backend_host = rbd:volumes
rbd_pool = volumes
volume_backend_name = rbd-1
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
vim /etc/kolla/config/cinder/cinder-backup.conf
[DEFAULT]
backup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_user = cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_pool = backups
backup_driver = cinder.backup.drivers.ceph
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true
cp /etc/ceph/ceph.conf /etc/kolla/config/cinder
# cinder keyring:卷 + 虚机盘读写,镜像池只读
ceph auth get-or-create client.cinder mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images' \
-O ceph.client.cinder.keyring
# cinder-backup 需要两个 keyring:分别连 volumes 与 backups pool
ceph auth get-or-create client.cinder-backup mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups' \
-O ceph.client.cinder-backup.keyring
cp ceph.client.cinder-backup.keyring /etc/kolla/config/cinder/cinder-backup/
cp ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/
cp ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-volume/
vim /etc/kolla/config/nova/nova-compute.conf
[libvirt]
images_rbd_pool = vms
images_type = rbd
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
ceph auth get-or-create client.nova mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms' \
-O ceph.client.nova.keyring
cp /etc/ceph/ceph.conf /etc/kolla/config/nova
cp ceph.client.nova.keyring /etc/kolla/config/nova/
cp ceph.client.cinder.keyring /etc/kolla/config/nova
cp /root/kolla-ansible/ansible/inventory/multinode /root/
vim multinode
# 按实际节点规划修改 [control] / [compute] / [storage] 分组
关键点:外接 Ceph 意味着集群里没有存储节点,而默认 cinder-volume 和 cinder-backup 容器跑在 storage 组上——必须在 multinode 里显式指定一台主机去运行这两个容器:
vim multinode
[storage]
compute01
kolla-ansible -i /root/multinode bootstrap-servers
kolla-ansible -i /root/multinode prechecks
kolla-ansible -i /root/multinode deploy
kolla-ansible -i /root/multinode post-deploy
cp /etc/kolla/admin-openrc.sh /root/
source admin-openrc.sh
cd /root/kolla-ansible/tools
vim init-runonce
EXT_NET_CIDR='10.0.2.0/24' # 外部网络
EXT_NET_RANGE='start=10.0.2.150,end=10.0.2.199' # 浮动 IP 池
EXT_NET_GATEWAY='10.0.2.1' # 外部网关
/bin/bash init-runonce
# 验证
openstack network list
openstack network agent list
openstack compute service list
openstack image list
CloudStack 和 OpenStack 的磁盘都在同一套 Ceph 上,所以不需要任何「跨云传输」:
/data/hosts/<宿主机名>/ 下;rbd export 导出、按大小在 OpenStack 建卷(--bootable)、rbd rm 删掉卷占位镜像后 rbd import 原位灌入;openstack server create 起虚机,再挂数据盘、绑浮动 IP;#!/bin/bash
# 用法:/bin/bash -x migrate.sh <宿主机名> <新pool名>
# 实测:CentOS 虚机迁移后网络正常;Debian 虚机存在无法 ping 通的情况(见文末经验教训)
hostname=$1
rm -rf /data/hosts/$hostname/backup*
vm_config_list=(`ls /data/hosts/$hostname`)
new_pool=$2
for vm_config in ${vm_config_list[@]}
do
vm_name=`echo $vm_config | cut -d '.' -f 1`
echo $vm_name
# 从 libvirt XML 解析系统盘 RBD 镜像名与所在 pool
vm_rbd=`grep rbd /data/hosts/$hostname/$vm_config | awk '{print $3}' \
| awk -F '=|/' '{print $3}' | awk -F"'" '{print $1}' | head -1`
vm_rbd_pool=`grep rbd /data/hosts/$hostname/$vm_config | awk '{print $3}' \
| awk -F "='|/" '{print $2}' | head -1`
old_pool=$vm_rbd_pool
# 盘大小换算成 GB(OpenStack 卷最小 1GB)
vm_rbd_size=`rbd info $old_pool/$vm_rbd | head -2 | tail -1 | awk '{print $2}'`
vm_rbd_unit=`rbd info $old_pool/$vm_rbd | head -2 | tail -1 | awk '{print $3}'`
echo $vm_rbd_unit
if [ "$vm_rbd_unit"x = "GB"x ]; then
vm_rbd_size_GB=$vm_rbd_size
else
if [ $vm_rbd_size -le 1024 ]; then
vm_rbd_size_GB=1
else
vm_rbd_size_GB=`echo $vm_rbd_size/1024 | bc`
fi
fi
# 导出系统盘
backup_vm_rbd=/data/hosts/$hostname/backup.$vm_rbd
rbd export -p $old_pool $vm_rbd $backup_vm_rbd
# 建 bootable 卷
new_vm_rbd=$hostname-$vm_rbd
openstack volume create $new_vm_rbd --size $vm_rbd_size_GB --bootable
vm_rbd_boot_uuid=`openstack volume list | grep $new_vm_rbd | awk '{print $2}'`
echo $vm_rbd_boot_uuid
rbd rm -p $new_pool volume-$vm_rbd_boot_uuid
# 灌入系统盘
rbd import -p $new_pool $backup_vm_rbd volume-$vm_rbd_boot_uuid
rm -rf /data/hosts/$hostname/backup*
# 按原虚机规格重建 flavor
vm_memory_KB=`grep "memory unit" /data/hosts/$hostname/$vm_config | awk -F '>|<' '{print $3}'`
vm_memory_MB=`echo $vm_memory_KB/1024 | bc`
vm_vcpus=`grep "vcpu" /data/hosts/$hostname/$vm_config | tail -1 | awk -F '>|<' '{print $3}'`
new_vm_flavor_id=$hostname-$vm_rbd
openstack flavor delete $new_vm_flavor_id
openstack flavor create --id $new_vm_flavor_id --ram $vm_memory_MB \
--vcpus $vm_vcpus --disk $vm_rbd_size_GB $new_vm_flavor_id
# 用 bootable 卷起虚机(安全组 / 网络 ID 按环境替换)
new_vm_name=$hostname-$vm_name
openstack server delete $new_vm_name
openstack server create $new_vm_name --volume $vm_rbd_boot_uuid \
--flavor $new_vm_flavor_id \
--security-group <安全组ID> --nic net-id=<外部网络ID>
# 数据盘:最后一卷与系统盘同名则视为没有数据盘
data_rbd=`grep rbd /data/hosts/$hostname/$vm_config | awk '{print $3}' \
| awk -F '=|/' '{print $3}' | awk -F"'" '{print $1}' | tail -1`
echo $data_rbd
if [ "$data_rbd"x = "$vm_rbd"x ]; then
echo "$new_vm_name have not data disk!"
else
data_rbd_pool=`grep rbd /data/hosts/$hostname/$vm_config | awk '{print $3}' \
| awk -F "='|/" '{print $2}' | tail -1`
old_pool=$data_rbd_pool
data_rbd_size=`rbd info $old_pool/$data_rbd | head -2 | tail -1 | awk '{print $2}'`
data_rbd_unit=`rbd info $old_pool/$data_rbd | head -2 | tail -1 | awk '{print $3}'`
if [ "$data_rbd_unit"x = "GB"x ]; then
data_rbd_size_GB=$data_rbd_size
else
if [ $data_rbd_size -le 1024 ]; then
data_rbd_size_GB=1
else
data_rbd_size_GB=`echo $data_rbd_size/1024 | bc`
fi
fi
# 导出数据盘并灌入新卷
backup_data_rbd=/data/hosts/$hostname/backup.$data_rbd
rbd export -p $old_pool $data_rbd $backup_data_rbd
new_data_rbd=$hostname-$data_rbd
openstack volume create $new_data_rbd --size $data_rbd_size_GB
data_rbd_uuid=`openstack volume list | grep $new_data_rbd | awk '{print $2}'`
rbd rm -p $new_pool volume-$data_rbd_uuid
rbd import -p $new_pool $backup_data_rbd volume-$data_rbd_uuid
rm -rf /data/hosts/$hostname/backup*
# 挂回虚机
openstack server add volume $new_vm_name $data_rbd_uuid
fi
# 绑浮动 IP;成功即删 XML 收尾(防重复迁移)
openstack floating ip create public1
floating_ip=$(openstack floating ip list | grep None | head -1 | awk '{print $4}')
openstack server add floating ip $new_vm_name $floating_ip
if [ $? -eq 0 ]; then
rm -rf /data/hosts/$hostname/$vm_config
fi
done
接管的存量虚机经常拿不到 root 密码。CentOS 7 在 GRUB 界面把光标移到 LANG=en_US.UTF-8 后追加 init=/bin/sh(同一行),Ctrl+X 启动进 shell 后:
mount -o remount,rw /
passwd root
# 若启用了 selinux,还需要:
touch /.autorelabel
exec /sbin/init # 或 exec /sbin/reboot
业务涨了随时加计算节点:新节点按前文的环境准备、yum 源、Ceph 客户端三步准备好,multinode 里加上主机名,重跑部署三连即可:
kolla-ansible -i /root/multinode bootstrap-servers
kolla-ansible -i /root/multinode prechecks
kolla-ansible -i /root/multinode deploy
openstack compute service list | grep nova-compute
生产环境每台 OSD 节点新加 4 块 480G SSD 放日志盘。以 hostS06 为例:27 个 OSD 对应 27 个分区,规划为 7 + 7 + 7 + 6——前三块 SSD 各划 7 个分区(前 6 个 66G、最后一个 84G),最后一块划 6 个分区(每个 80G)。
set -e
/usr/bin/ceph osd set noout
PARTUUIDDIR=/dev/disk/by-partuuid
OSDS=$(lsblk | grep ceph | awk -F'/|-' '{print $NF}')
# 27 个分区与 27 个 OSD 一一对应
DEVICES=(sdn5 sdn6 sdn7 sdn8 sdn9 sdn10 sdo1 sdo5 sdo6 sdo7 sdo8 sdo9 sdo10 \
sdp1 sdp5 sdp6 sdp7 sdp8 sdp9 sdp10 sdq1 sdq5 sdq6 sdq7 sdq8 sdq9)
for i in {2..27}
do
DEVICE=${DEVICES[$i]}
OSD_ID=${OSDS[$i]}
OSD_Journal=/var/lib/ceph/osd/ceph-$OSD_ID/journal
UUID=$(uuidgen)
ln -s /dev/$DEVICE $PARTUUIDDIR/$UUID
stop ceph-osd id=$OSD_ID
ceph-osd -i $OSD_ID --flush-journal
rm $OSD_Journal
ln -s $PARTUUIDDIR/$UUID $OSD_Journal
chown ceph:ceph $OSD_Journal
echo $UUID > /var/lib/ceph/osd/ceph-$OSD_ID/journal_uuid
ceph-osd -i $OSD_ID --mkjournal
restart ceph-osd id=$OSD_ID
done
ceph osd unset noout
--bootable 卷拿到 volume-UUID 命名的 RBD 镜像,再 rbd rm + rbd import,绕开了「卷必须由 Cinder 创建」的格式问题。2026 年发布说明:文中版本(Ceph Jewel、OpenStack Queens、Ubuntu 14.04 / CentOS 7.4)对应 2019 年生产环境,命令细节以当时版本为准。如今新项目我们默认走「Proxmox VE + Ceph」或国产私有云路线,OpenStack 线继续做存量集群的运维托管与故障救援;迁移脚本中的密钥、网络 ID 等客户环境信息已脱敏,个别原文笔误(服务名拼写、heredoc 重定向符、awk 引号)已按上下文修正。相关服务见 OpenStack 实施 · 运维 专题页,或参考本站 XenServer 群集搭建实录。