In the last few articles, I’ve detailed the workflow to deploy Mirantis Cloud Platform (aka MK now MCP) based on Mirantis OpenStack and a Reclass Model Driven Architecture (MDA). But you may want to use a different backend for storage then our standard MCP one, the reference Cinder LVM ISCSI Driver. In this MCP cookbook article, I’ll guide you step by step, to use Ceph as your storage backend for Glance (Images), Cinder (Volumes) and Nova (Guest Disks). You can use any of these options alone or combined.
Introduction
OpenStack Salt doesn’t deploy Ceph as Mirantis Fuel does, at Mirantis we use Decapod for this task instead. A detailed installation guide on how to use Decapod isn’t the objective of this post. I suppose you’ve already found your way and have a running Ceph cluster. I’ll focus instead on how to adapt our Reclass model to consume your Ceph cluster. So before we start, make sure your Ceph cluster is healthy
# ceph health
HEALTH_OK
glance-api, cinder-volume, nova-compute and cinder-backup will be acting as Ceph clients, so we’ll need a corresponding /etc/ceph/ceph.conf file on each of them.
But first of all, even if our model doesn’t deploy Ceph, it can at least prepare the required infrastructure. For example, by default our model propose only three Infrastructure compute nodes (kvm01, kvm02, kvm03), we could add more if necessary.
Adding Infra compute, updating control and compute Nodes
If you don’t want to host Decapod and Ceph monitoring vms on your existing infra nodes, you can add three more (kvm04,kvm05,kvm06). It can also help to better spread all the required services. Up to nine nodes are pre-configured in Mirantis system repo you just need to add their IPs and hostname as shown below.
#!yaml
# vi classes/cluster/xxx/init.yml
parameters:
_param:
infra_compute_node04_address: <CTL_NET.NODE04_IP>
infra_compute_node05_address: <CTL_NET.NODE05_IP>
infra_compute_node06_address: <CTL_NET.NODE06_IP>
...
infra_compute_node04_deploy_address: <PXE_NET.NODE04_IP>
infra_compute_node05_deploy_address: <PXE_NET.NODE05_IP>
infra_compute_node06_deploy_address: <PXE_NET.NODE06_IP>
...
infra_compute_node01_storage_address: <STORAGE_FRONTEND_NET.NODE01_IP> <.>
infra_compute_node02_storage_address: <STORAGE_FRONTEND_NET.NODE02_IP>
infra_compute_node03_storage_address: <STORAGE_FRONTEND_NET.NODE03_IP>
infra_compute_node04_storage_address: <STORAGE_FRONTEND_NET.NODE04_IP>
infra_compute_node05_storage_address: <STORAGE_FRONTEND_NET.NODE05_IP>
infra_compute_node06_storage_address: <STORAGE_FRONTEND_NET.NODE06_IP>
...
infra_compute_node04_hostname: kvm04
infra_compute_node05_hostname: kvm05
infra_compute_node06_hostname: kvm06
Replace xxx with your cluster name in the remaining of this article.
We are also adding a storage address on all our infra compute nodes.
Now we need to declare them in our linux.network.host pillar within infra/init.yml to add them to each /etc/hosts files.
#!yaml
# vi classes/cluster/xxx/infra/init.yml
linux:
network:
host:
kvm04:
address: ${_param:infra_compute_node04_address}
names:
- kvm04
- kvm04.${_param:cluster_domain}
kvm05:
address: ${_param:infra_compute_node05_address}
names:
- kvm05
- kvm05.${_param:cluster_domain}
kvm06:
address: ${_param:infra_compute_node06_address}
names:
- kvm06
- kvm06.${_param:cluster_domain}
Decapod and Ceph monitoring vms > sizing
We need to find a server to install Decapod, and at least three monitoring nodes, we can virtualize all of them and create the required vms with Salt Virt. So we first need to add sizing information for Decapod and Ceph monitoring nodes
#!yaml
# vi classes/cluster/xxx/infra/kvm.yml
parameters
salt:
control:
size:
infra.decapod:
cpu: 4
ram: 8192
disk_profile: small
net_profile: storage
infra.storage_monitor:
cpu: 2
ram: 8192
disk_profile: small
net_profile: storage
Decapod and Ceph monitoring VMs > placement | os release | vms specs
In the same file we can also tell Salt on which Infra compute node to deploy each of the required vms, the target operating system and vms specs.
#!yaml
# vi classes/cluster/xxx/infra/kvm.yml
cluster:
internal:
node:
ceph-mon01:
provider: kvm04.${_param:cluster_domain}
image: ${_param:salt_control_xenial_image}
size: infra.storage_monitor
ceph-mon02:
provider: kvm05.${_param:cluster_domain}
image: ${_param:salt_control_xenial_image}
size: infra.storage_monitor
ceph-mon03:
provider: kvm06.${_param:cluster_domain}
image: ${_param:salt_control_xenial_image}
size: infra.storage_monitor
decapod:
provider: kvm06.${_param:cluster_domain}
image: ${_param:salt_control_xenial_image}
size: infra.decapod
Ceph Storage and Ceph monitoring > ip’s
Not to repeat ourself, lets parameterized Ceph storage and monitoring nodes IP Addresses.
#!yaml
# vi classes/cluster/xxx/init.yml
parameters:
_param:
#Ceph storage nodes
infra_ceph_storage_node01_backend_address: <STORAGE_BACKEND_NET.CEPH01_IP> <.>
infra_ceph_storage_node02_backend_address: <STORAGE_BACKEND_NET.CEPH02_IP>
...
infra_ceph_storage_node01_frontend_address: <STORAGE_FRONTEND_NET.CEPH01_IP>
infra_ceph_storage_node02_frontend_address: <STORAGE_FRONTEND_NET.CEPH02_IP>
...
#Ceph monitor nodes
infra_ceph_storage_monitor_node01_address: <CTL_NET.CEPH_MON01_IP>
infra_ceph_storage_monitor_node02_address: <CTL_NET.CEPH_MON02_IP>
infra_ceph_storage_monitor_node03_address: <CTL_NET.CEPH_MON03_IP>
infra_ceph_storage_monitor_node01_backend_address: <STORAGE_BACKEND_NET.CEPH_MON01_IP>
infra_ceph_storage_monitor_node02_backend_address: <STORAGE_BACKEND_NET.CEPH_MON02_IP>
infra_ceph_storage_monitor_node03_backend_address: <STORAGE_BACKEND_NET.CEPH_MON03_IP>
infra_ceph_storage_monitor_node01_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>
infra_ceph_storage_monitor_node02_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>
infra_ceph_storage_monitor_node03_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>
#CTL nodes
openstack_control_node01_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE01_IP> <.>
openstack_control_node02_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE02_IP>
openstack_control_node03_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE03_IP>
#Compute nodes
openstack_compute_node01_storage_address: <STORAGE_FRONTEND_NET.COMPUTE_NODE01_IP>
openstack_compute_node02_storage_address: <STORAGE_FRONTEND_NET.COMPUTE_NODE02_IP>
...
We are also adding storage frontend IPs for openstack control, and compute nodes, it’s required for them to be able to consume Ceph Storage.
Ceph monitoring > net_profile
Our Ceph monitoring VMs needs to be connected to storage backend and frontend networks, so we have to define a new network profile for these vms
#!yaml
# vi classes/cluster/int/infra/kvm.yml
virt:
nic:
...
storage:
- name: eth0
bridge: br-pxe
model: virtio
- name: eth1
bridge: br-mgmt
model: virtio
- name: eth2
bridge: br-sto-front
model: virtio
- name: eth3
bridge: br-sto-back
model: virtio
Note: don’t think much about nic names, Salt Virt has weird sorting, it work from bottom up <.>
Ctl > Nics
Our OpenStack Controller will require connectivity to our storage frontend network, lets create another net_profile for these vms
#!yaml
# vi classes/cluster/int/infra/kvm.yml
virt:
nic:
...
storage-front:
- name: eth0
bridge: br-pxe
model: virtio
- name: eth1
bridge: br-mgmt
model: virtio
- name: eth2
bridge: br-sto-front
model: virtio
Update the openstack.control vms to use this profile instead
# vi classes/cluster/int/infra/kvm.yml
salt:
control:
size:
# Default production sizing
openstack.control:
cpu: 8
ram: 32768
disk_profile: small
net_profile: storage-front
You may already have your controller vms running, in such a case running salt.control state won’t touch the existing ones. So you have to manually add the required additional interface connected to our br-sto-front bridge.
Connect on each infra compute node to list vms
kvm01# virsh list
Id Name State
----------------------------------------------------
12 ctl01.domainname.com running
List existing interfaces of the ctl vm
kvm01# virsh domiflist 12
Interface Type Source Model MAC
-------------------------------------------------------
vnet8 bridge br-pxe virtio ac:de:48:ec:47:6d
vnet9 bridge br-mgmt virtio ac:de:48:c8:4b:2f
Attach the storage one, mac address has been generated randomly
kvm01# virsh attach-interface --domain 12 --type network \
--source br-sto-front --model virtio \
--mac 52:54:00:4b:73:5f --config --live
Check if it has been correclty added
kvm01# virsh domiflist 12
Interface Type Source Model MAC
-------------------------------------------------------
vnet8 bridge br-pxe virtio ac:de:48:ec:47:6d
vnet9 bridge br-mgmt virtio ac:de:48:c8:4b:2f
vmnet10 bridge br-sto-front virtio 52:54:00:4b:73:5f <.>
You can also check that it will persist
kvm01# virsh edit 12
Interface IP will be configured later on, when we’ll run the linux state on the control plane.
Infra compute > VLAN interfaces | Bonds
For our openstack.control VMs to be able to communicate on the storage frontend network, we need to create the corresponding bridges on our infrastructure compute nodes. They may not all be using the same VLANs, so we can parameterized the tag for each node by using our reclass.storage.node pillars like this for each of your infra compute nodes.
#!yaml
# vi classes/cluster/int/infra/config.yml
...
reclass:
storage:
node:
infra_compute_node01:
params:
keepalived_vip_priority: 100
linux_system_codename: xenial
sto_front_interface: <BOND>.<STO_FRONT_VLAN>
sto_back_interface: <BOND>.<STO_BACK_VLAN>
...
Now you can use this _params that will be generated when you’ll run reclass.storage state at the end of this article. Use it to create the required storage VLAN interfaces and bonds
#!yaml
# vi classes/cluster/int/infra/kvm.yml
parameters:
linux:
network:
interface:
...
sto_front_interface:
name: ${_param:sto_front_interface}
enabled: true
type: vlan
proto: manual
use_interfaces:
- bond0
br-sto-front:
enabled: true
type: bridge
proto: manual
use_interfaces:
- ${_param:sto_front_interface}
sto_back_interface:
name: ${_param:sto_back_interface}
enabled: true
type: vlan
proto: manual
use_interfaces:
- bond0
br-sto-back:
enabled: true
type: bridge
proto: manual
use_interfaces:
- ${_param:sto_back_interface}
Compute > VLAN interface | Bond
Compute node only need a connection to the storage frontend, so lets create such a bridge on them, this time without parameters
#!yaml
# vi classes/cluster/int/openstack/compute.yml
parameters:
linux:
network:
interface:
...
<BOND>.<VLAN>:
enabled: true
type: vlan
proto: static
address: ${_param:storage_address}
netmask: <NETMASK>
mtu: 9000
use_interfaces:
- <BOND>
br-sto-front:
enabled: true
type: bridge
proto: manual
use_interfaces:
- <BOND>.<VLAN>
As you see above, this time we need to bind an address to them.
Ceph Storage and Ceph monitoring > nodes declaration
For our Ceph monitoring and storage nodes to be managed by Salt, we need to declare them in our reclass.storage.node pillar. When we’ll run the corresponding reclass.storage state, it will magically create the nodes definitions in nodes/_generated, great isn’t it.
#!yaml
# vi classes/cluster/xxx/infra/config.yml
parameters:
reclass:
storage:
node:
openstack_compute_node01:
...
storage_address: ${_param:openstack_compute_node01_storage_address} <.>
...
...
...
infra_storage_node01: <.>
name: ceph01
domain: ${_param:cluster_domain}
classes:
- cluster.${_param:cluster_name}.infra.storage
params:
salt_master_host: ${_param:reclass_config_master}
linux_system_codename: xenial
single_address: ${_param:infra_ceph_storage_node01_frontend_address}
backend_address: ${_param:infra_ceph_storage_node01_backend_address}
infra_storage_node02:
name: ceph02
domain: ${_param:cluster_domain}
classes:
- cluster.${_param:cluster_name}.infra.storage
params:
salt_master_host: ${_param:reclass_config_master}
linux_system_codename: xenial
single_address: ${_param:infra_ceph_storage_node02_frontend_address}
backend_address: ${_param:infra_ceph_storage_node02_backend_address}
...
infra_storage_monitor_node01:
name: ceph-mon01
domain: ${_param:cluster_domain}
classes:
- cluster.${_param:cluster_name}.infra.storage_monitor
params:
salt_master_host: ${_param:reclass_config_master}
linux_system_codename: xenial
single_address: ${_param:infra_ceph_storage_monitor_node01_address}
frontend_address: ${_param:infra_ceph_storage_monitor_node01_frontend_address}
backend_address: ${_param:infra_ceph_storage_monitor_node01_backend_address}
infra_storage_monitor_node02:
name: ceph-mon02
domain: ${_param:cluster_domain}
classes:
- cluster.${_param:cluster_name}.infra.storage_monitor
params:
salt_master_host: ${_param:reclass_config_master}
linux_system_codename: xenial
single_address: ${_param:infra_ceph_storage_monitor_node02_address}
frontend_address: ${_param:infra_ceph_storage_monitor_node02_frontend_address}
backend_address: ${_param:infra_ceph_storage_monitor_node02_backend_address}
infra_storage_monitor_node03:
name: ceph-mon03
domain: ${_param:cluster_domain}
classes:
- cluster.${_param:cluster_name}.infra.storage_monitor
params:
salt_master_host: ${_param:reclass_config_master}
linux_system_codename: xenial
single_address: ${_param:infra_ceph_storage_monitor_node03_address}
frontend_address: ${_param:infra_ceph_storage_monitor_node03_frontend_address}
backend_address: ${_param:infra_ceph_storage_monitor_node03_backend_address}
We’ve also added a storage_address to each of our compute node.
Note: Keep in mind that infra_ceph_storage_monitor_node03_address is set in classes/cluster/int/init.yml and abstracted away by classes/cluster/int/infra/config.yml into single_address and consumed by classes/cluster/int/init.yml to create interface definitions.
Ceph monitoring > networking
As we’ve said earlier, Decapod will be used to install all of Ceph but Salt is creating the VMs. So lets decribe required network connectivity on each of the storage monitoring VM
#!yaml
# vi classes/cluster/int/infra/storage_monitor.yml
classes:
- system.linux.system.repo.saltstack_2016_3_xenial
- cluster.int
parameters:
linux:
network:
interface:
# CTL NET
eth1: ${_param:linux_single_interface}
# STO_FRONT NET
eth2:
enabled: true
type: eth
proto: static
address: ${_param:frontend_address}
netmask: 255.255.252.0
# STO_BACK NET
eth3:
enabled: true
type: eth
proto: static
address: ${_param:backend_address}
netmask: 255.255.252.0
We do not touch eth0 with our model, it will be configured by dhcp.
Note: This class is associated to each Ceph monitoring by classes/cluster/xxx/infra/config.yml when we declare them.
Ceph client > authentication [CLOUDVPS]
For our client to authenticate to Ceph using cephx we need to create new user for Nova/Cinder and Glance
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=object'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder, allow rwx pool=nova, allow rwx pool=images'
ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=nova'
Verify the users exist
ceph auth list
Ceph client > _params [CLOUDVPS]
system.ceph.client.single will configure our nodes as Ceph clients by running the salt formula ceph with the required Pillar data which are.
ceph_fsid: uuid of the Ceph Cluster
ceph_mon_initial_members: initial Ceph monitoring nodes FQDN
ceph_mon_host: IP:Port List of Ceph monitoring nodes
ceph_auth_client_required: choose cephx here
ceph_public_network: Ceph frontend Network CIDR
ceph_cluster_network: Ceph backend Network CIDR
So you now have to provide all the required parameters in classes/cluster/xxx/openstack/init.yml
#!yaml
# vi classes/cluster/xxx/openstack/init.yml
parameters:
_param:
...
cinder_storage_user: cinder
glance_storage_user: object
nova_storage_user: nova
nova_storage_secret_uuid: <NOVA_STORAGE_SECRET_UUID>
cinder_storage_client_key: <CINDER_STORAGE_KEY>
glance_storage_client_key: <GLANCE_STORAGE_KEY>
nova_storage_client_key: <NOVA_STORAGE_KEY>
admin_storage_client_key: <ADMIN_STORAGE_KEY>
cinder_storage_pool: cinder
glance_storage_pool: object
nova_storage_pool: nova
ceph_fsid: <CEPH CLUSTER UUID>
ceph_mon_initial_members: ceph-mon01,ceph-mon02,ceph-mon03
ceph_mon_host: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>:6789,<STORAGE_FRONTEND_NET.CEPH_MON02_IP>:6789,<STORAGE_FRONTEND_NET.CEPH_MON03_IP>:6789
ceph_auth_client_required: cephx
ceph_public_network: <STORAGE_FRONTEND_NET.CIDR>
ceph_cluster_network: <STORAGE_BACKEND_NET.CIDR>
ceph:
client:
keyring:
object:
key: ${_param:glance_storage_client_key}
cinder:
key: ${_param:cinder_storage_client_key}
nova:
key: ${_param:nova_storage_client_key}
admin:
key: ${_param:admin_storage_client_key}
To be able to mount Ceph backed volume, the nova compute nodes need to store the secret key of the client.cinder user in libvirt. So it will be stored behind the provided <NOVA_STORAGE_SECRET_UUID>. Ceph salt formula will be storing it using a secret.xml template file containing
#!xml
<secret ephemeral='no' private='no'>
<uuid>{{ compute.ceph.secret_uuid }}</uuid>
<usage type='ceph'>
<name>client.{{ compute.ceph.get('rbd_user', 'cinder') }} secret</name>
</usage>
</secret>
And to define and store the secret in libvirt
# virsh secret-define --file secret.xml
# virsh secret-set-value --secret NOVA_STORAGE_SECRET_UUID> --base64 <CINDER_STORAGE_KEY>
As we’ve seen earlier, you can get <CINDER_STORAGE_KEY> with
# ceph auth get-key client.cinder
On a compute node, you can list the existing libvirt secrets with
# virsh secret-list
UUID Usage
--------------------------------------------------------------------------------
a5d0dd94-57c4-ae55-ffe0-7e3732a24455 ceph client.nova secret
But do not care too much, Salt will do all this boring stuff for you. Thanks Salt !!!
Ceph Cluster uuid (generated with uuidgen at Ceph install time) can be found in your ceph cluster in /etc/ceph/ceph.conf
You can grab the required keys for client.cinder, client.glance, client.cinder-backup, client.nova and client.admin like this
ceph auth get-or-create client.glance
ceph auth get-or-create client.cinder
ceph auth get-or-create client.cinder-backup
ceph auth get-or-create client.nova
ceph auth get-or-create client.admin
[client.admin]
key = AQCdhCCCt2PmGRAAwww8oYhiiik80VQ==
You can also try authentication by requesting your cluster status
# ceph -s --name client.cinder --keyring /etc/ceph/ceph.client.cinder.keyring
Ceph Pools
We’ve configured the cinder, object, nova as the Ceph pools to be used by OpenStack. We need to create them on our Ceph Cluster
ceph osd pool create cinder 128
ceph osd pool create object 128
ceph osd pool create nova 128
ceph osd pool create backups 128 <.>
See Create a Pool for detail on specifying the number of placement groups for your pools, and Placement Groups for details on the number of placement groups you should set for your pools.
Verify the pools exists
ceph osd lspools
Ceph > Compute nodes <.>
We have to tell our compute nodes about Ceph and connect our hypervisor to the storage frontend network. The following code does most of the job for us. You just have to adapt it to your requirements.
#!yaml
vi classes/cluster/mb-staging/openstack/compute.yml <.>
classes:
...
- system.ceph.client.single
...
parameters:
linux:
network:
bridge: openvswitch
interface:
...
bond0.<STO_NET_VLAN>:
enabled: true
type: vlan
proto: static
address: ${_param:storage_address}
netmask: <STO_NET_NETMASK>
mtu: 9100
use_interfaces:
- bond0
...
nova:
compute:
ceph:
enabled: true
ephemeral: yes
rbd_pool: nova
secret_uuid: ${_param:nova_storage_secret_uuid}
client_cinder_key: ${_param:nova_storage_client_key}
rbd_user: nova
The above nova section configure nova to use Ceph as a backend for ephemeral storage.
Ceph > Ctl nodes
Our control nodes need a bit more classes
#!yaml
vi classes/cluster/mb-staging/openstack/control.yml <.>
classes:
...
- system.glance.control.storage.ceph
- system.ceph.client.single
vi classes/cluster/mb-staging/openstack/control_init.yml
classes:
...
- system.cinder.control.backend.ceph
- system.cinder.volume.backend.ceph
system.glance.control.storage.ceph configure Ceph RBD as a backend for Glance
system.cinder.control.backend.ceph configure Ceph as a backend for Cinder
system.cinder.volume.backend.ceph configure Ceph as a backend for Cinder
Note: if you don’t have these classes in your system repo, you may have to bump its version a bit.
Glusterfs > remove Glance volume
In our standard deployment we use a Glusterfs volume to store our Glance image. If you’ve switched to Ceph it’s a good idea to comment out this feature in your model
# vi classes/xxx/openstack/control.yml
...
#- system.glusterfs.client.volume.glance
...
#- system.glusterfs.server.volume.glance
If you’ve already ran the Glusterfs state, you’ll have to manually unmount and delete the Glance volumes which are shared by the infrastructure nodes. On each of your three controllers run
ctl01# umount /var/lib/glance/images
On one of your kvm node run
kvm01# gluster volume stop
kvm01# gluster delete volume glance
The above command does not delete the data, you have to do this manually too. <.>
git > commit
We now have to share our work to our team, commit your changes
# git add .
# git commit -m "Ceph backend for Nova, Glance and Cinder"
# git push
Now ssh to your MCP salt master to pull the updated model
cfg01# cd /srv/salt/reclass
cfg01# git pull
Salt states > provision Ceph infra
It’s now time apply our new model. First of all run the following state to create the ceph-mon01,ceph-mon02,ceph-mon03 node definition under nodes/_generated.
# salt 'cfg01*' state.sls reclass.storage
Next you can create the four VMs: ceph-mon01, ceph-mon02, ceph-mon03 and decapod.
# salt 'kvm*' state.sls salt.control
Since you have new nodes, refresh pillars and sync_all
# salt '*' saltutil.refresh_pillar && salt '*' saltutil.sync_all
Run the following states on your ceph-mon’s and decapod minions, also run on kvm04, kvm05, kvm06 if you’ve added them.
# salt -C 'E@ceph-mon* and E@decapod*' linux,openssh,rsyslog,ntp,salt.minion
Salt states > Ceph OpenStack backend configuration
Now that we have your Ceph cluster up and running and healthy, we are ready to tell OpenStack to consume it.
Run linux node on your compute, infra compute and ctl nodes to update networking with new storage networks.
# salt -C 'E@kvm* and E@cmp* and E@ctl*' state.sls linux
Run nova state on your computes
# salt -C 'cmp*' state.sls nova
Run cinder|glance on your control plane
# salt -C 'I@cinder:controller' state.sls cinder -b 1
# salt -C 'I@glance:server' state.sls glance -b 1
That should be it. If I forgot something it will be added above soon, as of January 31, 2017 ;)
Glance Image
If we are lucky, we should now be able to upload a Glance image, remember that Ceph expect the image to be in RAW format if you want to boot from it, you can convert it from qcow2 like this
qemu-img convert -f qcow2 -O raw ubuntu-16.04-xenial-server-cloudimg-amd64-disk1.img ubuntu-16.04.raw
upload it
# source /root/keystonerc
# glance image-create --name ubuntu-16-04 --is-public=true --disk-format=raw --container-format=bare < ubuntu-16.04.raw
You can verify that the image is stored in Ceph by querying the image ID in the Ceph image pool
# rados -p object ls --name client.glance --keyring /etc/ceph/ceph.client.glance.keyring | grep -i id
Create a volume from it, to be able to boot an instance using copy-on-write
# cinder create --image-id <IMAGE_ID> --display-name <VOL_NAME> <SIZE>
You can also verify volume is stored within Ceph cinder pool
# rados -p cinder ls --name client.cinder --keyring /etc/ceph/ceph.client.cinder.keyring | grep -i id
Conclusion
Having the capability to model our infrastructure using Reclass and Salt from a git repository is really nice. As we’ve demonstrated in this cookbook article. It gives us the ability to adapt our model to our need, from Cinder iSCSI LVM to Ceph. Not only that, it also gives you a total visibility about who’s doing what over time on the versionned repository, it’s infrastructure as code, at its core.
At the same time leveraging hundreds of salt formulas to do anything from linux configuration to Kubernetes deployment is a great bonus. Our model could also be adapted to use Neutron ml2 Open vSwitch plugin instead of OpenContrail, but I think you see my point and that’s the subject of another cookbook article, maybe.
See you next time. In the meantime, take care !
PS: Before I leave I want to thank Ondrej Smola, Pavel Cizinsky and Jakub Pavlik for their great support :)
Links
- salt-formula-ceph
- Ceph Block Devices and OpenStack documentation
- Using Libvirt with Ceph RBD documentation
- Decapod documentation