Restoring the Undercloud

The following restore process assumes you are recovering from a failed undercloud node where you have to reinstall it from scratch. It assumes that the hardware layout is the same, and the hostname and undercloud settings of the machine will be the same as well. Once the machine is installed and is in a clean state, re-enable all the subscriptions/repositories needed to install and run TripleO.

Note that unless specified, all commands are run as root.

Restoring a backup of your Undercloud on a Fresh Machine

Install the MariaDB server with:

yum install -y mariadb-server

Now restore the MariaDB configuration file and database backup, then start the MariaDB server and load the backup in:

tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/mariadb-server.cnf /root/undercloud-all-databases.sql
# Edit /etc/my.cnf.d/server.cnf and comment out 'bind-address'
systemctl start mariadb
cat /root/undercloud-all-databases.sql | mysql
# Now we need to clean out some old permissions to be recreated
for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
mysql -e 'flush privileges'

Note that depending on the MySQL version, the config file can be /etc/my.cnf.d/mariadb-server.cnf or /etc/my.cnf.d/server.cnf.

Now create the stack user and restore the stack users home directory:

sudo useradd stack
sudo passwd stack  # specify a password

echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
sudo chmod 0440 /etc/sudoers.d/stack

Next restore the stack users home directory:

tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

We have to now install the swift and glance base packages, and then restore their data:

yum install -y openstack-glance openstack-swift
tar -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images
# Confirm data is owned by correct user
chown -R swift: /srv/node
chown -R glance: /var/lib/glance/images

Finally, we rerun the undercloud installation from the stack user, making sure to run it in the stack user home dir:

su - stack
sudo yum install -y python-tripleoclient
# Double check hostname is correctly set in /etc/hosts
openstack install undercloud

If you are using Pike and Ceph will be used in the overcloud, install ceph-ansible on the undercloud:

sudo yum install -y ceph-ansible

Reconnect the restored undercloud to the overcloud

Having completed the steps above, the undercloud can be expected to automatically restore its connection to the overcloud. The nodes will continue to poll Orchestration (heat) for pending tasks, using a simple HTTP request issued every few seconds.