Migration from Mitaka to Newton

This document details a simple procedure to upgrade an OpenStack Cloud from Mitaka to Newton. A short downtime of 2 hours is required to perform the upgrade and test the services.

To upgrade to Newton, the following steps are performed:

1. Stop the daemons of the configuration management tool (Puppet, Chef, Quattor, ...) to ensure that it will not interfere with the upgrade procedure. We are using Quattor at IPHC. Two daemons need to be stopped:
   
   [root@controller ~]# service ncm-cdipsd stop
   [root@controller ~]# service cdp-listend stop
2. Stop the OpenStack services and ensure with the systemctl command that they are effectily stopped:
   [root@controller ~]# for service in "nova neutron cinder glance"; do \
       service openstack-${service} stop \
   done
   [root@controller ~]# service httpd stop
   Note: we have created startup scripts, like /etc/init.d/openstack-nova, that manage all related daemons.
3. We took advantage of this upgrade to perform some database cleanup:
   
   • Backup the database using mysql_dump
   • [root@controller ~]# keystone-manage token_flush
   • On our test infrastructure, we were not able to update the keystone database. This issue was caused by a UTF-8 charset problem. To fix this, we had to set correctly the charset (utf8/utf8_general_ci) of each table and database using the following script. This step is probably not  required if your installation of OpenStack if younger than Juno.
4. Replace the RDO Mitaka repo by the Newton repo and update the RPMs:
   [root@controller ~]# cat /etc/yum.repos.d/newton.repo
   [x86_64]
   name=OpenStack Newton Repository
   baseurl=http://mirror.centos.org/centos/7/cloud/x86_64/openstack-newton/
   enabled=1
   skip_if_unavailable=0
   gpgcheck=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
   priority=98
   [root@controller ~]# rm /etc/yum.repos.d/mitaka.repo
5. Install the configuration files for the new version. We are using our configuration management tools,  in manual mode):
   [root@controller ~]# ccm-fetch
   [root@controller ~]# ncm-ncd --config filecopy
   [root@controller ~]# ncm-ncd --config mysql
6. Once all OpenStack components are configured, each database needs to be updated to the current schema:
   
   1. Keystone
      [root@controller ~]# su -s /bin/sh -c "keystone-manage db_sync" keystone
   2. Glance
      [root@controller ~]# su -s /bin/sh -c "glance-manage db_sync" glance
   3. Cinder
      [root@controller ~]# su -s /bin/sh -c "cinder-manage db sync" cinder
      With cinder, we hit the following issue:
      ERROR oslo_service.service ServiceTooOld: One of the services is in Liberty vers
      ion. We do not provide backward compatibility with Liberty now, you need to upgra
      de to Mitaka first.
      If you check the enabled services, you can see that a cinder-backup service is registered:
      [root@controller ~]# cinder-manage service list
      Binary           Host                                 Zone             Status     State Updated At           RPC Version  Object Version  Cluster
      cinder-scheduler controller                           nova             enabled    XXX   2017-07-27 17:38:25  3.0          1.11
      cinder-volume    controller                           nova             enabled    XXX   2017-07-27 17:38:25  3.0          1.11
      cinder-backup    controller                           nova             enabled    XXX   2014-07-15 05:49:40  None         None
      The solution is to remove the out-dated service:
      [root@controller ~]# cinder-manage service remove cinder-backup controller
   4. Neutron
      [root@controller ~]# su -s /bin/sh -c "neutron-db-manage upgrade heads" neutron
   5. Nova
      First, if your configuration management tool do not create database, manually create the nova_api database and give the nova user access to it.
      [root@controller ~]# su -s /bin/sh -c "nova-manage api_db sync" nova
      [root@controller ~]# su -s /bin/sh -c "nova-manage db sync" nova
   6. Heat
      [root@controller ~]# su -s /bin/sh -c "heat-manage db_sync" heat
   7. Magnum
      [root@controller ~]# su -s /bin/sh -c "magnum-db-manage upgrade heads" magnum
7. The /etc/keystone/credential-keys/ directory has to be created. It is owned by the keystone user.
8. The use of Keystone was really slow after the upgrade. To return to a normal state, be sure that Keystone is using memcache and that your token get flushed regularly (we are using a cron file).
9. The upgrade is completed, restart the OpenStack services, as well as the configuration management tool daemon(s). Look at the OpenStack log files for any errors and test your services using your favorite probe platform.
10.  After this update, the creation of new flavor fails with the following error:
    

    not all flavors have been migrated to the API database

    This is caused by a known bug. To revolve this issue, we have run:
    

    nova-manage db online_data_migrations