Migration from Liberty to Mitaka at IPHC

This document details a simple procedure to upgrade OpenStack from Liberty to Mitaka. A short downtime of 2 hours is required to perform the upgrade and test the services.

To upgrade to Mitaka, the following steps are performed:

1. Stop the daemons of the configuration management tool (Puppet, Chef, Quattor, ...) to ensure that it will not interfere with the upgrade procedure. We are using Quattor at IPHC. Two daemons need to be stopped:
   
   [root@controller ~]# service ncm-cdipsd stop
   [root@controller ~]# service cdp-listend stop
2. Stop the OpenStack services and ensure with the systemctl command that they are effectily stopped:
   [root@controller ~]# for service in "nova neutron cinder glance"; do \
       service openstack-${service} stop \
   done
   [root@controller ~]# service httpd stop
   Note: we have created startup scripts, like /etc/init.d/openstack-nova, that manage all related daemons.
3. We took advantage of this upgrade to perform some database cleanup:
   
   • Backup the database using mysql_dump
   • [root@controller ~]# keystone-manage token_flush
   • On our test infrastructure, we were not able to update the keystone database. This issue was caused by a UTF-8 charset problem. To fix this, we had to set correctly the charset (utf8/utf8_general_ci) of each table and database using the following script. This step is probably not  required if your installation of OpenStack if younger than Juno.
4. Replace the RDO Liberty repo by the Mitaka repo and update the RPMs:
   [root@controller ~]# cat /etc/yum.repos.d/mitaka.repo
   [x86_64]
   name=OpenStack Mitaka Repository
   baseurl=http://mirror.centos.org/centos/7/cloud/x86_64/openstack-mitaka/
   enabled=1
   skip_if_unavailable=0
   gpgcheck=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
   priority=98
   [root@controller ~]# rm /etc/yum.repos.d/liberty.repo
5. Install the configuration files for the new version. We are using our configuration management tools,  in manual mode):
   [root@controller ~]# ccm-fetch
   [root@controller ~]# ncm-ncd --config filecopy
   [root@controller ~]# ncm-ncd --config mysql
6. Once all OpenStack components are configured, each database needs to be updated to the current schema:
   
   1. Keystone
      [root@controller ~]# su -s /bin/sh -c "keystone-manage db_sync" keystone
   2. Glance
      [root@controller ~]# su -s /bin/sh -c "glance-manage db_sync" glance
   3. Cinder
      [root@controller ~]# su -s /bin/sh -c "cinder-manage db sync" cinder
   4. Neutron
      [root@controller ~]# su -s /bin/sh -c "neutron-db-manage upgrade heads" neutron
   5. Nova
      First, if your configuration management tool do not create database, manually create the nova_api database and give the nova user access to it.
      [root@controller ~]# su -s /bin/sh -c "nova-manage api_db sync" nova
      [root@controller ~]# su -s /bin/sh -c "nova-manage db sync" nova
7. The upgrade is completed, restart the OpenStack services, as well as the configuration management tool daemon(s). Look at the OpenStack log files for any errors and test your services with Tempest.

Restore iSCSI configuration for Cinder / Nova

In few cases (i.e. cinder-volume crash), some cinder volumes cannot be accessed by a VM (I/O errors), but are still displayed as associated when using cinder or nova CLI. Looking at the hypervisor's log, you may see:
May 11 13:26:45 cloudhyp1 iscsid: conn 0 login rejected: target error (03/01)
May 11 13:26:45 cloudhyp1 iscsid: conn 0 login rejected: initiator failed authorization with target
May 11 13:26:45 cloudhyp1 iscsid: conn 0 login rejected: initiator failed authorization with target

On the cinder-volume host, check the configuration of iSCSI target:
[root@controller ~]# targetcli ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e  [/dev/cinder-volumes/volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e (20.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 7]
  | o- iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e ............................................. [TPGs: 1]
  | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0  [block/iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e (/dev/cinder-volumes/volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 192.168.1.1:3260 ................................................................................................. [OK]
  o- loopback ......................................................................................................... [Targets: 0]

In that case, the cloudhyp1 cannot connect to the target because no ACL are defined ([ACLs: 0])

 You have to setup the ACL manually:
[root@controller ~]# mysql -u cinder -p -e "select provider_auth from volumes where id='6e95e5b6-83e1-4958-a5e1-ba5afc94559e'" cinder
Enter password:
+------------------------------------------------+
| provider_auth                                  |
+------------------------------------------------+
| CHAP xjrFIwOQ66ktkxjrFIwO vr2twXxoDww7wvr2twXx |
+------------------------------------------------+

The first entry is the username and the second the password. You can check that you have the same value on the hypervisor (1, 2):
[root@cloudhyp1 ~]# grep node.session.auth /var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e/192.168.1.1,3260,1/default
node.session.auth.authmethod = CHAP
node.session.auth.username = xjrFIwOQ66ktkxjrFIwO
node.session.auth.password = vr2twXxoDww7wvr2twXx

On the hypervisor, you need also to get the initiator id (3):

[root@cloudhyp1 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:1abc12d345e6
 
To update the ACL, first save the targetcli configuration:
[root@controller ~]# targetctl save
[root@controller ~]# cp /etc/target/saveconfig.json /etc/target/saveconfig.old

Replace:
          "node_acls": [] 

By for the right volume (iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e in our case) : 
          "node_acls": [
            {
              "attributes": {
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "chap_password": "vr2twXxoDww7wvr2twXx",
              "chap_userid": "xjrFIwOQ66ktkxjrFIwO",
              "mapped_luns": [
                {
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1994-05.com.redhat:1abc12d345e6"
            }
          ]

You have to replace chap_userid, chap_password and node_wwn by values obtained in steps 1, 2 and 3 respectively.

Then check and load the configuration:
[root@controller ~]# cat /etc/target/saveconfig.json | json_verify
JSON is valid
[root@controller ~]# targetctl restore

You can connect again to the iSCSI target from the hypervisor:
[root@cloudhyp1 ~]# iscsiadm -m node -T iqn.2010-10.org.openstack:volume-6e95e5b6-83e1-4958-a5e1-ba5afc94559e -l