Upgrading OpenStack by Updating Each Service Individually
1. Overview
Note
2. Configure RHN/CDN Channels
Warning
-
Red Hat OpenStack 1.0 (Essex) -- rhel-server-ost-6-preview-rpms
-
Red Hat OpenStack 2.1 (Folsom) -- rhel-server-ost-6-folsom-rpms
-
Red Hat Enterprise Linux OpenStack Platform 3 (Grizzly) -- rhel-server-ost-6-3-rpms
-
Red Hat Enterprise Linux OpenStack Platform 4 Beta (Havana) -- rhel-6-server-openstack-beta-rpms
-
Red Hat Enterprise Linux OpenStack Platform 4 (Havana) -- rhel-6-server-openstack-4.0-rpms
Note
#
subscription-manager repos \
--enable=rhel-6-server-rh-common-rpms
2.1. Content Delivery Network Channels
#
subscription-manager repos --enable=[reponame]
#
subscription-manager repos --disable=[reponame]
The following tables outline the channels for Red Hat Enterprise Linux 6.
Table 1. Required Channels
Channel | Repository Name |
---|---|
Red Hat Enterprise Linux 6 Server (RPMS) |
rhel-6-server-rpms
|
Red Hat OpenStack 5.0 (RPMS) for Server 6 |
rhel-6-server-openstack-5.0-rpms
|
Table 2. Optional Channels
Channel | Repository Name |
---|---|
RHEL Server Load Balancer (v6 for 64-bit x86_64) |
rhel-lb-for-rhel-6-server-rpms
|
Red Hat Enterprise Linux 6 Server - Optional |
rhel-6-server-optional-rpms
|
The following tables outline the channels for the Red Hat Enterprise Linux OpenStack Platform installer.
Table 3. Required Channels
Channel | Repository Name |
---|---|
Foreman-RHEL-6 |
rhel-6-server-openstack-foreman-rpms
|
Red Hat Enterprise Linux 6 Server (RPMS) |
rhel-6-server-rpms
|
Red Hat Software Collections RPMs for Red Hat Enterprise Linux 6 Server |
rhel-server-rhscl-6-rpms
|
The following table outlines the channels you must disable to ensure Red Hat Enterprise Linux OpenStack Platform 5 functions correctly.
Table 4. Disable Channels
Channel | Repository Name |
---|---|
Red Hat CloudForms Management Engine |
"cf-me-*"
|
Red Hat CloudForms Tools for RHEL 6 |
"rhel-6-server-cf-*"
|
Red Hat Enterprise Virtualization |
"rhel-6-server-rhev*"
|
Red Hat Enterprise Linux 6 Server - Extended Update Support |
"*-eus-rpms"
|
2.2. Red Hat Network (RHN) Channels
#
rhn-channel --add --channel=[reponame]
#
rhn-channel --remove --channel=[reponame]
The following tables outline the channels for Red Hat Enterprise Linux 6.
Table 5. Required Channels
Channel | Repository Name |
---|---|
Red Hat OpenStack 5.0 for RHEL 6 Server x86_64 |
rhel-x86_64-server-6-ost-5
|
Red Hat Enterprise Linux Server (v6 for 64-bit AMD64 / Intel64) |
rhel-x86_64-server-6
|
Table 6. Optional Channels
Channel | Repository Name |
---|---|
RHEL Server Load Balancer (v6 for 64-bit x86_64) |
rhel-x86_64-server-lb-6
|
RHEL Server Optional (v. 6 64-bit x86_64) |
rhel-x86_64-server-optional-6
|
MRG Messaging v2 (for RHEL 6 Server x86_64) |
rhel-x86_64-server-6-mrg-messaging-2
|
3. Upgrade Services Individually
Table 7. Service Upgrade Order and Instructions
Service | Notes |
---|---|
Identity (keystone) |
Because the Identity service in Red Hat Enterprise Linux OpenStack Platform 4 never purged expired tokens, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
To flush expired tokens from the database and alleviate the problem, the
keystone-manage token_flush command can be used before running the Identity database upgrade.
On your Identity host, run:
|
Object Storage (swift) |
On your Object Storage hosts, run:
|
Block Storage (cinder) |
On your Block Storage host, run:
|
Image Service (glance) |
On your Image Service host, run:
|
Telemetry (ceilometer) |
On all nodes hosting Telemetry component services, run:
For a list of Telemetry component services, refer to Launch the Telemetry API and Agents. After completing the package upgrade, restart the Telemetry service by running the following command on all nodes hosting Telemetry component services:
|
Orchestration (heat) |
On your Orchestration host, run:
|
Dashboard (horizon) |
On your Dashboard host, run:
|
4. Upgrade Compute Service
Procedure 1. Upgrading Compute controller
-
On your Compute controller node, stop any running
nova
services:#
openstack-service stop nova
Then, upgrade the nova packages on this node:#
yum -y upgrade \*nova\* python-migrate -y
-
Upgrade the compute database by running the database upgrade script on the Compute controller node:
#
openstack-db --service nova --update
-
Restart the controller services. To do so, run the following on the controller node:
#
openstack-service start nova
4.1. Configure VIF Plugging
libvirt_vif_driver
in /etc/nova/nova.conf
, as the old hybrid driver is now deprecated. To do so, run the following on your Compute API host:
#
openstack-config --set /etc/nova/nova.conf \
DEFAULT libvirt_vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver
#
openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_is_fatal true
#
openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_timeout VIFTIMEOUT
300
.
4.2. Upgrade Compute Nodes
libvirt_vif_driver
in /etc/nova/nova.conf
(Section 4.1, “Configure VIF Plugging”), your controller node should now be running the Red Hat Enterprise Linux OpenStack Platform 5 version of nova
. Your Compute nodes, however, would still be running the Red Hat Enterprise Linux OpenStack Platform 4 version of nova
.
Procedure 2. Upgrading a Compute node
Note
-
Prevent Compute from scheduling any new instances on the node. To do so, set the
nova-compute
service todisabled
:#
nova service-disable --reason upgrade NODENAME nova-compute
Replace NODENAME with the name of the compute host you wish to upgrade. NODENAME must be the host's name as it is known to Compute. To see a list of compute host names recognized by Compute:#
nova service-list --binary nova-compute
-
Stop the
nova-compute
service on the node.#
openstack-service stop nova-compute
-
Upgrade all the nova packages, along with python-migrate:
#
yum upgrade \*nova\* python-migrate
-
Start the
nova
services:#
openstack-service start nova
-
Re-enable the
nova-compute
service:#
nova service-enable NODENAME nova-compute
5. Upgrade Networking Service
#
openstack-service stop neutron
#
yum -d1 -y upgrade \*neutron\*
#
neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp havana
#
openstack-db --service neutron --update
5.1. Configure Interactions Between Compute and Networking Services
Procedure 3. Configuring neutron and nova interactions
-
Configure
neutron
to notifynova
when the port status is active:#
openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes true
-
Configure
neutron
to use the new address of your Compute API service:#
openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_url http://COMPUTEAPIADDRESS:8774/v2
Replace COMPUTEAPIADDRESS with the host name or IP address of the Compute API service (openstack-nova-api
). In most deployments, this service is also hosted in the Compute controller node. -
Provide
neutron
with the required credentials:#
openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_username NOVAADMIN
#
openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_password NOVAPASSWD
#
openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_tenant_id NOVATENANT
#
openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_auth_url KEYSTONEENDPOINT
Where:-
NOVAADMIN is a user with administrative rights within the OpenStack environment.
Note
The Deploying OpenStack: Learning Environments (Manual Set Up) contains instructions on how to configure Compute service authentication. Here, a user namedcompute
with the roleadmin
and tenantservices
is created specifically for the Compute service. This user is appropriate as NOVAADMIN. -
NOVAPASSWD is the corresponding password of NOVAADMIN.
Note
If you cannot find or determine the password of an existing NOVAADMIN, you can create a new administrative user instead. For instructions, see https://access.redhat.com/articles/1247413. -
NOVATENANT is the UUID of the primary tenant associated with NOVAADMIN.
-
KEYSTONEENDPOINT is the administrative endpoint URL of the
keystone
service.
-
neutron
and nova
interaction, consult the /etc/neutron/neutron.conf.rpmnew
file generated after upgrading OpenStack Networking.
5.2. Migrate OpenStack Networking Agent to ML2
Warning
Procedure 4. Setting up initial ML2 configuration
-
Install the openstack-neutron-ml2 package:
#
yum install openstack-neutron-ml2
-
Create a symbolic link to direct Networking to the ML2 config file
ml2_conf.ini
:#
ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
-
Add the required plug-in settings to the
[ml2]
section of theml2_conf.ini
file:
Table 8. Required plug-in settings for ML2
Configuration option Description mechanism_drivers An ordered, comma-delimited list of networking mechanism driver entrypoints to be loaded from the neutron.ml2.mechanism_drivers namespace. Supported values are local
,flat
,vlan
,gre
, andvxlan
.tenant_network_types An ordered, comma-delimited list of network_types
to allocate as tenant networks. Supported values areopenvswitch
,linuxbridge
, andl2population
.type_drivers Comma-delimited list of network type driver entry points to be loaded from the neutron.ml2.type_drivers namespace. Supported values are the same as mechanism_drivers
. -
Add the required settings for your chosen mechanism driver. Refer to the following table for the required setting/s per network type driver (
type_drivers
):
Table 9. Required settings per driver type
type_driver Section Configuration option Description flat [ml2_type_flat] flat_networks List of physical_network names with which flat networks can be created. Use *
to allow flat networks with arbitraryphysical_network
names.gre [ml2_type_gre] tunnel_id_ranges Comma-separated list of MIN:MAX tunneling tuples enumerating ranges of GRE tunnel IDs that are available for tenant network allocation vlan [ml2_type_vlan] network_vlan_ranges List of physical network names usable for VLAN provider and tenant networks; each name in the list can include ranges of VLAN tags available for allocation to tenant networks. The syntax for each name in the list can be in the form physical_network:vlan_min:vlan_max or physical_network. vxlan (1 of 2) [ml2_type_vxlan] vni_ranges Comma-separated list of vni_min:vni_max tuples enumerating ranges of VXLAN VNI IDs that are available for tenant network allocation vxlan (2 of 2) [ml2_type_vxlan] vxlan_group Multicast group for VXLAN. If unset, disables VXLAN multicast mode.
Note
Procedure 5. Migrating to ML2
-
To migrate the plug-in database using neutron.db.migration.migrate_to_ml2, run:
#
python -m neutron.db.migration.migrate_to_ml2 PLUGIN mysql://DBADMIN:DBPASS@DBHOST/DBNAME
Where:-
PLUGIN is the original plug-in used. Supported values are
linuxbridge
andopenvswitch
. -
DBADMIN is a valid admin username to the OpenStack Networking database (for example,
neutron
). -
DBPASS is the corresponding password of DBADMIN.
-
DBHOST is the IP address of the OpenStack Networking database host.
-
DBNAME is the name of the OpenStack Networking database. By default, PackStack assigns the database name
ovs_neutron
orneutron_linux_bridge
, depending on the plug-in selected.
Note
The actual values used by the Networking service for DBADMIN, DBPASS, DBHOST, and DBNAME are set in the[database]
section of the/etc/neutron/neutron.conf
. Specifically, these values are configured in theconnection
setting with the following syntax:connection = mysql://DBADMIN:DBPASS@DBHOST/DBNAME
Optional parameters include:-
--tunnel-type TUNNEL
: if you configured your plug-in to use tunneling, then use this parameter to also migrate tunneling data across. Replace TUNNEL with the type of tunneling you configured; supported values areGRE
andVXLAN
. -
--vxlan-udp-port VXLANPORT
: if you are migrating VXLAN tunneling data, then you can use this parameter to supply the UDP port that VXLAN tunnels should use (replacing VXLANPORT accordingly). If you do not use this parameter with--tunnel-type VXLAN
, then VXLAN tunnels will use port4789
by default.
-
-
After migrating the plug-in database, manually enable the ML2 plug-in. To do so, first open the
/etc/neutron/neutron.conf
configuration file and set thecore_plugin
parameter to the ML2 plug-in:core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
-
Then, add the L3 router plugin (
neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
) to theservice_plugins
parameter:service_plugins = neutron.services.firewall.fwaas_plugin.FirewallPlugin,neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
Both parameters are in theDEFAULT
section of/etc/neutron/neutron.conf
. When manually setting configuration files, ensure that each line has no whitespace at the beginning; these could cause parsing errors and prevent services from functioning properly.
Note
5.3. Finalize OpenStack Networking Upgrade
Procedure 6. Configuring the OpenStack Networking service to complete its upgrade
-
Set a longer heartbeat for all Networking service agents. The following commands set the recommended values for this release.On the Networking controller node, run:
#
openstack-config --set /etc/neutron/neutron.conf DEFAULT agent_down_time 75
Then, on each node hosting a Networking service agent, run:#
openstack-config --set /etc/neutron/neutron.conf agent report_interval 30
With these settings, the Networking service will check every 30 seconds whether an agent is down; the agent is only declareddown
if it is unreachable for 75 seconds. -
Kill the
dnsmasq
process on the Networking controller node (this is required for the DHCP agent to work):#
killall dnsmasq
-
Restart the OpenStack Networking service to apply all the settings. To do so, run the following command on all nodes hosting Networking services:
#
openstack-service start neutron
6. Configure the Firewall to Allow Console Access to Instances
Note
Procedure 7. Configuring the firewall to allow Compute Service traffic
-
Log in as the
root
user to the Compute node. -
Open the
/etc/sysconfig/iptables
file in a text editor. -
Add an INPUT rule allowing TCP traffic on ports used for console traffic by adding the following line to the file:
-A INPUT -p tcp --dport 5900:6900 -j ACCEPT
The new rule must appear before any INPUT rules that REJECT traffic. -
Save the changes to the
/etc/sysconfig/iptables
file. -
Restart the
iptables
service to ensure that the change takes effect.#
service iptables restart
Note
#
service iptables restart
iptables: Setting chains to policy ACCEPT: mangle nat filte[ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: iptable_nat iptable_filter ip[FAILED]t iptable_filter ip_tables iptables: Applying firewall rules: [ OK ]
7. Manually Configure Dashboard Local Settings
Procedure 8. Manually configuring dashboard local settings for upgrade
-
Back up your existing
/etc/openstack-dashboard/local_settings
file. -
Replace the contents of
/etc/openstack-dashboard/local_settings
with the contents oflocal_settings.rpmnew
. -
Update your new
/etc/openstack-dashboard/local_settings
file with theOPENSTACK_HOST
andSECRET_KEY
values from your backup/etc/openstack-dashboard/local_settings
.If you have any other required, non-default settings from the backup/etc/openstack-dashboard/local_settings
file, copy them to the new/etc/openstack-dashboard/local_settings
file as well. -
If you are running Django 1.5 (or later), you must ensure that there is a correctly configured
ALLOWED_HOSTS
setting in yourlocal_settings
file.ALLOWED_HOSTS
contains a list of host names that can be used to contact your Dashboard service:-
If people will be accessing the Dashboard service using "http://dashboard.example.com", you would set:
ALLOWED_HOSTS=['dashboard.example.com']
-
If you are running the Dashboard service on your local system, you can use:
ALLOWED_HOSTS=['localhost']
-
If people will be using IP addresses instead of, or in addition to, hostnames, an example might be:
ALLOWED_HOSTS=['dashboard.example.com', '192.168.122.200']
Note
For more information about theALLOWED_HOSTS
setting, see the Django Documentation. -
-
Restart the web server to apply all changes:
#
service httpd restart
8. Upgrade the Database Server
Important
Procedure 9. Upgrading the Database Server
-
From each node in your Red Hat Enterprise Linux OpenStack Platform environment, stop all Red Hat Enterprise Linux OpenStack Platform services that access the database server.
-
On the system on which the database server is installed, stop the database server:
#
service mysqld stop
-
Remove the MySQL server package:
#
yum -y remove mysql-server
-
Ensure all packages are up to date:
#
yum -y update
-
Install the MariaDB server packages:
#
yum -y install mariadb-galera-server
-
Configure the database server to start at boot time:
#
chkconfig mysqld on
-
Start the database server:
#
service mysqld start
-
Upgrade the internal
mysql
database schema, entering the password for the administrative user when prompted:#
mysql_upgrade -u root -p
-
From each node in your Red Hat Enterprise Linux OpenStack Platform environment, start all Red Hat Enterprise Linux OpenStack Platform services that access the database server.
9. Unset VIF Plugging Configuration
#
openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_is_fatal false
#
openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_timeout 0
10. Finalize System Upgrade
-
Upgrade the system:
#
yum upgrade
This upgrades the client packages on all of your systems (for example, packages likepython-keystoneclient
orpython-glanceclient
) as well as generally ensuring that you have the appropriate versions of all supporting tools. -
Restart the Compute service (which otherwise will encounter errors due to the upgrade of the Image client package):
#
service openstack-nova-compute restart
-
If this results in a new kernel being installed on your systems, you will probably want to schedule a reboot at some point in the future in order to activate the kernel.
Comments