Tuesday, May 17, 2016

A Guide to Database Service Trove for OpenStack Liberty on Ubuntu 14.04

Let's start with some brief introduction of four components of Trove and their functions in the whole OpenStack environment.

We can image the whole OpenStack as a construction company which helps people build houses. Note that virtual machine servers built by OpenStack are analogous to houses built by this construction company. The components of OpenStack are like departments of the company, which provide different services to make the company run well. The Keystone department is responsible for authorizing customers' identifications. The Glance department is responsible for providing the blueprint of the house. The Nova department is responsible for construction work, The Cinder department is responsible for providing bricks.

In this construction company, we are going to establish a department called Trove. It is responsible for installing an intelligent equipment in the house. It is common sense that people take different responsibilities in a department. So in the department of Trove, there are three full-time guys working there.

Trove-api: He acts like the department front-desk. All requests from customers come to him first. He is responsible for handling and dispatching requests. For example, if a customer requests the service of  installing an intelligent equipment for him or her, but he or she doesn't have a house to place the intelligent equipment and permits Trove department to do everything in order to install that equipment. Trove-api will tell the manager Trove-taskmanager about the request.

Trove-taskmanager: He acts like the department manager.  He has the right to request other departments (Nova, Cinder, Glance, etc) to help him finish the work of building the house. Because in order to install the intelligent equipment, an appropriate house has to be built first. So Trove-taskmanager will ask other departments to help build the house first. Once the house is ready, Trove-taskmanager will get the message. He will hire a new technician guy called Trove-guestagent and send him to go to the house and install the intelligent equipment.

Trove-conductor: He acts like the department secretary. The technician guy Trove-guestagent will update the progress of installation of the intelligent equipment, such as what error he encounters, and whether the installation is completed successfully.

Trove-guestagent: He acts like the technician guy who is responsible for installing the intelligent equipment. He sends the installation progress message to the secretary Trove-conductor.

These four guys talk to each with cell phone, which is the role of Rabbit client in OpenStack.

The below deployment assumes that an OpenStack environment has been set up and runs correctly with services Keystone, Horizon, Glance, Nova, Cinder, and Swift. Note that Swift is optional for Trove deployment unless the backup operation is expected.

Note: Replace capital words like TROVE_DBPASS, NETWORK_LABEL, RABBIT_USERNAME, RABBIT_PASS, TROVE_PASS with appropriate values.

Prerequisites 
  1. Create the database for trove.
    • $ mysql -u root -p
    • > CREATE DATABASE trove;
    • > GRANT ALL PRIVILEGES ON trove.* TO 'trove'@'localhost' IDENTIFIED BY 'TROVE_DBPASS';
    • > GRANT ALL PRIVILEGES ON trove.* TO 'trove'@'%' IDENTIFIED BY 'TROVE_DBPASS';
    • > FLUSH PRIVILEGES;
  2. Source the admin credentials of OpenStack environment.
    • $ source admin-openrc.sh
  3. Create the service credentials for trove.
    • $ openstack user create --domain default --password-prompt trove
    • $ openstack role add --project service --user user trove admin
    • $ openstack service create --name trove --description "OpenStack Database Service" database
  4. Create the database service endpoints.
    • $ openstack endpoint create --region RegionOne database public http://controller:8779/v1.0/%\(tenant_id\)s 
    • $ openstack endpoint create --region RegionOne database internal http://controller:8779/v1.0/%\(tenant_id\)s 
    • $ openstack endpoint create --region RegionOne database admin http://controller:8779/v1.0/%\(tenant_id\)s 
Install and Configure Components
  1. Install the packages.
    • # apt-get install python-trove python-troveclient trove-common trove-api trove-taskmanager trove-conductor
  2. Edit the /etc/trove/trove.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • rpc_backend = rabbit
      • auth_strategy = keystone
      • trove_auth_url = http://controller:5000/v2.0
      • nova_compute_url = http://controller:8774/v2
      • cinder_url = http://controller:8776/v2
      • neutron_url = http://controller:9696/
      • add_addresses = True
      • network_label_regex = ^NETWORK_LABEL$
      • log_dir = /var/log/trove/
      • log_file = trove-api.log
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
    • [keystone_authtoken]
      • auth_uri = http://controller:5000
      • auth_url = http://controller:35357
      • auth_plugin = password
      • project_domain_id = default
      • user_domain_id = default
      • project_name = service
      • username = trove
      • password = TROVE_PASS
  3. Edit the /etc/trove/trove-taskmanager.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • rpc_backend = rabbit
      • trove_auth_url = http://controller:5000/v2.0
      • nova_compute_url = http://controller:8774/v2
      • cinder_url = http://controller:8776/v2
      • neutron_url = http://controller:9696/
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • taskmanager_manager = trove.taskmanager.manager.Manager
      • add_addresses = True
      • network_label_regex = ^NETWORK_LABEL$
      • log_dir = /var/log/trove/
      • log_file = trove-taskmanager.log
      • guest_config = /etc/trove/trove-guestagent.conf
      • guest_info = guest_info.conf
      • injected_config_location = /etc/trove/conf.d
      • cloudinit_location = /etc/trove/cloudinit
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  4. Edit the /etc/trove/trove-conductor.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-conductor.log
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  5. Edit the /etc/trove/trove-guestagent.conf file.
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-guestagent.log
      • datastore_registry_ext = vertica:trove.guestagent.datastore.experimental.vertica.manager.Manager
      • (Based on the notes above datastore_registry_ext in the configuration file, add datastore you want.)
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  6. Edit the /etc/init/trove-taskmanager.conf file.
    • --exec /usr/bin/trove-taskmanager -- --config-file=/etc/trove/trove-taskmanager.conf ${DAEMON_ARGS}
  7. Edit the /etc/init/trove-conductor.conf file.
    • --exec /usr/bin/trove-conductor -- --config-file=/etc/trove/trove-conductor.conf ${DAEMON_ARGS}
  8. Populate the trove service database.
    •  # trove-manage db_sync
Finalize Installation
  1. Restart the database services.
    • # service trove-api restart
    • # service trove-taskmanager restart
    • # service trove-conductor restart
  2. Remove the SQLite database file.
    • # rm -f /var/lib/trove/trove.sqlite


Configure the database guest. 
  1. Edit the /etc/trove/trove-guestagent.conf file.
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-guestagent.log
      • datastore_registry_ext = vertica:trove.guestagent.datastore.experimental.vertica.manager.Manager
      • (Based on the notes above datastore_registry_ext in the configuration file, add datastore you want.)
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = openstack
      • rabbit_password = RABBIT_PASS
  2. Edit the /etc/init/trove-guestagent.conf file.
    • --exec /usr/bin/trove-guestagent -- --config-file=/etc/trove/conf.d/guest_info.conf --config-file=/etc/trove/trove-guestagent.conf ${DAEMON_ARGS}
Debug Common Problems.

Read logs and find out specific information. The first place to check is trove-taskmanager.log on the controller node and then trove-guestagent.log on the guest.

  1. No corresponding Nova instances are launched.
    • Check the configuration values for nova_proxy_admin_user, nova_proxy_admin_pass, nova_proxy_admin_tenant_name, nova_proxy_admin_tenant_id.
  2. Trove taskmanager, trove conductor, and trove guestagent cannot talk to each other via RPC.
    • Check their logs to see whether there is "connected to AMQP". If there is no such message, check the configuration values of rabbit_host, rabbit_userid, rabbit_password, and rabbit_backend.
  3. The guest_info.conf cannot be injected to the guest.
    1. 1. Install the following packages on compute nodes

      # apt-get install libguestfs-tools python-libguestfs
      # sudo update-guestfs-appliance
      # sudo usermod -a -G kvm yourlogin
      # chmod 0644 /boot/vmlinuz*

      2. Add the following configuration options in /ect/nova/nova-compute.conf on
      compute nodes.

      [DEFAULT]
      compute_driver=libvirt.LibvirtDriver
      rootwrap_config = /etc/nova/rootwrap.conf

      [libvirt]
      virt_type=qemu
      inject_partition = -1

      3. Restart nova-compute service on compute nodes

      # service nova-compute restart
  4. Trove guestagent cannot get guest id from guest_info.conf and Trove instance is stuck at 'BUILD' status forever, but guest info is already injected to the guest.
    • Check /etc/init/trove-guestagent.conf file on the guest and make /etc/trove/conf.d/guest_info.conf as one of configuration files.