Tuesday, May 17, 2016

A Guide to Database Service Trove for OpenStack Liberty on Ubuntu 14.04

Let's start with some brief introduction of four components of Trove and their functions in the whole OpenStack environment.

We can image the whole OpenStack as a construction company which helps people build houses. Note that virtual machine servers built by OpenStack are analogous to houses built by this construction company. The components of OpenStack are like departments of the company, which provide different services to make the company run well. The Keystone department is responsible for authorizing customers' identifications. The Glance department is responsible for providing the blueprint of the house. The Nova department is responsible for construction work, The Cinder department is responsible for providing bricks.

In this construction company, we are going to establish a department called Trove. It is responsible for installing an intelligent equipment in the house. It is common sense that people take different responsibilities in a department. So in the department of Trove, there are three full-time guys working there.

Trove-api: He acts like the department front-desk. All requests from customers come to him first. He is responsible for handling and dispatching requests. For example, if a customer requests the service of  installing an intelligent equipment for him or her, but he or she doesn't have a house to place the intelligent equipment and permits Trove department to do everything in order to install that equipment. Trove-api will tell the manager Trove-taskmanager about the request.

Trove-taskmanager: He acts like the department manager.  He has the right to request other departments (Nova, Cinder, Glance, etc) to help him finish the work of building the house. Because in order to install the intelligent equipment, an appropriate house has to be built first. So Trove-taskmanager will ask other departments to help build the house first. Once the house is ready, Trove-taskmanager will get the message. He will hire a new technician guy called Trove-guestagent and send him to go to the house and install the intelligent equipment.

Trove-conductor: He acts like the department secretary. The technician guy Trove-guestagent will update the progress of installation of the intelligent equipment, such as what error he encounters, and whether the installation is completed successfully.

Trove-guestagent: He acts like the technician guy who is responsible for installing the intelligent equipment. He sends the installation progress message to the secretary Trove-conductor.

These four guys talk to each with cell phone, which is the role of Rabbit client in OpenStack.

The below deployment assumes that an OpenStack environment has been set up and runs correctly with services Keystone, Horizon, Glance, Nova, Cinder, and Swift. Note that Swift is optional for Trove deployment unless the backup operation is expected.

Note: Replace capital words like TROVE_DBPASS, NETWORK_LABEL, RABBIT_USERNAME, RABBIT_PASS, TROVE_PASS with appropriate values.

Prerequisites 
  1. Create the database for trove.
    • $ mysql -u root -p
    • > CREATE DATABASE trove;
    • > GRANT ALL PRIVILEGES ON trove.* TO 'trove'@'localhost' IDENTIFIED BY 'TROVE_DBPASS';
    • > GRANT ALL PRIVILEGES ON trove.* TO 'trove'@'%' IDENTIFIED BY 'TROVE_DBPASS';
    • > FLUSH PRIVILEGES;
  2. Source the admin credentials of OpenStack environment.
    • $ source admin-openrc.sh
  3. Create the service credentials for trove.
    • $ openstack user create --domain default --password-prompt trove
    • $ openstack role add --project service --user user trove admin
    • $ openstack service create --name trove --description "OpenStack Database Service" database
  4. Create the database service endpoints.
    • $ openstack endpoint create --region RegionOne database public http://controller:8779/v1.0/%\(tenant_id\)s 
    • $ openstack endpoint create --region RegionOne database internal http://controller:8779/v1.0/%\(tenant_id\)s 
    • $ openstack endpoint create --region RegionOne database admin http://controller:8779/v1.0/%\(tenant_id\)s 
Install and Configure Components
  1. Install the packages.
    • # apt-get install python-trove python-troveclient trove-common trove-api trove-taskmanager trove-conductor
  2. Edit the /etc/trove/trove.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • rpc_backend = rabbit
      • auth_strategy = keystone
      • trove_auth_url = http://controller:5000/v2.0
      • nova_compute_url = http://controller:8774/v2
      • cinder_url = http://controller:8776/v2
      • neutron_url = http://controller:9696/
      • add_addresses = True
      • network_label_regex = ^NETWORK_LABEL$
      • log_dir = /var/log/trove/
      • log_file = trove-api.log
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
    • [keystone_authtoken]
      • auth_uri = http://controller:5000
      • auth_url = http://controller:35357
      • auth_plugin = password
      • project_domain_id = default
      • user_domain_id = default
      • project_name = service
      • username = trove
      • password = TROVE_PASS
  3. Edit the /etc/trove/trove-taskmanager.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • rpc_backend = rabbit
      • trove_auth_url = http://controller:5000/v2.0
      • nova_compute_url = http://controller:8774/v2
      • cinder_url = http://controller:8776/v2
      • neutron_url = http://controller:9696/
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • taskmanager_manager = trove.taskmanager.manager.Manager
      • add_addresses = True
      • network_label_regex = ^NETWORK_LABEL$
      • log_dir = /var/log/trove/
      • log_file = trove-taskmanager.log
      • guest_config = /etc/trove/trove-guestagent.conf
      • guest_info = guest_info.conf
      • injected_config_location = /etc/trove/conf.d
      • cloudinit_location = /etc/trove/cloudinit
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  4. Edit the /etc/trove/trove-conductor.conf file.
    • [database]
      • connection = mysql+pymysql://trove:TROVE_DBPASS@controller/trove
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-conductor.log
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  5. Edit the /etc/trove/trove-guestagent.conf file.
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-guestagent.log
      • datastore_registry_ext = vertica:trove.guestagent.datastore.experimental.vertica.manager.Manager
      • (Based on the notes above datastore_registry_ext in the configuration file, add datastore you want.)
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = RABBIT_USERNAME
      • rabbit_password = RABBIT_PASS
  6. Edit the /etc/init/trove-taskmanager.conf file.
    • --exec /usr/bin/trove-taskmanager -- --config-file=/etc/trove/trove-taskmanager.conf ${DAEMON_ARGS}
  7. Edit the /etc/init/trove-conductor.conf file.
    • --exec /usr/bin/trove-conductor -- --config-file=/etc/trove/trove-conductor.conf ${DAEMON_ARGS}
  8. Populate the trove service database.
    •  # trove-manage db_sync
Finalize Installation
  1. Restart the database services.
    • # service trove-api restart
    • # service trove-taskmanager restart
    • # service trove-conductor restart
  2. Remove the SQLite database file.
    • # rm -f /var/lib/trove/trove.sqlite


Configure the database guest. 
  1. Edit the /etc/trove/trove-guestagent.conf file.
    • [DEFAULT]
      • verbose = True
      • debug = True
      • trove_auth_url = http://controller:5000/v2.0
      • nova_proxy_admin_user = admin
      • nova_proxy_admin_pass = admin
      • nova_proxy_admin_tenant_name = admin
      • nova_proxy_admin_tenant_id = *************
      • rpc_backend = rabbit
      • log_dir = /var/log/trove/
      • log_file = trove-guestagent.log
      • datastore_registry_ext = vertica:trove.guestagent.datastore.experimental.vertica.manager.Manager
      • (Based on the notes above datastore_registry_ext in the configuration file, add datastore you want.)
    • [oslo_messaging_rabbit]
      • rabbit_host = controller
      • rabbit_userid = openstack
      • rabbit_password = RABBIT_PASS
  2. Edit the /etc/init/trove-guestagent.conf file.
    • --exec /usr/bin/trove-guestagent -- --config-file=/etc/trove/conf.d/guest_info.conf --config-file=/etc/trove/trove-guestagent.conf ${DAEMON_ARGS}
Debug Common Problems.

Read logs and find out specific information. The first place to check is trove-taskmanager.log on the controller node and then trove-guestagent.log on the guest.

  1. No corresponding Nova instances are launched.
    • Check the configuration values for nova_proxy_admin_user, nova_proxy_admin_pass, nova_proxy_admin_tenant_name, nova_proxy_admin_tenant_id.
  2. Trove taskmanager, trove conductor, and trove guestagent cannot talk to each other via RPC.
    • Check their logs to see whether there is "connected to AMQP". If there is no such message, check the configuration values of rabbit_host, rabbit_userid, rabbit_password, and rabbit_backend.
  3. The guest_info.conf cannot be injected to the guest.
    1. 1. Install the following packages on compute nodes

      # apt-get install libguestfs-tools python-libguestfs
      # sudo update-guestfs-appliance
      # sudo usermod -a -G kvm yourlogin
      # chmod 0644 /boot/vmlinuz*

      2. Add the following configuration options in /ect/nova/nova-compute.conf on
      compute nodes.

      [DEFAULT]
      compute_driver=libvirt.LibvirtDriver
      rootwrap_config = /etc/nova/rootwrap.conf

      [libvirt]
      virt_type=qemu
      inject_partition = -1

      3. Restart nova-compute service on compute nodes

      # service nova-compute restart
  4. Trove guestagent cannot get guest id from guest_info.conf and Trove instance is stuck at 'BUILD' status forever, but guest info is already injected to the guest.
    • Check /etc/init/trove-guestagent.conf file on the guest and make /etc/trove/conf.d/guest_info.conf as one of configuration files.

11 comments:

  1. hi
    用户私有网络需要跟rabbitmq网络能通吗

    ReplyDelete
    Replies
    1. Hi,

      Technically, they don't have to communicate with each other. It depends on how we configure them. In this post, I only used one network interface, so they were in the same network. But I guess in the production environment, they should be configured in different networks for the security reason.

      Please refer to the network planning and configuration of an OpenStack environment in the OpenStack installation guide. http://docs.openstack.org

      For example, the Liberty version of OpenStack installation guide can be found below.

      Architecture and networking overview: http://docs.openstack.org/liberty/install-guide-ubuntu/overview.html
      Host networking: http://docs.openstack.org/liberty/install-guide-ubuntu/environment-networking.html

      Delete
    2. Thank you, I now environment is the rabbitmq network is isolated and alone now create an instance of the state is runing in nova but in trove state has been building, can you leave a contact number?

      Delete
    3. This comment has been removed by the author.

      Delete
    4. I build new image, and image installed inside the trove - guestagent, will also be trove - guestagent. Conf configuration file is written on the inside

      Delete
    5. 绍远你好,请问你的mysql镜像成功了吗,能否加QQ讨论下,我的是532741480

      Delete
  2. I found this blog very helpful and easy to understand. Thanks for sharing helpful guide.
    OpenStack Trove API

    ReplyDelete
  3. Hi Lili , First congratulation with all information . I have a question . Why I can not left virt_type = kvm . Will The inject only work with virt_type = qemu ?

    ReplyDelete

Note: Only a member of this blog may post a comment.