Chapter 3. OpenStack Integration Test Suite Tests
The OpenStack Integration Test Suite is designed to be useful for a large number of different environments, for example, gating commits to OpenStack core projects, to validate OpenStack cloud implementations for both correctness, as well as a burn in tool for OpenStack clouds. As such the OpenStack Integration Test Suite tests come in many flavors, each with their own rules and guidelines. The different types of tests are as follows:
- Scenario Tests
- API Tests
- Stress Tests
While these tests are run against your OpenStack cloud deployment, the OpenStack Integration Test Suite also provides
unit tests that you can run to verify your
tempest implementation. The following sections briefly describes each of the tests and how you can implement them.
3.1. Scenario Tests
Scenario tests are through path tests of OpenStack function. They are typically a series of steps where complicated state requiring multiple services is set up exercised, and torn down. They ideally involve the integration between multiple OpenStack services to exercise the touch points between them.
Scenario tests should not use the existing python clients for OpenStack, but should instead use the
tempest implementations of clients.
Any scenario test should have a real-life use case. For example, as an operator I want to start with a blank environment and then perform the following functions:
- Upload an Image Service image
- Deploy an instance from it
Log in to the guest using
- Create a snapshot of the instance
Scope - Scenario tests used for testing the integration between OpenStack projects is the core purpose of the OpenStack Integration Test Suite. Scenario tests should always use the OpenStack Integration Test Suite implementation of the OpenStack API, to ensure that bugs are not hidden by the official clients. Tests should be tagged with which services they exercise, as determined by which client libraries are used directly by the test.
Example of a good scenario test - Be specific in the interactions, while testing for an interaction between two or more services. A giant “this is my data center” smoke test is hard to debug when it goes wrong. A flow of interactions between Image Service and Compute, as mentioned in the introduction, is a good example. Especially if it involves a repeated interaction when a resource is setup, modified, detached, and then reused later again.
3.2. API tests
API tests are validation tests for the OpenStack API.
API tests should not use the existing python clients for OpenStack, but should instead use the
tempest implementations of clients.
This allows you to test JSON. Having raw clients also lets you pass invalid JSON to the APIs and see the results, something you could not get with the native clients.
Scope - API tests should always use the OpenStack Integration Test Suite implementation of the OpenStack API, to ensure that bugs are not hidden by the official clients. They should test specific API calls, and can build up complex state if it is needed for the API call to be meaningful. They should send not only good data, but bad data at the API and look for error codes. They should all be able to be run on their own, not depending on the state created by a previous test.
3.3. Stress Tests
Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks. The stress test framework runs several test jobs in parallel and can run any existing test in the OpenStack Integration Test Suite as a stress job.
Environment - This particular framework assumes your working Compute cluster understands
nova API 2.0. The stress tests can read the logs from the cluster. To enable this, you have to provide the hostname to call
nova-manage and the private key and user name for
ssh to the cluster in the
[stress] section of
tempest.conf. You also need to provide the location of the log files:
target_logfiles = “regexp to all log files to be checked for errors” target_private_key_path = “private ssh key for controller and log file nodes” target_ssh_user = “username for controller and log file nodes” target_controller = “hostname or ip of controller node (for nova-manage) log_check_interval = “time between checking logs for errors (default 60s)”
To activate logging on your console, make sure that you activate
tempest.conf or use the default
Running default stress test set - The stress test framework can automatically discover test inside the OpenStack Integration Test Suite. All test flags with the
@stresstest decorator will be executed. In order to use this discovery, you have to install the OpenStack Integration Test Suite CLI, navigate to the
root directory and execute the following:
# tempest run-stress -a -d 30
Run the sample test to test installation:
# tempest run-stress -t tempest/stress/etc/server-create-destroy-test.json -d 30
This sample test tries to create a few virtual machines and kill a few virtual machines.
Sometimes the tests don’t finish, or there are failures. In these cases, you may want to clean out the
nova cluster. There are some scripts provided in the tools subdirectory. You can use the following script to destroy any keypairs, floating IP addresses, and servers:
3.4. Unit Tests
Unit tests are the self checks for the OpenStack Integration Test Suite. They provide functional verification and regression checking for the internal components of
tempest. They should be used to just verify that the individual pieces of the OpenStack Integration Test Suite are working as expected. These tests can be run by specifying the test discovery path as follows:
# OS_TEST_PATH=./tempest/tests testr run --parallel
By setting the
./tempest/tests, you are specifying that the test discover should only be run on the unit test directory. The default value of
OS_TEST_PATH=./tempest/test_discover which will only run test discover on the OpenStack Integration Test Suite.
Alternatively, you can use the
run_tests.sh script which will create a
venv and run the unit tests. There are also the
py34 tox jobs which will run the unit tests with the corresponding version of Python.
Scope - The unit tests exist to make sure that the mechanisms that used in the OpenStack Integration Test Suite to are valid and remain functional. They are only for self validation of the OpenStack Integration Test Suite. Unit tests should not require an external service to be running or any extra configuration to run. Any state that is required for a test should either be mocked out or created in a temporary test directory. (see test_wrappers.py for an example of using a temporary test directory)