Scenario tests are through path tests of OpenStack function. They are typically a series of steps where complicated state requiring multiple services is set up exercised, and torn down. They ideally involve the integration between multiple OpenStack services to exercise the touch points between them.

Any scenario test should have a real-life use case. For example, as an operator I want to start with a blank environment and then perform the following functions:

Scope - Scenario tests used for testing the integration between OpenStack projects is the core purpose of the OpenStack Integration Test Suite. Scenario tests should always use the OpenStack Integration Test Suite implementation of the OpenStack API, to ensure that bugs are not hidden by the official clients. Tests should be tagged with which services they exercise, as determined by which client libraries are used directly by the test.

Example of a good scenario test - Be specific in the interactions, while testing for an interaction between two or more services. A giant “this is my data center” smoke test is hard to debug when it goes wrong. A flow of interactions between Image Service and Compute, as mentioned in the introduction, is a good example. Especially if it involves a repeated interaction when a resource is setup, modified, detached, and then reused later again.

API tests are validation tests for the OpenStack API.

This allows you to test JSON. Having raw clients also lets you pass invalid JSON to the APIs and see the results, something you could not get with the native clients.

Scope - API tests should always use the OpenStack Integration Test Suite implementation of the OpenStack API, to ensure that bugs are not hidden by the official clients. They should test specific API calls, and can build up complex state if it is needed for the API call to be meaningful. They should send not only good data, but bad data at the API and look for error codes. They should all be able to be run on their own, not depending on the state created by a previous test.

Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks. The stress test framework runs several test jobs in parallel and can run any existing test in the OpenStack Integration Test Suite as a stress job.

Environment - This particular framework assumes your working Compute cluster understands nova API 2.0. The stress tests can read the logs from the cluster. To enable this, you have to provide the hostname to call nova-manage and the private key and user name for ssh to the cluster in the [stress] section of tempest.conf. You also need to provide the location of the log files:

target_logfiles = “regexp to all log files to be checked for errors” target_private_key_path
= “private ssh key for controller and log file nodes” target_ssh_user = “username for
controller and log file nodes” target_controller = “hostname or ip of controller node (for
nova-manage) log_check_interval = “time between checking logs for errors (default 60s)”

To activate logging on your console, make sure that you activate use_stderr in tempest.conf or use the default logging.conf.sample file.

Running default stress test set - The stress test framework can automatically discover test inside the OpenStack Integration Test Suite. All test flags with the @stresstest decorator will be executed. In order to use this discovery, you have to install the OpenStack Integration Test Suite CLI, navigate to the tempest root directory and execute the following:

# tempest run-stress -a -d 30

Run the sample test to test installation:

# tempest run-stress -t tempest/stress/etc/server-create-destroy-test.json -d 30

This sample test tries to create a few virtual machines and kill a few virtual machines.

Sometimes the tests don’t finish, or there are failures. In these cases, you may want to clean out the nova cluster. There are some scripts provided in the tools subdirectory. You can use the following script to destroy any keypairs, floating IP addresses, and servers:

# tempest/stress/tools/cleanup.py

Unit tests are the self checks for the OpenStack Integration Test Suite. They provide functional verification and regression checking for the internal components of tempest. They should be used to just verify that the individual pieces of the OpenStack Integration Test Suite are working as expected. These tests can be run by specifying the test discovery path as follows:

# OS_TEST_PATH=./tempest/tests testr run --parallel

By setting the OS_TEST_PATH to ./tempest/tests, you are specifying that the test discover should only be run on the unit test directory. The default value of OS_TEST_PATH is OS_TEST_PATH=./tempest/test_discover which will only run test discover on the OpenStack Integration Test Suite.

Alternatively, you can use the run_tests.sh script which will create a venv and run the unit tests. There are also the py27 and py34 tox jobs which will run the unit tests with the corresponding version of Python.

Scope - The unit tests exist to make sure that the mechanisms that used in the OpenStack Integration Test Suite to are valid and remain functional. They are only for self validation of the OpenStack Integration Test Suite. Unit tests should not require an external service to be running or any extra configuration to run. Any state that is required for a test should either be mocked out or created in a temporary test directory. (see test_wrappers.py for an example of using a temporary test directory)