Based on some of the comments that were posted on the recent “Which Program for Rally” ML thread I feel that there’s been some continued confusion around exactly how all the  projects work together in the QA program. So after discussing it with a wise council of my elders, I decided to start a blog so that I had a place to post more details and I could give a high level overview and clarify how everything works. I’m not really sure how much I’ll be using this blog in the future, as having one is something I’ve resisted for quite some time. But, I felt that making this post warranted me giving in to peer pressure.

Today’s the QA program :

So today in the QA program we have 3 projects, here is a high level over:

• Tempest: The OpenStack Integrated test suite, it’s concerned with just  with having tests and running them
• devstack: A documented shell script to build complete OpenStack development environments.
• grenade: Upgrade testing using 2 versions of devstack to test an offline upgrade. Tempest can optionally be used after each version is deployed to verify the cloud.

Each of these projects is independent and is useful by itself. They have  defined scope (which admittedly gets blurred and constantly evolves) and  when used together along with external tools they can be used in different pipelines for certain goals.

Then there are several external projects which consume and use these for different configurations. Here are just a few examples:

Normal Gating Jobs:

Upgrade Test Gating Jobs:

Cloud burn-in/stress testing:

Cloud Monitoring:

These are just a few of the example use cases that I know of, and I’m sure  there are more in the wild. (for example the monitoring one I only found out  about in Atlanta)

My Current Vision of the Future:

Let me preface this with: This is just my personal view of where I want to see things moving long term. (at least for today, I’m sure my mind will change in the future) It’s just something that I’ve been spinning around in the back of  my head. Everything here is hypothetical for the most part, except where implementation has already started.

Where I see the project list moving in the long term is something along the lines of:

• Tempest Library
• Tempest
• Devstack
• Grenade
• Subunit2sql
• Javelin2
• Stress test tool
• Analysis/post-processing utility,

The increased number of utilities, and libraries allow for more diverse configurations, and combinations besides allowing us to have more targetted development for each repository. Which should hopefully lead to a faster  path to contribution for everyone by making everything simpler.

A brief overview of what all the new projects are:

• tempest library: This is my current big TODO in the next few weeks, the spec for this can be found here
• subunit2sql: this already actually exists  but the basic intent is to provide a utility and library for the storage of subunit streams. One of my TODOs here is to also integrate it into testrepository so it can be used directly to store testr results in a SQL DB.
• javelin2: This will be an externalized version of this
• Stress test tool: This is will be an externalization of the current stress test framework here and it’s entry point
• Analysis/Post-processing utility: This is something that is used for both visualizing performance data from a subunit stream as well as providing some analysis and provide useful insight from larger sources of test output. (like a subunit2sql DB)

This leaves tempest to really be just the tests. The common functionality, like  the unified REST client, are part of the library which allows reuse by other  external tools. This allows people to use all the work we’ve been doing in tempest around building a black box integrated test suite to build their own functional test suites (which was a motivation for starting the library now) or whatever other purpose they have. As part of breaking the functionality out  we make a number of small utilities that are currently in the tempest repo  that were only there because they were tied to the common code in  tempest. Part of the current issue we’re seeing with tempest is because a  large portion of the common functionality in the tree is not really modular  we have things in tree which aren’t necessarily related to integrated testing.  (ie, javelin2, run_stress) Also, using tempest as a base for testing in a different  repo is very difficult. I have seen a couple cases where other projects want to spin up a functional test suite similar to tempest with the hopes of converging down the road. But, this is very difficult to accomplish today because nothing in tempest is really modular or provides a stable interface.

Also, having a leaner tempest should make it a bit easier to concentrate on some of the usability gaps which have been pointed out in this thread, and things that were already on our radar which we’ve been discussing. Improving the output and results view are definitely big ones on this list. Another possibility might be creating a tempest CLI instead of just calling tox or run_tempest.sh. Which is something that we’ve discussed in the past.

Another key area were I expect to see improvements is the instrumenation of tempest. Right when it comes to metrics around performance of anything that tempest is doing we have only rough idea of the execution time. For example, through the subunit stream we know how much time a test case takes to execute, but the setUpClass is not captured in the subunit data. Or for API calls we meaure the time it takes for the tempest rest_client’s request method to finish executing. But, this doesn’t necessarily give a complete picture, especially around async requests. To use tempest as a load generator for non-functional testing fixing these issues becomes important. I expect after libification of parts of tempest, primarily the rest client, making these improvements will become simpler and be prioritized.

With this new constellation of smaller more logically split projects we have additional modularity and each component is useful on its own. In conjunction they can be used to build even more customized pipelines. It’s really just an expansion of the model we currently have, but making the  logical splits a bit more clearly. I also expect that additional projects will be created and added as we move forward. Which is a major advantage with  using the Unix philosophy as that things can grow organically and build off  of each other.

Here are some example configurations:

Gating:

Upgrade Gating:

Then from the gating runs we can use the data stored in the database in conjunction with the analysis tool to visualize long term trends in  performance data. As well as use it to track general testing trends over time. Having the data in a DB also allows to build tooling to query it to find other useful trends or information. Which is something we’re lacking today.

Cloud Burn-in/Stress testing:

Cloud Monitoring:

So, this one is basically the same, but I want to point out that in conjunction with some of the other tools this could be a bit more custom. Like using subunit2sql as I outlined for the gating examples to track trends over the lifetime of the cloud.

Local Benchmarking:

The real advantage here is the flexibility this approach allows. These configurations can either use or not use any of the other pieces to build a custom pipeline for testing depending on the requirements.

I feel that we’re not all that far off from something like this. We’re starting the first steps working towards this now. But, I do think it will probably be at least 2 cycles before we have all the pieces as I outlined here.