This is an overview of a simple set of functional tests created in the Hexawise test design tool.
What are we testing in this example?
This Hexawise sample plan tests basic functionality of a simplified flight reservation system.
What are our testing objectives?
We want to test this simple flight reservation process relatively thoroughly.
We know that testing each item in our system once is not sufficient; we know that interactions between the different things in our system could well cause problems. As thoughtful test designers, we want to be smart about testing for potential problems caused by interactions without going off the deep end and trying to test every possible combination.
Hexawise makes it quick and simple for us to select an appropriate set of tests whatever time pressure might exist on the project or whatever testing thoroughness requirements we might have. Hexawise-generated tests automatically maximize variation, maximize testing thoroughness, and minimize wasteful repetition.
What interesting Hexawise features are highlighted in this sample plan description?
This is a simple plan. If you're looking for a complicated sample plan, this ain't it.
What interesting test design considerations are raised in this particular sample plan?
How should we describe the quantity of adult tickets that can be purchased? How many values should we use here? This test design consideration comes up very frequently. How we answer the question has significant implications here (and in many Hexawise test plans).
The drop down menu above shows 6 different quantities. The drop down list under Children includes "zero" as an option as well.
- An obvious solution would be to include separate Values for each of the numbers.
- If we did that though, we would create a plan with at least 6 X 7 = 42 tests.
- An alternative option that we've decided to use here is the idea of Equivalence Classes.
- Here's the basic idea behind equivalence classes: whether there are 2 tickets or 6 tickets probably won't impact the system. Or so everyone tells us. So we decide to describe our system using a smaller number of ticket quantity Values (in our case: 0, 1, and "more than 1"). The equivalence class test design approach has the advantage of keeping the number of tests much lower without sacrificing testing coverage.
It is often useful to start by identifying a verb and a noun for the scope of our tests
Consider additional variation ideas to ask about our verb and noun using "newspaper questions" - who, what, when, why, where, how, how many?
Designing powerful software tests requires people to think carefully about potential inputs into the system being tested. As described in this blog post, we strongly encourage test designers to ask the newspaper questions of who?, what? when? where? why? how? and how many?
- What class of ticket is being reserved? (Coach, Business, First)
- Where is the flight leaving from? (India, the Philippines, the United States)
- Where is the flight going to? (the United States, the Philippines, India)
- How many adult tickets? 1 to 6 --> (1, more than one)
- How many children tickets? 0 to 6 --> (0, 1, more than one)
Parameters and values entered into Hexawise's Inputs screen
Once we have thought through what inputs are important enough to include in your test inputs (and probably excluded things that will not impact how the system being tested operates), Hexawise makes it easy to quickly create powerful tests that will allow us to maximize our test execution efficiency.
Once we enter our test inputs into Hexawise, we simply click on the "Create Tests" button at the top of the screen.
Hexawise helps us identify a set of high priority scenarios within seconds
The coverage achieved in the 10 tests above is known as pairwise testing coverage (or 2-way testing coverage). Hexawise-generated pairwise tests have been proven in many contexts and types of testing to deliver large thoroughness and efficiency benefits.
Hexawise gives test designers control over how thorough they want their testing coverage to be. While this is a very simple plan with only 162 total possible scenarios, in many cases, Hexawise would allow testers to quickly generate dozens, hundreds, or thousands of tests using Hexawise's "coverage dial."
For more detailed explanations describing the approach Hexawise uses to maximize variation, maximize coverage, and minimize wasteful repetition in test sets, please see this image-heavy introductory presentation, this 3-page article on Combinatorial Testing (published by IEEE), and/or this detailed explanation comparing the differences between 2-way coverage and 3-way coverage.
Selecting "3-way interactions" generates a longer set of tests which cover every single possible "triplet" of Values
Hexawise generates and displays this set of 3-way tests to you within a second or two. This 3-way coverage strength would be dramatically more thorough than typical sets of manually selected test scenarios typically used by large global firms when they test their systems.
If a tester spent a couple hours trying to select tests by hand that achieved 100% coverage of every single possible "triplet" of Values (e.g., (i) From the United States, (ii) flying First Class, and (iii) with More than 1 child," the following results would probably occur:
- It would far longer for a tester to try to select a set of tests that would be nearly as thorough
- The tester trying to select tests by hand would create far more than 34 tests (which is the optimized solution, shown above) to achieve this extremely thorough 3-way coverage goal
- Almost certainly, if the tester tried to achieve this coverage goal in 50 or fewer tests, there would be many, many gaps in coverage (e.g., 3-way combinations of Values that the tester accidentally forgot to include)
- Unlike the Hexawise-generated tests, many of the tester's hand-selected scenarios would probably be highly repetitive from one test to the next; that wasteful repetition would result in significant wasted effort in the test execution phase
We could force specific scenarios to appear in tests and/or prevent "impossible to test for" combinations from appearing
We are keeping things simple for this example.
Auto-scripting allows you to turn test scenario tables (from the "Create Tests" screen) into detailed test scripts
The Auto-scripting feature saves testers a lot of time.
You document a single test script in detail from the beginning to the end. As you do so, you indicate where your variables (e.g., "the United States" or "india" or "the Philippines") are in each sentence. That's it. You're ready to export all of your tests.
From there, Hexawise automatically modifies your sample/template test script and inserts the appropriate Values into every test in your plan (whether your plan has 10 tests or 1,000).
We can even add simple Expected Results to your detailed test scripts
If you describe Expected Results like the one above, Hexawise will automatically add Expected Results into every applicable test step in every one of your tests.
Hexawise's Expected Results feature makes it easier to maintain your test sets over time because Expected Results in tests automatically update as test sets get changed over time.
Coverage charts allow teams to make fact-based decisions about "how much testing is enough?"
After the first 23 tests were executed in the "3-way coverage" version of this plan, more than 84% of all possible "triples" would have been tested for.
With Hexawise's matrix reports, we can even tell which specific pairs have not been tested after any test
The matrix coverage chart shows that after the first 8 tests (of the 2-way coverage version of this plan), there are exactly 11 pairs of values that have not been tested together yet. Pairs like "Flying to = India" and "Class = Coach" have not yet been tested in the same test.
If you open up this sample plan and look at the matrix chart view when number of tests is set to 10, you will see that the entire chart is green (because all pairs of values have been tested together at least once in the plan's 10 tests).
Before exporting our tests to discuss them with stakeholders, we review the Test Plan Scorecard
Hexawise's Test Plan Scorecard provides automated analysis to help you review each of your plans.
Mistake Identification: We have helped testers build test plans for years. In doing so, we have seen the most common test design mistakes that new users sometimes make. We have put in automated warnings to alert testers when something looks like it might be wrong with their set of tests.
Feature Usage Ideas: The scorecard identifies features that have not yet been used in your plan. Here, for example, the scorecard highlights that the Value Expansion feature has not been used. A tester clicking on the Learn More link would discover that using the Value Expansion feature would be able to make this particular test plan more thorough than it is without adding any more tests.
Exports from Hexawise can look like this...
You can export from Hexawise in several different formats. Mind maps can be a great option if a tester wants to get quick, actionable guidance from stakeholders about which test inputs should (or should not) be included.
As shown below, detailed test scripts (complete with tester instructions and Expected Results) can be exported also.
The detailed test scripts shown above were created using Hexawise's Auto-Scripts feature.