I. Step One - Capture ~ 5 Screen Shots from the Application You are Testing (if Available)
Two illustrative screen shots are included below.
II. Step Two - Identify a Verb and a Noun that Best Describes the Scope of Your Set of Tests
In this example, "Trade" would make an appropriate Verb and "Stocks" would be an appropriate Noun.
... And Ask "Newspaper Reporter Questions" to Identify Potentially Important Variation Ideas to Include in Your Set of Tests.
Designing powerful software tests requires that people think carefully about potential inputs into the system being tested. And how those potential inputs might impact the behavior of the system. As described in this blog post, we strongly encourage test designers to start with a verb an a noun to frame a sensible scope for a set of tests and then ask the "newspaper reporter" questions of who? what? when? where? why? how? and how many?
- Who trades the stocks? (e.g., what is the user type?)
- When will they trade stock? (e.g., good-till-date used)
- When was the order created? (e.g., trade a previously saved order)
How / How Many
- How large of a transaction will they trade?
- How do they trade stock? (online, call in, etc.)
- How many transactions are placed per order?
What / What Kind
- What authorizations do they have to trade stock?
- What kind of changes can be made to an existing order?
- What kind of order is it? (buy or sell)
- What type of order is it? (Market or Limit order)
- Where do they trade stock? (e.g., what country is the exchange located in)
Make Some Educated Guesses about What Potentially-Significant Variation Ideas Might be. And What Kinds of Variation Ideas are NOT Important Enough to Include.
This process of designing tests is VERY different than the test selection process that most testers are used to. One of the biggest differences is that testers using traditional test design approaches will, more often than not, make an assumption that most variation ideas discussed above probably do not matter. After all, they might think, if such variation ideas mattered, wouldn't someone on the team tell the testers that they were important to vary?
The fact is, most testers using traditional approaches to test case selection fail to include variation ideas that address who? what? when? where? why? how? or how many? their tests. And as a result, the tests they select too often result in tests on the left hand of this spectrum...
Consider the Difference between "Active" and "Passive" Inputs When Determining what Variation Ideas are Potentially-Significant.
Active Inputs are essential to implementing an effective test design using Hexawise. It is important to include these types of inputs in your design to increase the amount of coverage achieved by the auto-generated tests. Active inputs are derived from asking newspaper questions regarding the verb-noun pair that is in consideration. Passive Inputs are also derived from these type of questions. Understanding the difference between the two is critical in achieving maximum coverage without creating test cases for specific details that often have no effect on the System Under Test. The same newspaper question may lead to both active inputs and passive inputs. For example, "Who trades the stock?" There are multiple answers to this question depending on the context and level of specitivity. One acceptable answer relates to the type of account the user has. The user could be trying to trade stocks using a cash account, margin account, or an option account. Another acceptable answer hits on a more specific level of detail, first and last name. In order to assess which category, active or passive, a parameter belongs to, it is helpful to begin to make educated guesses on the impact each different value related to a parameter may have on the system. To clarify, assume we have just tested the pairwise combination James Johnson and Cash Account. I will ask you to consider the following question:
Which combination of values will likely cause the System Under Test to behave most differently?
A) Matthew Williams and Cash Account
B) James Johnson and Margin Account
If you are still unsure of the answer, step back and think to yourself the underlying processes and business rules associated with these parameters and their different values. Would the system more likely perform differently if a user had changed his/her name or would it perform more differently if the user had changed the type of account he/she was trading with? It is most probable that the system's logic will vary more depending on the type of account the trader is using rather than the name of the trader. It is often true that the parameters that tend to have a much larger impact on the logic of the System Under Test are considered active inputs. Active inputs must be included in your test design on the Define Inputs screen. Conversely, the parameters that tend to have little to no effect on the system are usually considered passive inputs. These inputs should not be included in your test design as they will have significant effects on the number of total possible tests as well as the number of tests generated by Hexawise.
Examples of Active Inputs: Stock Exchange = NYSE vs. LSE, Transaction Type: Buy vs. Sell, Type of Order = Market vs. Limit, Account Type = Cash vs. Margin vs. Option
Examples of Passive Inputs: First Name, Last Name, Comment Field, Phone Number
Deciding which parameters are active or passive play a very important role in test design. Therefore, it is important to carefully think through these inputs and assign them to one of the two categories. While these decisions are binary, meaning they are either active or passive, it may be difficult to be 100% confident when classifying these inputs. We say an input is ambiguous when it is difficult to categorize an input as active or passive. In the case of an ambiguous input it is best to include the input with a question mark in its parameter name. Be sure to note these uncertainties as you come across them and discuss these uncertainties with a Subject Matter Expert to gain further insight as to whether the inputs in question are highly likely (active) or highly unlikely (passive) to affect the performance of the system.
Examples of Inputs "Ambiguous" Inputs: Size of Transaction? = Very Small vs. Very Large, Location of Counter-Party? = Same Country vs. Different Country, Time of Day? = After Market Open vs. Before Market Close
III. Step Three: Enter Parameters and Values into Hexawise
IV. Step Four: Remove Impossible to Test For Combinations
Let's say that there are business rules regarding which users are allowed to trade using a Margin Account. For example, users who have been trading with this broker for less than a month (User Type = New) are not allowed to open a Margin Account. Under this rule, it would be impossible to test User Type = New and Account Type = Margin Account. To prevent this combination from being generated, and to further maximize coverage, a smart test designer would implement an invalid pair between these two values. To do this, hover your cursor over the table entry containing one of the two values to be part of the invalid pair. The background color of the table entry will change to orange and there will be an icon on each side of the written value: an 'X' and two arrows. To begin the process of creating an invalid pair, click the 'X' next to the first value. To finish creating the invalid pair, find the second value and click the 'X' beside it as well.
You can refer to the help manual to learn how invalid pairs work in more detail here.
V. Step Five: Force "The Most Common Use Case" to appear in your tests.
After defining all the input parameters and values related to the System Under Test, clicking the "Create Tests" button, it is possible that a specific test case for a high-priority scenario may not have been included in the test design. Remember, when generating tests based on 2-way interactions Hexawise ensures that every combination of two different parameter's values are tested. Let's say for example that more than 75% of the trades executed on the System Under Test are by Existing Customers trade using Cash Accounts on a Web Channel with a transaction size ranging from 1 to 5,000. While skimming through the generated test cases, you may notice that that 4-way interaction does not appear in your design. This is problematic because 75% of all trades executed are reliant on these 4 parameter values together. Hexawise provides a feature to force this high-priority or most common scenario. In between the "Define Inputs" button and the "Create Tests" button you will see another button labeled "Requirements." Here is where Hexawise ensures that a specific test case is included in the generated results. If you are unsure how to implement a "Most Common Use Case" please follow the instructions provided here.
If you are not sure wha tthe most commmon use case is at this point, that is OK. Take your best guess and force an end-to-end test to appear using the Requirements feature.
If You Are Aware of One or More Specific High-Priority Combinations That are Important to Include in Your Test Set, Force Them to Appear Using the Requirements Feature As Shown Above.
Best practice tip: specify the fewest number of Values that are needed to achieve your testing objective. For example, if there is a special business rule that would be triggered only when 4 specific Values were all present in the same transaction, then you should use the Requirements feature to force those specific 4 Values to appear in a single test case; but you should not specify more than 4 specific Values.
VI. Step Six: Use the Auto-Scripting Feature to Automatically Generate Detailed Tester Instructions
After you have generated a set of test cases, you can see (in a table) which combinations of parameter values are designed to be tested together. Hexawise provides a feature that allows users to automatically generate clearly written instructions for the testers. These instructions are laid out step-by-step for each test case. The screenshot above provides some context as to how you can provide instructions for each selection to be made as well as specific instructions that are only triggered when certain conditions are true. In the example above, when Channel = VIP Lounge, each processed order should come complimentary Scotch. More details on how to use the Auto-Scripting Feature are provided here.
VII. Step Seven: Export Your Test Set into Excel and Inspect the Contents of the Auto-Scripts Tab.
VIII. Step Eight: Share Your Draft Set of Tests with firstname.lastname@example.org and the Hexawise training lead(s) within your company.
It is OK at this point if the test set is incomplete and/or has a few problems. The most important objective of this exercise is to have you get started using Hexawise on your actual project.