Software Testing Methods
I introduce multiple methods currently being used to test software.
1. Fuzz Testing
- Input massive invalid, unexpected, or random data(Fuzz) to find errors or vulnerabilities in the software.
- Effective on blackbox testing
Pros — Simple and provides results with little effort
Cons — May miss bugs that don’t trigger a full program crash and is less likely to trigger bugs that are only triggered in highly specific circumstances.
2. Stochastic Testing
- Input random sequence of datas
3. Model-based Testing
- Generates meaningful test case models which can detect errors on the desired behavior.
4. Symbolic Testing
- Replace inputs of program with symbolic variables.
- Track symbols rather than concrete input
- When execution path diverges, fork and add constraints on symbolic values
- Find the feasibility of all paths in the program.
- Quite expensive for finding feasibility of path.
x = read();if (x > 3) { y = 1; if (x < 0) y = 2;}
else y = 3;
After changing with symbolic variables, it checks the feasibility of paths. On above example, the expression (y = 2;) will not be reached.
Get Chang Min Park’s stories in your inbox
Join Medium for free to get updates from this writer.
Challenges
- Path explosion — (1) Exponential in branching structure and (2) loop on symbolic variables.
- Complex code and environment dependencies — (1) System calls, (2) library calls, and (3) recursive calls.
Possible Solutions
- For “path Explosion” — (1) Random search (e.g., randomly restart in a while loop, pick next path randomly), but check coverage implementation is required and (2) Combined search.
- For “complex code and environment dependencies” — (1) Simulate system call, (2) build simple versions of library calls, (3) run library an system calls with a concrete value, (4) summarize the loops.
Applications
- Detecting infeasible paths
- Generating test inputs
- Finding bugs and vulnerabilities
- Proving two code segments are equivalent (Code Hunt)
5. Concolic Testing (Concrete + Symbolic)
- Input actual concrete value to the program, then generate new input by changing the path condition.
- Less expensive than Symbolic testing.
6. A/B (or split, bucket) Testing
- Comparing two versions of software which performs better
- Statistical analysis which performed better
e.g., randomly showing two different versions to users, and see which one had more uses
7. Boundary Testing
Extreme ends like Start- End, Lower- Upper, Maximum-Minimum, Just Inside-Just Outside values are called boundary values and the testing is called “boundary testing”. Input variables are selected at their:
- Minimum
- Just above the minimum
- A nominal value
- Just below the maximum
- Maximum
Boundary Testing comes after the Equivalence Class Partitioning.
8. Equivalence partitioning or equivalence class partitioning (ECP)
Software testing technique that divides the input data of a software unit into partitions of equivalent data from which test cases can be derived
In principle, test cases are designed to cover each partition at least once. This technique tries to define test cases that uncover classes of errors, thereby reducing the total number of test cases that must be developed.
Pros — reduction in the time required for testing software due to lesser number of test cases
