A Course in Black Box Software Testing
Examples of Test Oracles
See lecture notes on oracles and test automation.
Copyright (c) Cem Kaner, 2004
An oracle is a mechanism for determining whether the program has passed or failed a test.
A complete oracle would have three capabilities and would carry them out perfectly:
- A generator, to provide predicted or expected results for each test.
- A comparator, to compare predicted and obtained results.
- An evaluator, to determine whether the comparison results are sufficiently close to be a pass.
One of the key problems with oracles is that they can only address a small subset of the inputs and outputs actually associated with any test. The tester might intentionally set the values of some variables, but all of the program's other variables have values too. Configuration settings, amount of available memory, and program options can also affect the test results. As a result, our evaluation of the test results (that we look at) in terms of the test inputs (that we set intentionally) is based on incomplete data, and may be incorrect.
Any of the oracle capabilities may be automated. For example, we might generate predictions for a test from previous test results on this program, from the behavior of a previous release of this program or a competitor's program, from a standard function or from a custom model. We might generate these by hand, by a tool that feeds input to the reference program and captures output or by something that combines automated and manual testing. We might instead generate predictions from specifications, advertised claims, regulatory requirements or other sources of information that require a human to evaluate the information in order to generate the prediction.
The following examples illustrate the use of oracles:
- (Oracles) Testing of Copy/Paste in Office Suites
- (Oracles) Drag and Drop Images into Spreadsheets
- (Oracles) Testing of DoS Attacks in Browsers
- (Oracles) Testing Page Display in Mozilla FireFox
Copyright (c) Cem Kaner 2004
This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
