-
Notifications
You must be signed in to change notification settings - Fork 1
Specification
The universal access point for testing is the $TEST_SUITE global array ($TEST_SUITE ||= []).
A test framework need only append compliant test objects to $TEST_SUITE. When
executed, Ruby Test will iterate through each test object, handling each object
in the suite as either a test case or a test procedure.
A test object that responds to #each is taken to be test case. Test cases are iterated
over and each entry goes though the same determination.
If a test case also responds to #call, it is taken to be a per-case fixture setup/teardown
procedure in the form of around advice. Ruby Test will invoke #call passing it a block
which it must yield to continue running the test case. If it does not yield than the test case
will behave as if it is void of any tests.
A test object that responds to #call, but not #each, is handled as a test procedure. The #call
method is invoked wrapped is rescue clause that catches all exceptions raised. If no exception is
(and does not return false or nil when using hard test mode) than the test passes.
All test objects must respond to #to_s so their description can be used in test reports.
If the return value of #to_s is a multi-line string, the first line is
taken to be the label or summary of the test object. The remaining
lines are taken to be additional detail. A report format might only
show the detail if the verbose option is set.
If a test procedure raises an error, that is not an assertion (see below), Ruby Test records
is as an "error". Ruby Test catches almost all exceptions types. The only exception classes it
does not catch are NoMemoryError, SignalException, Interrupt and SystemExit.
Any raised exception that responds to #assertion? in the affirmative is taken
to be a failed assertion rather than simply an error. Ruby Test extends the
Exception class to support this method for all exceptions.
There are two types of test omissions: pending and offending.
A test may raise a NotImplementedError to have a test considering pending,
which is used to remind developer's that a test still needs to be written.
However, if the NotImplmentedError responds in the affirmative to #assertion?
then the test is taken to be a purposeful omission, rather than simply pending.
This is used when a test can't be run because of an "offending" circumstance,
such as a platform dependence, or a limitation of the test framework, etc.
Note that NotImplementedError is a standard Ruby exception and a subclass of
ScriptError.
As Ruby Test executes a test suite, it files tests into different categories based on the result of their execution, such as "pass" for passing tests and "fail" for failing tests. Ruby Test records pending tests under "todo" and offending tests under "omit".
A test case that responds to #ordered? indicates if its tests must be run
in order. If false, which is the default, then the test runner can randomize
the order for more rigorous testing. But if true, then the tests will always
be run in the order given. Also, if ordered, a case's test units cannot be
selected or filtered independent of one another.
A test object may provide a #type, which can be used to characterize the
type of test case or test unit. For example, RSpec might return "describe"
as the type of a test case and "it" as the type of a test unit, where as
Cucumber would use "Feature" and "Scenario" for test cases and "Then"
for test units (depending on choices made by the implementors).
(TODO: Is "Setup" a better name for this?)
A test object can respond to #topic which can be used to provide
a description of the setup associated with a test object. The return
value of #topic is sent #to_s to ensure it is a string.
If a test object responds to #source_location it will be taken to
identify the file and line in which the the test was defined. The
return value of #source_location must be a two-element array of
[file, line].
A "unit" (as defined in the software testing literature) is the smallest piece of code software that can be tested in isolation. In Ruby software, being object-oriented, these are classes, metaclasses and their corresponding instances, methods and functions (class methods).
A test object may respond to #unit to identify the particular
component of the software being tested. The return value must be the
the full name of a class, module, or method. Methods must be represented
with fully qualified names, e.g. Foo::Bar#baz and Foo::Bar.baz for an
instance method or class method, respectively.
A test object can provide #tags which must return a one-word string or
array of one-word strings. The test runner can use the tags to limit the
particular tests to be run.
If any test object responds to #skip? it indicates that the test unit or
test case is to be skipped and not tested. If the return value is a string
the string is take to be a description of the reason it is being skipped.
It may or may not be mentioned in the test output depending on the reporter
used and the verbosity mode at run-time.
Skipping test differs from omission in that omitted test are still executed. They simply raise an error at the point of omission. Unlike skipped tests, this allows omitted tests to have active partial implementations.
Skipped tests are filed under "skip".