-
Notifications
You must be signed in to change notification settings - Fork 1
Specification
The universal access point for testing is the $TEST_SUITE global array ($TEST_SUITE ||= []).
A test framework need only append compliant test objects to $TEST_SUITE. When
executed, Ruby Test will iterate through each test object, handling each object
in the suite as either a test case or a test procedure.
A test object that responds to #each is taken to be a test case. Test cases are iterated
over and each entry goes though the same determination.
If a test case also responds to #call, it is taken to be a per-case fixture setup/teardown procedure (i.e. before-all and after-all) in the form of around advice. Ruby Test will invoke #call passing it a block which it must yield to continue running the test case. If it does not yield than the test case will behave as if it is void of any tests.
A test object that responds to #call, but not #each, is handled as a test procedure. The #call
method is invoked wrapped is rescue clause that catches all exceptions raised. If no exception is
raised (and does not return false or nil when using assertionless test mode) than the test passes.
All test objects must respond to #to_s so their description can be used in test reports.
If the return value of #to_s is a multi-line string, the first line is taken to be the label or summary of the test object. The remaining lines are taken to be additional detail. It is at the discretion of the report format whether to show the detail or not, e.g. if the verbose option is set.
(Multi-line descriptions are very uncommon though, so it rarely matters).
If a test procedure raises an error, that is not an assertion (see below), Ruby Test records is as an "error". Ruby Test catches almost all exceptions types. The only exception classes it does not catch are NoMemoryError, SignalException, Interrupt and SystemExit.
Any raised exception that responds to #assertion? in the affirmative is taken to be a failed assertion rather than simply an error. Ruby Test extends the Exception class to support this method for all exceptions. This is in compliance with the BRASS assertion standards.
There are two types of test omissions: those that are pending in need of eventual attention, and those that are simply non-applicable and must be by-passed altogether.
The first type only applies to test procedures, not test cases. A test procedure can raise a NotImplementedError to have the test considered pending and in need of attention. This is used to continually remind developer's that a test still needs to be written or completed. In other words, a todo item. RubyTest recognizes exception priorities. If the priority of the NotImplmentedError is below zero
then it is taken to be of minor significance and, for current practical purposes at least, will simply be ignored. This later form is used when a test should ultimately be written but is not urgent enough that it should constantly come to the attention of testers. Report formats should only show this later priority of pending test when a verbose flag or other such configuration is set. (Note the exception priorities are a recent introduction to the specification and their usage is still being fine tuned.)
Note that NotImplementedError is a standard Ruby exception and a subclass of ScriptError. Technically using NotImplementedError for this purpose is not an optimal solution since it prevents NotImplementedError from being used as a ordinary test expectation. However, this is a fairly rare case and can be worked around easily with a more basic assertion, e.g. assert NotImplementedError.raised?{ ... }.
The second type of omission applies when a test can't be run because of certain circumstance, such as a special platform dependence, or a limitation of the test framework, or what have you. These types of tests are filed under the term skip. Any test object responds to #skip? indicates that the test unit or test case is to be skipped and not processed at all. If the return value is a string, the string is taken to be a description of the reason it is being skipped. It may or may not be mentioned in the test output depending on the report format used and the verbosity mode at run-time.
Pending tests differ from skipped tests in that pending tests are still executed. They simply raise an error at the point of omission. Unlike skipped tests, this allows pending tests to have active partial implementations.
A test case that responds to #ordered? indicates if its tests must be run
in order. If false, which is the default, then the test runner can randomize
the order for more rigorous testing. But if true, then the tests will always
be run in the order given. Also, if ordered, a case's tests cannot be
selected or filtered independent of one another.
A test object may provide a #type, which is used to characterize type of test
case or test procedure. For example, RSpec might return "describe" as the type
of a test case and "it" as the type of a test procedure, since that is the RSpec
nomenclature.
If a test object responds to #source_location it will be taken to
identify the file and line in which the the test was defined. The
return value of #source_location must be a two-element array of
[file, line].
A "unit" (as defined in the software testing literature) is the smallest piece of code software that can be tested in isolation. In Ruby software, being object-oriented, these are modules, classes, meta-classes, their corresponding instances, methods and class methods.
A test object may respond to #unit to identify the particular component of
the software being tested. The return value must be the the full name of a class,
module or method. Methods must be represented with fully qualified names,
e.g. Foo::Bar#baz or Foo::Bar.baz for an instance method or class method,
respectively.
A test object can provide #tags which must return an an array of one-word
symbols and optionally a tail entry of a Hash where the keys are symbols
indexing any object type. This later entry is generally called test metadata
or keys to differentiate it from tags. Tags are used by Ruby Test to
allow testers to selectively filter tests.
RubyTest supports universal before and after hooks. If #scope is defined on a test
object scope will be used as the evaluation context for these hooks, rather than the
context in which they were defined.
A test object can also respond to #topic which can be used to provide
a description of the setup or subject associated with a test object.
The return value of #topic is sent #to_s to ensure it is a string.