Skip to content

Experiment.md

Latest commit

 

History

History
258 lines (202 loc) · 10.9 KB

Experiment.md

File metadata and controls

258 lines (202 loc) · 10.9 KB

ELAPS Experiment

The Experiment class is the central component of the elaps Python module. It both statically represents the experiments within the ELAPS framework and features functionality to support their design and handling.

Table of Contents generated with DocToc

Setup and Attributes

The Experiment setup essentially describes how and in what environment the set of kernel invocations defined in its calls attribute shall be executed. The attributes of the Experiment class that form this setup are described in this section.

For each attribute *, there exists a setter method set_*(). These methods incorporate type and consistency checks for the attribute values and should be preferred over setting the attributes directly. Upon encountering invalid or incompatible values, the setters will throw corresponding exceptions. Each setter allows two optional keyword arguments:

  • force=True tells the Experiment to, if possible, avoid exceptions by replacing incompatible values with default ones or otherwise adjusting itself to ensure consistency.
  • check_only=True will only perform the consistency checks (and possibly throw exceptions) but will not not actually set the attribute to the given value

The following subsections present the Experiment attributes "top down", i.e., each attribute may be affected by the preceding attributes' values.

sampler

A data structure (dict) describing the Sampler used in the experiment. This data structure is generated automatically alongside each Sampler compilation as info.py in its build directory. To load such structures, the package elaps.io provides the routines the utility routines load_sampler(name) and load_all_samplers().

Relevant for the Experiment setup are the following sampler entries:

  • papi_enabled: Whether PAPI is available. If so:
    • papi_counters_avail: A list of available PAPI event names.
    • papi_counters_max: Maximum number of parallel counters.
  • nt_max: The maximum number of threads on the system.
  • omp_enabled: Wether OpenMP support is available (for sumrange_parallel and calls_parallel).
  • kerlens: A dict of available kernel names with their C signatures.

Setter: set_sampler(sampler)

papi_counters

The list of PAPI counters (as a tuple) to be measured. The counters have to be supported by the Sampler and cannot exceed its counter limit.

Setter: set_papi_counters(papi_countes)

range

[range_var, range_vals], a range variable and range values as a 2-element list, or None. If set, range_var is an elaps.symbolic.Symbol, while the range_vals is a finite iterable (such as a list or an elaps.symbolic.Range).

With the range option set, the Experiment will repeat the experiment as defined by the options below for each value in range_vals. In the Report generated by the Experiment, each iteration of the range can be accessed independently or presented as a line plot.

Setters:

  • set_range([range_var, range_vals]) or set_range(None)
  • set_range_var(range_var) and set_range_vals(range_vals)

nthreads

The number of threads used by the kernel library, limited by the Sampler (positive int or symbolic in range_var).

The range variable can be used to let the number of threads depend on the range's value.

Setter: set_nthreads(nthreads)

nreps

The number of repetitions (positive int or symbolic in range_var).

The experiment as defined by the options below is repeated nreps times, allowing the resulting Report to compute statistics of the measurements' variations.

The range variable can be used to let the number of repetitions depend on the range's value.

Setter: set_nreps(nreps)

shuffle

Whether the repetitions loop is moved above the range loop.

Only possible if both nthreads and nreps are non-symbolic.

Setter: set_shuffle(shuffle)

sumrange

[sumrange_var, sumrange_vals], a range variable and range values as a 2-element list, or None. If set, sumrange_var is an elaps.symbolic.Symbol, while the sumrange_vals is a finite iterable (such as a list or an elaps.symbolic.Range) that may depend symbolically on range_var. (e.g., if range_var == Symbol("i"), one could have range_vals = Range("1:i").)

With the sumrange options set, the Experiment will repeat the calls for each entry in sumrange_vals, while the resulting Report will sum up these contributions into a single value.

Setters:

  • set_sumrange([sumrange_var, sumrange_vals]) or set_sumrange(None)
  • set_sumrange_var(sumrange_var) and set_sumrange_vals(sumrange_vals)

sumrange_parallel

Wether the calls in the sum-range shall we executed in parallel. The used Sampler must support OpenMP for this option.

Setter: set_sumrange_parallel(sumrange_parallel)

calls_parallel

Wether the calls for one sum-range iteration shall be executed in parallel. The used Sampler must support OpenMP for this option.

Setter: set_calls_parallel(calls_parallel)

calls

The list of kernel invocations to be sampled. Each kernel in this list must be supported by the Sampler.

There are two types of calls:

  • elaps.signature.Call instances represent calls with associated Signatures.
  • elaps.signature.BasicCall instances can be used if no corresponding Signature, however these calls have severely limited functionality with respect to data handling.

The types of arguments for calls and their possible values are discussed in the following subsections.

Setters:

  • set_calls(calls)
  • add_call(call), set_call(callid, call), and remove_call(callid)
  • set_arg(callid, argid, value)

Calls with a Signature (Call)

Signatures can be loaded through elaps.io's load_signature(name) or load_all_signatures(). Calling such a Signature, results in a corresponding Call instance.

Within an Experiment, a Call's values have to conform with its Signature and possibly other calls' in the Experiment:

  • Flag arguments must contain allowed values (e.g., N or T for a trans argument).
  • Dimension arguments must be positive ints or elaps.symbolic.Expressions (symbolic expressions), which may depend on both the range_var and sumrange_var.
  • Scalar arguments must be numbers or symbolic Expressionss that evaluate to the associated data types (int, float, or complex).
  • Data arguments (or operands) must be strings beginning with a letter. Operands may occur repeatedly both within one call and across calls; in this case all corresponding dimension and leading dimension arguments must have consistent values.
  • Leading dimension arguments must be ints or symbolic Expressions that are at least as large as the first dimension of the corresponding operand.

Calls without a Signature (BasicCall)

Each sampler provides minimal signatures for each of its associated kernels, which basically are lists of the involved C arguments (e.g. ("dcopy", "int *", "double *", "int *", "double *", "int *")). The BasicCall constructor takes such a signature followed by a list of arguments of appropriate length. By passing only the kernel name in a list of length 1 to set_call (e.g. set_call(0, ["dcopy"])), the Experiment will automatically generate a BasicCall if the kernel is available.

The allowed argument formats are:

  • char * arguments can be passed any string.
  • Any other argument (int *, float *, or double *) can be passed values in two different formats, both of which may contain symbolic Expressions:
    • A single value or a list of values, e.g. 1,2 to represent the complex number 1 + 2 i.
    • A size in brackets (e.g. "[100000]") will pass a buffer of such many elements from the Sampler's Dynamic Memory buffer.

vary

Wether operands refer to the same memory buffers across repetitions and iterations of the sum-range and how they depend on these iterations. vary is a dict that has one entry for each operand, which in turn is a dict with they following keys:

  • with: A set of variables with which the operand varies. Can contain "rep" (to vary with repetitions) and the sumrange_var.

    Setters: set_vary_with(name, with_), add_vary_with(name, with_var), and remove_vary_with(name, with_var)

  • along: Along which dimension the variable shall vary. (only makes sense for matrix operands.) If along == 0, matrix arguments for different iterations would be stacked along their first dimension (in BLAS data layout forming a column stack), thereby increasing the leading dimension; along == 1 would stack them along their second dimension (in BLAS data layout forming a row stack).

    Setter: set_vary_along(name, along)

  • offset: An additional offset to further separate consecutive iterations' operands. Can be an int or a symbolic Expression.

    Setter: set_vary_offset(name, offset)

All vary entries can also be set at once:

Setter: set_vary(name, with_=None, along=None, offset=0)

Execution and Job Submission

Experiment are executed or submitted as jobs through its submit(reportname) routine. This routine will generate input files for the specified Sampler, as well as a shells script that invokes the Sampler accordingly. Depending on the Sampler, this script is then either executed locally or submitted to a batch job system, such as LSF or LoadLeveler.

Upon execution, the script generates a report file (extension: .elr, which can be loaded as a Report through elaps.io's load_report(filename). If any errors occur, an error file (extension: .err) is generated.

Storing and Loading Experiments

Experiments can be represented as strings in two ways:

  • As a formatted and human readable visual representation obtained through str().
  • As a python parsable text obtained through repr(). The module elaps.io further provides the utility routines load_experiment(filename) and write_experiment(experiment, filename).