result_summarizer

MLPerf result summarizer

MLPerf result summarizer

Usage

To summarize an organization's submission results:

python3 -m mlperf_logging.result_summarizer FOLDER USAGE RULESET

Alternatively, multiple organizations' submissions can be processed:

Currently, USAGE in ["training"] and RULESET in ["0.6.0", "0.7.0", "1.0.0", "1.1.0", "2.0.0"] are supported. FOLDER can be a single organization's submission folder like /path/to/training_results_v0.6/COMPANY_NAME. For example,

python3 -m mlperf_logging.result_summarizer ../training_results_v0.7/Google/ training 0.7.0

Alternatively, FOLDER can be a multi-org pattern like "/path/to/training_results_v0.6/{COMPANY_A,COMPANY_B}" or a wildcard pattern like "../training_results_v0.6/{\*}". For example,

python3 -m mlperf_logging.result_summarizer "../training_results_v1.0/{Google,NVIDIA}" training 1.0.0
# or
python3 -m mlperf_logging.result_summarizer "../training_results_v1.0/{*}" training 1.0.0

With the --csv flag, the result summarizer exports a CSV line for each system, corresponding to one row of a table like the 0.6 results.

In order to generate private systems-specific IDs for each submission, please use the "--generate_private_ids" flag to generate a "private_id.json" file per each system. This will contain the private id assigned to each system, which will be exported in an exported CSV file under the "private-id" column.

Tested software versions

Tested and confirmed working using the following software versions:

Python 3.7.7 Python 3.9.2 Python 3.9.10