ExperimentRunner

This package provides a command-line tool experiment-runner to facilitate running a grid search over the udacity simulator. Load configurations from a YAML file, execute experiments, and save the results in as json lines in a file (.jsonl).

Usage

Install

To install the package there are two major steps to follow:

• install the package wheel in a python 3.8 environment
• download the simulator and self driving model

Package - Conda + Pip

Install python 3.8 and create the environment:

conda install python=3.8 -y
conda create --name exp_runner python=3.8 -y
conda activate exp_runner

Download the wheel file from the distributions directory and install it with:

pip install ./experimentrunner-version.whl

Package - Poetry + Pyenv

First setup an environment with python 3.8

pyenv install 3.8
pyenv local 3.8

Download the wheel file from the distributions directory and install it with:

poetry add experimentrunner-version.whl
poetry install

Simulator & Model executables

To run the package is also necessary to provide the simulator and model executables as arguments.

Both can be downloaded here

Once downloaded, extract the contents obtaining:

simulator&model
├── SelfDrivingModels
│   └── mixed-chauffeur.h5 # the self driving model
└── Simulator
    └── ubuntu_binaries
        ├── LinuxPlayer_s.debug
        ├── ubuntu_Data
        ├── ubuntu.x86_64   # the simulator executable 
        └── UnityPlayer.so

Run

To start the grid search is then sufficient to run the following command:

poetry shell
experiment-runner \
    --config path_to_grid_search_config_file.yaml \
    --output path_to_file_that_will_store_output.jsonl \
    --simulator-path path_to simulator_executable \
    --model-path path_to_self_driving_model

an example on how the command might look like is:

 experiment-runner \
--config example/configs.yaml \
--output results.jsonl \
--simulator-path ./simulator\&model/Simulator/ubuntu_binaries/ubuntu.x86_64 \
--model-path ./simulator\&model/SelfDrivingModels/mixed-chauffeur.h5 

Experiment configuration

The configuration is read from a YAML file and translated into a grid search. An example configuration can be found at ./example and looks like this:

landscape_map: # wrapper level
  minSpeed:
    __range__:
      from: 10
      to: 15
  maxSpeed: 100

For usage grid search details, refer to the GridSearch documentation.

For possible fields in the configuration file refer to the UdacitySimulatorConfig class

Development

Setup

The package uses python 3.8 and poetry for dependency management. To set up the development environment:

pyenv install 3.8
pyenv local 3.8
poetry env use $(pyenv prefix 3.8)/bin/python
poetry install

Then it is needed to add the simulator executables and the self-driving model to the project.

Project Structure

.
├── cli.py # entry point for the command line tool
├── poetry.lock
├── pyproject.toml
├── README.md
├── src
│   ├── expr_runner # 
│   │   ├── ExperimentRunnder.py # main wrapper to load config, run experiments and save results
│   │   ├── __init__.py
│   └── simulator # simulator related files
│       ├── __init__.py
│       ├── lanekeeping # code for lanekeeping experiments (generate road, run experiment, etc)
│       ├── SelfDrivingModels # location for self-driving models
│       └── SimulatorExec # location for simulator executables
└── tests

Building the package

To build the package run:

poetry build

This will generate the distribution files in the dist/ folder. The package can then be installed with pip as described in the Install section.