Scenario generator

[vkresch]

Reference to a related issue in the repository

This PR is a first draft of the scenario generator to benchmark the performance of OSI as mentioned here (#428).

Add a description

You can generator variable amount of moving object with variable amount of timestamps/messages.

Further TODOs for this PR

• support unlimited moving object generation
• support unlimited stationary object generation
• support unlimited pedestrain generation
• basic model to transform into SD
• add test for each functionality
• modular refactoring
• add pipeline into current CI as benchmark

Check the checklist

• I have made corresponding changes to the documentation.
• My code and comments follow the style guidelines and contributors guidelines of this project.
• I have performed a self-review of my own code.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests / travis ci pass locally with my changes.

[vkresch]

Added stationary objects and some more dynamics which can be visualized in osi-visualizer:

[pmai]

Looking like a very good start to me, will form the basis of our further performance work going forward.

[DerBaertige]

During the Q4-meeting I became attention to this pull request. I have done a similar approach generating random objects for performance testing some time ago.

My intention was a bit different, as I have compared different programming languages and implementations of protobuf, butthere was no storing of generated scenarios and no sequential movement of the objects. I compared the native C++ and Python implementation and the third-party C implementation nanopb, which has no dependencies to external libraries.
The original German publication (Aufbau einer Mixed-Reality-Versuchsumgebung zur Absicherung autonomer Systeme) is available at Springer (paywall), but the presentation is public available. The similar talk (Design of a Mixed Reality Test Environment to Validate Autonomous Vehicles) was also presented at the Webinar / Crash Course -- ASAM OSI.

If there are interests ofthe performance tests, I can publish the code in the next weeks.

[vkresch]

@DerBaertige thanks for the info. I am quite interested in the code for the performance test of the creation of the data structures. I think it could be added as a seperate unit test into the CI then. Feel free to link it here from your repository. Ofc I would add attribution in the comments if I use any of the code in this PR.

[DerBaertige]

I added the C++ and Python implementation here, as it was not in a public repository yet.
The nanopb implementation has to be rewritten in parts, as I do not have the ownership of parts of the code. If I find time, I will submit it later.

[Kohda-adc4g]

I modified the performance program to make a flatbuffers version. The flatbuffers version of OSI was created based on # 427.

Environment:
Ubuntu 20.04
Open Simulation Interface 3.3.1
google Flatbuffers 2.0.0

---

// \file performance.cpp
// \brief Simple performance evaluation example
//
// Simple test application, which measures the execution time of different
// parts of the open simulation interface and the protobuf implementation.
//
// The regions of interests are the filling of the structure of a
// OSI GroundTruth message, the serialization and the parsing process using
// the native C++ interfaces of protobuf.
//
// \author Georg Seifert Georg.Seifert@carissma.eu
// \version 0.1
// \date 2021
//
// The regions of interests are the filling of the structure of a
// OSI GroundTruth message, the serialization and the parsing process using
// the native C++ interfaces of flatbuf.
//
// \author M.Kohda kohda@adac.co.jp
// \date 2021
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License v. 2.0. If a copy of the MPL was not distributed
// with this file, you can obtain one at http://mozilla.org/MPL/2.0/
//

#include <random>
#include <ctime>
#include <iostream>
#include <limits>
#include <unistd.h>
#include "flatbuffers/flatbuffers.h"
#include <osi_groundtruth_generated.h>
#include <osi_object_generated.h>
#include "clock.h"

// \def ITERATIONS
// Default number of iterations of the benchmark
//
#define ITERATIONS (1000)

// \def OBJECTS
// Default number of moving objects used to fill the structure
//
#define OBJECTS (10)

int main(int argc, char** argv)
{
uint64_t start, stop;
int c;
size_t i, num_object, num_object_max = OBJECTS, num_iterations_max = ITERATIONS;
size_t *ts_ser;
size_t *ts_deser;
size_t *ts_gen;

// FlatBuffer allocated buffer
flatbuffers::FlatBufferBuilder flatbuf;

std::mt19937 generator(time(0));
std::uniform_real_distribution<> distribution(0.0, 1.0);

while ((c = getopt(argc, argv, "o:i:h")) != -1) {
    switch (c) {
    case 'o':
        num_object_max = atoi(optarg);
        break;
    case 'i':
        num_iterations_max = atoi(optarg);
        break;
    case '?':
    case 'h':
        std::cout << "Argumente:" << std::endl << 
                      "\t-o <num>\tCount of the moving objects, default = " << OBJECTS << std::endl <<
                      "\t-i <num>\tIteration of the benchmark, default = " << ITERATIONS << std::endl;
        return 1;
    default:
        break;
    }
}

std::cout << "Iterationen: " << num_iterations_max << 
             ", Moving Objects: " << num_object_max << std::endl;

ts_ser = new size_t[num_iterations_max];
ts_deser = new size_t[num_iterations_max];
ts_gen = new size_t[num_iterations_max];

for(i = 0; i < num_iterations_max; i++)
{
const osi3::GroundTruth* output;
flatbuffers::Offset<osi3::GroundTruth> gt_ser;
    flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<osi3::MovingObject>>>  moving_object;
    
    start = clock_get_ns();
    for(num_object = 0; num_object < num_object_max; num_object++)
    {
        auto moving_type = osi3::MovingObject_::Type::TYPE_VEHICLE;
    auto strategy_type = osi3::StationaryObject_::Classification_::Type::TYPE_OTHER;
        auto classification_type = osi3::MovingObject_::VehicleClassification_::Type::TYPE_SMALL_CAR;
        auto vehicle_classification = osi3::MovingObject_::CreateVehicleClassification(flatbuf, moving_type, 0, (bool)false);

        // osi3::Identifier * 
    flatbuffers::Offset<osi3::Identifier> id = osi3::CreateIdentifier(flatbuf, num_object);
        
        // length = 12.0m (StVZO 32, 3,1)
        // width = 2.55 (StVZO 32, 3.1)
        // height = 4.0 (STVZO 32, 2),
        // osi3::Dimension3d* 
    flatbuffers::Offset<osi3::Dimension3d> dimension = osi3::CreateDimension3d(flatbuf, 
	/* length */
	(12.0 * distribution(generator)),
	/* width */
	(2.55 * distribution(generator)),
	/* height */
	(4.0 * distribution(generator))
    );
        
        // position: x, h, z (-512 -- 512)
        // osi3::Vector3d* 
        flatbuffers::Offset<osi3::Vector3d> position = osi3::CreateVector3d(flatbuf, 
        	/* x */
	(1024 * distribution(generator) - 512),
	/* y */
	(1024 * distribution(generator) - 512),
	/* z */
	(1024 * distribution(generator) - 512)
    );

        // roll, pitch, yaw (0 -- 2pi)
        // osi3::Orientation3d* 
    flatbuffers::Offset<osi3::Orientation3d> orientation = osi3::CreateOrientation3d(flatbuf,
	/* roll */
	(6.28 * distribution(generator)),
	/* pitch */
	(6.28 * distribution(generator)),
	/* yaw */
	(6.28 * distribution(generator))
    );

        // velocity (-30 -- 100) 
        // osi3::Vector3d* 
    flatbuffers::Offset<osi3::Vector3d> velocity = osi3::CreateVector3d(flatbuf,
	/* x */
        	(130.0 * distribution(generator) - 20.0),
	/* y */
        	(130.0 * distribution(generator) - 20.0),
	/* z */
        	(130.0 * distribution(generator) - 20.0)
    );

        // acceleration (-12 -- 10), based on Pkw-Pkw-Unfalle
        // https://doi.org/10.1007/978-3-8348-9974-3_12
        // osi3::Vector3d* 
    flatbuffers::Offset<osi3::Vector3d> acceleration = osi3::CreateVector3d(flatbuf,
	/* x */
        	(22.0 * distribution(generator) - 12),
	/* y */
        	(22.0 * distribution(generator) - 12),
	/* z */
        	(22.0 * distribution(generator) - 12)
    );

        // osi3::BaseMoving
    auto base = osi3::CreateBaseMoving(flatbuf, dimension, position, orientation, velocity, acceleration, 0, 0, 0);

        // osi3::MovingObject
    auto tmp_moving_object = osi3::CreateMovingObject(flatbuf, id, base, strategy_type, 0, 0, vehicle_classification, 0, 0, 0);
    moving_object = (flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<osi3::MovingObject>>> &) tmp_moving_object;
    }

    // Make GroundTruth
    gt_ser = osi3::CreateGroundTruth(flatbuf, 0, 0, 0, 0, moving_object, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);

    stop = clock_get_ns();
    ts_gen[i] = stop - start;

    start = clock_get_ns();
    osi3::FinishGroundTruthBuffer(flatbuf, gt_ser);
    stop = clock_get_ns();
    ts_ser[i] = stop - start;

    start = clock_get_ns();
uint8_t *buf = flatbuf.GetBufferPointer();
output = osi3::GetGroundTruth(buf);
    stop = clock_get_ns();
    ts_deser[i] = stop - start;
}

double min = std::numeric_limits<double>::max(), max = 0, mean = 0, variance = 0, stddev = 0;
for(i = 0; i < num_iterations_max; i++)
{
    min = (min > ts_gen[i]) ? ts_gen[i] : min;
    max = (max < ts_gen[i]) ? ts_gen[i] : max;
    mean += ts_gen[i];
    variance += ts_gen[i]*ts_gen[i];
}
mean /= num_iterations_max;
variance = variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Fill:" << std::endl << 
   "\tmin = " << min << " ns" << std::endl <<
   "\tmax = " << max << " ns" << std::endl <<
   "\tmean = " << mean << " ns" << std::endl << 
   "\tstandard deviation = " << stddev << " ns" << std::endl;

min = std::numeric_limits<double>::max(), max = 0, mean=0, variance=0, stddev = 0;
for(i=0; i < num_iterations_max; i++)
{
    min = (min > ts_ser[i]) ? ts_ser[i] : min;
    max = (max < ts_ser[i]) ? ts_ser[i] : max;
    mean += ts_ser[i];
    variance += ts_ser[i]*ts_ser[i];
}
mean /= num_iterations_max;
variance= variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Serialize:" << std::endl << 
   "\tmin = " << min << " ns" << std::endl <<
   "\tmax = " << max << " ns" << std::endl <<
   "\tmean = " << mean << " ns" << std::endl << 
   "\tstandard deviation = " << stddev << " ns" << std::endl;
    
min = std::numeric_limits<double>::max(), max = 0, mean = 0, variance = 0, stddev = 0;
for(i = 0; i < num_iterations_max; i++)
{
    min = (min > ts_deser[i]) ? ts_deser[i] : min;
    max = (max < ts_deser[i]) ? ts_deser[i] : max;
    mean += ts_deser[i];
    variance += ts_deser[i]*ts_deser[i];
}
mean /= num_iterations_max;
variance = variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Parse:" << std::endl << 
   "\tmin = " << min << " ns" << std::endl <<
   "\tmax = " << max << " ns" << std::endl <<
   "\tmean = " << mean << " ns" << std::endl << 
   "\tstandard deviation = " << stddev << " ns" << std::endl;

delete [] ts_ser;
delete [] ts_deser;
delete [] ts_gen;
return 0;

}