Slow performance of osrm-extract inside Docker container

[danpat]

When running osrm-extract from inside a docker container, extraction is slower than if performed on the host machine itself, by about 25%.

I can think of a few possibilities:

1. Something like Slow IO performance inside container compared with the host. moby/moby#21485
2. Different compiler optimizations inside our docker images (poor optimization?)
3. Docker overhead (should be 0 in a Linux environment).

STXXL performs lots of I/O during osrm-extract - any overhead from the Docker FS layer here would hurt us.

I have not tested osrm-contract or osrm-routed performance at this stage.

/cc @miccolis

[daniel-j-h]

@miccolis @danpat any updates on your side here?

We now publish tags and master to https://hub.docker.com/r/osrm/ and in the future we may want to use the Docker images for the demo server.

The linked issue was fixed in Docker 1.12, and was caused by Linux' CFQ scheduler trying to balance I/O between cgroups.

Docker 1.10.3 is in Ubuntu 16.04 but according to issue deadline sched (Ubuntu default) not affected.

• https://www.kernel.org/doc/Documentation/block/cfq-iosched.txt
• https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt

[miccolis]

One thing that helped was explicitly setting the number of CPUs available to the container. Docker doesn't completely hide info from a container, so if you have a containers that tries to spawn threads for all CPUs there will be contention. Using the --threads argument and setting the OMP_NUM_THREADS & MALLOC_CONF environment variables helped.

[danpat]

@sreeramvuppala You could just make a new Dockerfile that uses the OSRM image as a baselayer:

FROM osrm/osrm-backend:latest
WORKDIR /data

Then, you can put the .stxxl file in the same directory as your .osm.pbf files, outside your container.