Devel images are build upon base images and provide a light-weight and safe environment to develop a functional project- or workspace-specific docker setup. If you are starting out a new docker development workflow for your project, a devel image will not yet be available on the registry. Refer to the create devel image subsection to create one. The devel images contain project specific libraries and can be pushed to the registry to provide a common foundation for cooperative development.
Just follow the instructions given in docker usage for using a container and running commands in it.
Make sure you have access to the Docker registry if you haven't already pulled the image (might require docker login and potentially a VPN connection).
Make sure that you use the exec mode devel (in your settings.bash or as argument to exec.bash).
The devel image is pulled with only the dependencies of the workspace installed. The devel container is set up to mount several directories within the host's repository clone:
- /opt/startscripts corresponds to the host's
docker_development/startscripts - /opt/workspace corresponds to the host's
docker_development/workspace - /home/devel corresponds to the host's
docker_development/home
These are initially empty. That means, in order to run applications from the workspace, it needs to be set up (and usually built) at the first run to fully initialize the container.
In order to do this, instantiate and attach to the container using ./exec.bash devel (as explained above).
If the image was created properly, instructions are printed how to proceed from there.
Usually this will include running the script bash /opt/setup_workspace.bash potentially followed by additional build steps.
To wrap it up, you can edit files in the workspace from your host system, but to compile and run, you need to enter the container as explained above (via ./exec.bash devel).
In case you are using VSCode also check out the "Remote - Containers" extension.
IMPORTANT: In order to apply new settings (like 3d acceleration), the container and saved settings have to be reset: rm -f -- has_gpu_support.txt {devel,release}-container_id.txt.
Devel images contain pre-installed libraries of specific projects (not shared as commonly as the ones installed in the base images).
The general steps are:
- Fork this repository for your project.
- Select the desired base image (Ubuntu, Rock, Ros etc).
- Add the installation of OS dependencies to the Dockerfile to add them to the environment.
- Add workspace initialization scripts/howto, e.g.
/opt/setup_workspace_script.bash
The suggested workflow and related topics are explained in detail in the following sections.
The image is accompanied by this repository. Therefore you should fork this repo to your project's namespace or into a new group using the git web interface. This way, changes and updates can be tracked and integrated more easily in both directions.
Then clone your fork to your system into the desired folder.
Afterwards you edit the settings.bash in order to
- set a new project name,
- make sure the default exec mode is specified
devel, - select the base image to use,
- set your registry (empty if none).
Now you can make the first attempt to build the new image, using bash build.bash. Without changes to the Dockerfile, this will be equal to the base image.
This is useful to see if the general process is working, and to proceed initializing the workspace within the newly generated container (try to run bash exec.bash too).
Go to docker_development/image_setup/02_devel_image and add installation commands (apt install ...) in order to pre-install all required workspace dependencies in the Dockerfile.
The Dockerfile is intended to contain any OS dependencies the workspace has (that are not already provided by the base image).
Especially if you are not sure which packages need to be installed at this point, it might make sense to start with the next step, to prepare the workspace. Then you can complete the dependencies in the Dockerfile, using the helper scripts for dependency enumeration.
The list_*_osdeps.* scripts help to determine which workspace-specific packages are required by the workspace.
These are available for Rock and ROS workspaces and list all OS dependencies depending on the packages' provided metadata. They can be found in the image_setup/02_devel_image folder and the output format allows easy copy/pasting into the Dockerfile.
The later release image is not based on the devel container, it is based on the image. Any tools or dependencies not installed via the Dockerfile of the devel image will not be available in the release.
When the devel container is started for the first time, the script /opt/init_workspace.bash is executed within the container.
You can use it to initialize the workspace, but it is executed each time a new image is available. It is recommended to make it print instructions for necessary post-initialization steps to this script, e.g., add a hint to run /opt/setup_workspace.bash after initialization:
RUN echo 'echo -e "\n\e[32mplease run /opt/setup_workspace.bash to initialize the workspace\e[0m\n"' >> /opt/init_workspace.bash
For the devel image the setup_workspace.bash script is copied to /opt/setup_workspace.bash within the container. It is intended to contain all the steps to initialize the actual workspace, i.e., clone the required repositories via autoproj_bootstrap and aup or via wstool.
It is recommended to not include building/compiling the workspace (e.g., via amake or catkin build) in this script, but again rather print instructions for the required follow-up steps for the finalization of the container setup that are then to be executed manually.
This decoupling is intended to make it easier to spot the source of problems in the process. As it can take rather a long time overall, and if not decoupled it might not be obviously at which point the error occurred.
You may need to run this script to test your workspace setup (clone repos, etc.) repeatedly until it works.
You may need to clean up the workspace (i.e., the mounted directories) manually in between those runs, in order to simulate a fresh setup.
Keep in mind that when using the devel image, the setup script is copied into the image during build, rather than being mounted as is the case for base images.
If your workspace does not require a lot of OS dependencies, i.e., the workspace setup is rather quick, than you might consider developing the setup_workspace.bash script using the base image.
To speed up the debugging process by using workspace related libraries you may use the devel image, where they are pre-installed (if listed in the Dockerfile).
In order to avoid rebuilding the image for each try, you can edit and test the script inside the container and finally copy the content of the functional script into the one located in the image_setup/02_devel_image directory. This will be copied into the devel image during the next build.
If during the process you notice that dependencies external to the workspace are still missing, don't forget to add them to the Dockerfile too, and potentially go back to the previous step.
As a recap, here the steps in short:
| Step | ToDo | File to edit/execute | |
|---|---|---|---|
| 0. | Fork this repository | => | |
| 1. | Edit project name, select base image, exec mode & registry | => | edit (in the repo's root directory): settings.bash |
| 2. | Add the steps to setup your workspace (e.g. using buildconf, wstool, git-repo, etc.) |
=> | edit: setup_workspace.bash |
| 3. | Start container and attach shell to it to test the setup script | => | exec (in the repo's root directory): bash exec.bash base |
| 4. | Test the script for workspace initialization | => | exec (in the attached container shell): bash /opt/setup_workspace.bash |
| 5. | Find out what OS dependencies are required | => | ROS: - Add list_ros_osdeps.bash to your mounted workspace and run it there.- Remember to source the ROS environment and rosdep update firstROCK: - Add list_rock_osdeps.rb to your mounted workspace and run it there.- Remember to source your env.sh first. |
| 6. | Add those OS dependencies for installation to Dockerfile | => | edit: copy/paste output from previous step into your 02_devel_image/Dockerfile |
| 7. | Build the initial version of the devel image with OS dependencies | => | exec (in image_setup/02_devel_image): bash build.bash |
| 8. | You should change the default exec mode to devel and push to your fork after pusing the image. |
=> | edit settings.bash |
As a prerequiste for the installation of a non-trivial Autoproj or wstool environment you will probably want to set up a method for authentication at some remote git server. There are at least two options for this.
- Git Credential Cache via HTTPS Protocol
- SSH (Public Key) Authentication
To commit/push your changes, it is recommended to do this from the host workspace folder and not from the mounted one within the container. This way, the committer and credentials of the host system is used.
The idea of this option is to use https user authentication in combination with the git credential cache so that git will only asks once per repository server (github, gitlab etc.).
You may want to add these lines to the .bashrc of your container:
git config --global credential.helper 'cache --timeout=2000'
git config --global url."https://".insteadOf git://Important Extra caution is required if you use ssh keys in your Docker containers. Make sure that the images you produce do not contain your private keys (also not any of the intermediate images). The images might be distributed to other people and these people might thus gain access to your key. Note that the release images that are produced by these scripts may store copies of all files in the mounted directories, possibly including hidden directories and sensitive data.
- Mount an additional volume containing your ssh key
To use a mounted ssh key, please add the mount instructions in the settings.bash file. Don't copy or link your personal ssh key to any of the default mounted volumes.
Updated expression in settings.bash for mounting the folder <host_folder_containing_ssh> as volume in the container at <mounted_ssh_folder_in_container>:
export ADDITIONAL_DOCKER_RUN_ARGS=" \
--dns-search=dfki.uni-bremen.de \
-v <host_folder_container_ssh>:<mounted_ssh_folder_in_container> \
"Then, in the container, you will have to add that key to the container's ssh-agent.
-
Log in your container
-
Add the key that you mounted from the host to the container's ssh-agent
eval "$(ssh-agent -s)"
ssh-add <mounted_ssh_folder_in_container>/id_rsa