Enable different HOST_DIRECTORY

[jbehley]

Brief Description

As we need a different host_directory not located in /codabench, but a different path due to mounting points, e.g., /other/path/codabench, we run into the problem that the worker raised a permission exception. As the directories are created inside the container of the worker, we identified the problem as not using the right location.

Thus, this PR should make it possible to have a different directory on the host.

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[Didayolo]

Hi @jbehley,

Thank you very much for your contribution.

I have the following doubt regarding your change.

Having first the directory creation:

        # Create if necessary
        os.makedirs(path, exist_ok=True)

And then the path concatenation:

        path = os.path.join(HOST_DIRECTORY, path)
        return path

It means that the path returned by the function is not the path passed to makedirs. It looks like we have then no guarantee that the directory exists. I may need to check it out more in depth I feel like this could introduce a bug.

[ihsaan-ullah]

I think this PR needs to be reabsed after #2294 and #2303 are merged into develop

[jbehley]

I think I have to close it anyway; I wasn't particularly careful with adding unrelated changes into the branch.

Still the problem is not resolved...in our current setup, I still need to run my customized version, which I synced to the latest version.

I'm still confident that the order should be different; otherwise, one tries to create a directory inside the container that cannot be accessed.

[ihsaan-ullah]

@jbehley Feel free to open a new PR when it is ready. In any case, there is no harm in adding details about the tweaks that work for you to this PR or another one, just to keep something for other users who may need this in the future

[jbehley]

I opened a new pull request with just the required change: #2311