Poc for a RecipesTable without Sqlite#19734
Conversation
AbrilRBS
left a comment
AbrilRBS
left a comment
There was a problem hiding this comment.
Not a full review yet, but the new performance tests look great, +1
bentonj-omnissa
left a comment
bentonj-omnissa
left a comment
There was a problem hiding this comment.
I am unsure if you're going ahead with this but we're very interested in seeing something like this progress - we have had quite a few CI failures from sqlite lock contention, additionally it would be great if the same conan directory could be shared by multiple simultaneous conan processes.
If you are intending to support multiple simultaneous conan processes then you'd have to change your data layout slightly, e.g. consider two processes conan install the same package at the same time, if one finishes before the other and goes on to start a build whilst the other process is still writing to the package directory then you could get half written files being used during compilation. One way to solve that could be that packages are always installed to a unique random directory path and then that path is written into the package path after the installation, yes you could end up with the same package installed twice but its a reasonable trade off to support multiple concurrent installs to a shared conan home folder.
| msg = None | ||
| for attempt in range(_READ_JSON_MAX_RETRIES): | ||
| try: | ||
| with open(tmp, "w", encoding="utf-8") as f: |
There was a problem hiding this comment.
This is not atomic because the tmp file uses the same filename for every run, so two simultaneous write_json_atomic can race over the same tmp file, as in another process/thread could write tmp inbetween this one writing tmp and calling os.replace.
|
|
||
| def remove(self, ref: RecipeReference): | ||
| rev_dir = self._revision_dir(ref) | ||
| assert os.path.isdir(rev_dir) |
There was a problem hiding this comment.
you wouldn't be able to assert os.path.isdir here if someone else called remove at same time
3 participants
Changelog: (Feature | Fix | Bugfix): Describe here your pull request
Docs: https://github.com/conan-io/docs/pull/XXXX