Cache hash computations for repro

[falko17]

This adds a _hash_cache to repos to store computed hashes and metadata in memory when a command is run. This way, the hash does not need to be recomputed in every step and _build can be skipped (at least for dependencies, outputs obviously need to update the cache).

It also adds some tests for the basic cases (hash can be used, hash must be overwritten, and hash must be reset when repo is reset).

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ falko17
❌ MeganKW
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 95.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.01%. Comparing base (2431ec6) to head (b2e55e6).
⚠️ Report is 165 commits behind head on main.

Files with missing lines	Patch %	Lines
dvc/output.py	84.61%	0 Missing and 2 partials ⚠️
dvc/repo/__init__.py	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10927      +/-   ##
==========================================
+ Coverage   90.68%   91.01%   +0.33%     
==========================================
  Files         504      505       +1     
  Lines       39795    41027    +1232     
  Branches     3141     3248     +107     
==========================================
+ Hits        36087    37340    +1253     
- Misses       3042     3047       +5     
+ Partials      666      640      -26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[skshetry]

What are you trying to optimize for? Are you trying to reduce multiple build() calls within a single stage run, or are you trying to reduce for a complete repro session?

If former, I think we should reorganize/rearchitect so that we don't make duplicated calls, avoiding the need for caching.

If you are optimizing for the whole session, it may not be safe because it could have been modified, maybe by the user, or a different dvc process executing separately.
(I don't think your code even works for that scenario because we reset the repo after each run).

DVC does not re-hash files, so successive build() will only stat the directory/files, unless their mtime/inode/size changes. Which should be fast enough on modern machines. If stat is slow, then everything will be slower for you.

If you can share the perf numbers (before and after), then that would be great. But I'd avoid adding caching as much as possible.

[falko17]

Are you trying to reduce multiple build() calls within a single stage run, or are you trying to reduce for a complete repro session?

The latter. Looking into it more now, I think you're right with all of your criticisms, it seems like I was a bit too zoomed-in to see the bigger picture 😅

We could technically change the architecture here to store the cache across repo resets, but I agree with you that this isn't worth the added complexity (now that I know DVC doesn't re-hash files) since it's just for stat calls. I don't really think there's any reason to keep this change, so we can just close it IMO.

[skshetry]

The hashes are saved into a sqlite database, which is used more like a key-value store.
If mtime, inode and size of the file does not match with what we have in the db or is missing, we hash the file. Otherwise, we reuse the hashes from the db.

Some additional references where we get values from db and where we save them back:

• https://github.com/treeverse/dvc-data/blob/ba5e2c4e37e051a40e5905e25f821cd8b6c126ea/src/dvc_data/hashfile/build.py#L123-L126

• https://github.com/treeverse/dvc-data/blob/ba5e2c4e37e051a40e5905e25f821cd8b6c126ea/src/dvc_data/hashfile/build.py#L140

[falko17]

Thanks for the explanation and links to the codebase. Yeah, in this case I don't think the sort of caching this PR adds is of any use, so I'll close it now.