Add support for detection of unused java dependencies#29770
Add support for detection of unused java dependencies#29770JonathanPerry651 wants to merge 11 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
6bb50db to
ec8091f
Compare
There was a problem hiding this comment.
A lot of comments but I really like this! Thank you very much!
Two more requests:
- Can you benchmark a clean bazel build on your machine with no unused deps checks, and again with warning, to estimate how much this increases Java compilation?
- Can you log the warnings from bazel and attach that file to the PR to see where bazel is failing here?
|
|
||
| @Test | ||
| public void testUnusedDepsTagOptIn() throws Exception { | ||
| // Global unused_deps is off by default |
There was a problem hiding this comment.
Set it off explicitly in case we update the default in the future.
| } | ||
|
|
||
| @Test | ||
| public void testUnusedDepsFlagPropagation() throws Exception { |
There was a problem hiding this comment.
Nit(naming): For these tests, use testUnusedDeps_flag, testUnusedDeps_tagOptIn, and testUnusedDeps_tagOptOut. This helps to clearly group these together and clarify what the difference between them is.
|
|
||
| @Test | ||
| public void unusedDepsFlagExposed_error() throws Exception { | ||
| scratch.file( |
There was a problem hiding this comment.
Can the starlark setup be put in a shared method for these two tests? It helps to minimize the similarities so that the differences stand out more.
| /** Converter for the --strict_*_deps option. */ | ||
| public static class StrictDepsConverter extends EnumConverter<StrictDepsMode> { | ||
| /** Converter for the dependency checking/handling option. */ | ||
| public static class StrictDepsConverter extends EnumConverter<DepsCheckingMode> { |
There was a problem hiding this comment.
This should probably also rename.
⚡ Java Compile Times & Performance ImpactTo evaluate the performance impact of the new unused dependencies check on the compiler, we ran a full clean build of the Bazel Java codebase (
Conclusion: The compilation critical path is virtually identical under both modes. Enabling unused dependency warnings has no measurable performance overhead on compiler execution. 📊 Unused Dependencies Findings ReportA full clean build with warnings enabled identified 3,100 unique unused direct dependencies across the Bazel repository.
The results have been published in a public Gist here: To make this easy to work with (and avoid GitHub UI browser truncation limits for large tables):
You can download the raw # Download the raw buildozer commands file:
curl -sL https://gist.githubusercontent.com/JonathanPerry651/3318f32c52e93779dab2811863456e07/raw/buildozer_commands.txt > buildozer_commands.txt
# Run buildozer in batch mode to clean up all unused dependencies:
buildozer -f buildozer_commands.txt |
katre
left a comment
There was a problem hiding this comment.
Thanks for running the analysis. I admit I'm happily surprised that this doesn't have an effect on build times. Also I am kind of startled by how many unused deps are in Bazel itself. We should definitely fix those: want to shard out your report and start submitting PRs, or would you prefer someone else to handle cleanup work?
|
Wow thank you for the incredibly quick turnaround. I’m happy to spawn some sharded PRs |
|
Could you make it so that warnings are only shown (or even generated) for targets in the main repo? Users can't really do anything about unused deps of deps and the buildozer fixups wouldn't work either. |
|
@fmeum that's a really good point thank you, let me see how I can do that |
|
heads up - there's an undesirable behaviour here in that it will flag as unused a dependency that got injected due to an exports clause in an explicit direct dep. that's obviously outside the control of the consuming target, so I'll need to figure out a way around it; I should hopefully get that fixed this afternoon BST. (RESOLVED) |
|
@bazel-io fork 9.2.0 |
cushon
left a comment
There was a problem hiding this comment.
Thanks for raising this!
I made a first pass over the changes and left some comments, but want to give this more thought.
there is prior art in out-of-bazel detection, that requires extra repo-local build actions (via analysis-time-expensive aspects), extra CI tasks, or perhaps bot workflow to remediate the source code post-hoc. None of these are as attractive as bazel-native support.
Could you say more about experience you've had with the existing support for unused deps, and the workflow you want with the new proposal?
One potential disadvantage of doing this in JavaBuilder as an error is that it creates more friction for development, it's possible deps will be temporarily unused while code is being iterated on, and having a blocking error for that is less helpful than having something that runs during code review or on CI.
One high level question I have here is about whether we want to have two solutions for this, or we can make the existing support more useful, or perhaps whether we should try to converge the existing support on the proposed new approach. I'd be nice to avoid supporting and maintaining two different solutions for unused deps in the Java rules.
|
|
||
| private final StrictJavaDeps strictJavaDeps; | ||
| private final DepsCheckingMode strictJavaDeps; | ||
| private final DepsCheckingMode unusedDeps; |
There was a problem hiding this comment.
Is there shared code that relies on having strictJavaDeps and unusedDeps both be the same type? I'd prefer to have separate enum types for them so the strict deps and unused deps mode can't get mixed up
| this.targetDeclaredDeps = targetDeclaredDeps; | ||
| } | ||
|
|
||
| public ImmutableSet<String> getTargetDeclaredDeps() { |
| } | ||
|
|
||
| @CanIgnoreReturnValue | ||
| public Builder addTargetDeclaredDeps(Collection<String> targetDeclaredDeps) { |
| Map<Path, Dependency> implicitDeps = dependencyModule.getImplicitDependenciesMap(); | ||
| String targetLabel = | ||
| dependencyModule.getTargetLabel() == null | ||
| ? "<target>" |
There was a problem hiding this comment.
It looks like this is just used in the buildozer command below, is that right? If so I think it'd be better to omit the buildozer command if there isn't a label instead of providing a command with <label>. Also consider using Optional<String> here
| return target; | ||
| } | ||
|
|
||
| private static String normalizeLabelForComparison(String label) { |
There was a problem hiding this comment.
Can you add javadoc? (What kind of normalization is this doing? Why does the label need normalization?)
| return customJavacOpts; | ||
| } | ||
|
|
||
| private DepsCheckingMode getUnusedDepsMode() { |
There was a problem hiding this comment.
I think we may want to consider other ways of configuring this, like a new attribute instead of relying on tags, and perhaps also a global configuration mechanism in java_toolchain that could use a package_group instead of requiring attributes on individual rules
There was a problem hiding this comment.
Happy to add a new attr instead of tags - you’re right that’s definitely a clearer api
I can also attack your second option. I’m not sure I fully get it though sorry - could you sketch an example of what you’re thinking please?
Thank you, I appreciate the time.
With the existing workflow, there’s no way of knowing whether a fully green test //… will in fact result in a desirable diff. So you end up either failing late in a pipeline, and having to root around to see what happened, or else have bots which go and fix up the code later. And the thing about unused deps is that sometimes they are actually used at runtime. Discovering that you got that wrong later makes it that much more likely you’ll break prod. Generally I really like keeping all my builds and tests in bazel, as that means they are guaranteed to work in every developer’s environment.
Agreed. But if that becomes an issue, people could easily decide to set to WARN on, say, an IDE sync config, but ERROR everywhere else?
That’s a great point. I’ll have to dig a bit to see if there’s stuff in the existing support that this doesn’t cover. Let me come back to you on Monday. |
d29ceda to
43619a8
Compare
|
@cushon - thanks again for your feedback. I've introduced the new attr over in rules_java - it's way nicer, for sure. What's the workflow for getting a cross-repo change like this through - how do I show you what the code (and green tests!) would look like over here before the change over there is merged? Re the functionality of the prior art: I reviewed the code over at https://github.com/bazelbuild/buildtools/blob/main/unused_deps/unused_deps.go, and I don't think there's anything there that's not covered here. I also looked at https://github.com/bazel-contrib/unused-jvm-deps, and again, didn't see anything we're missing here. I'm thinking I'll ping the person who pointed me at the second one to see if they agree though - I totally agree with the sentiment that we don't want two. |
| } | ||
|
|
||
| /** Returns the unused dependency checking setting. */ | ||
| public UnusedDepsMode getUnusedDeps() { |
There was a problem hiding this comment.
Given that the new getTargetDeclaredDeps returns a list of dependencies, this should be renamed to getUnusedDepsMode to clarify that it does not return the list of unused dependencies.
The getStrictJavaDeps method should also be renamed.
| * returning the part starting with "//") and canonicalizing the target label format (e.g., | ||
| * removing the target name if it matches the package name). This ensures that labels from | ||
| * different repository contexts or formatting styles can be compared correctly during unused | ||
| * dependency checking. |
There was a problem hiding this comment.
Doesn't this mean that if my java_library declares a dependency on @foo//util:library and @bar//util//:library, but only one is actually used, they'll both match and be treated the same?
Removing the repository name seems very dangerous for this analysis, I'd like to hear more about why it's needed and safe.
There was a problem hiding this comment.
wow yes that was not clever sorry, I'll fix.
|
I'm as averse to bureaucracy as anyone else, but ISTM such a change would have benefited from following the process outlined in https://bazel.build/contribute/patch-acceptance. In particular item (2) and possibly (3), would have been useful here. I would request holding off on investing more time/effort on changes before we reach some consensus on the general shape this should take. Some points for discussion:
Footnotes
|
|
@hvadehra - fair, I agree that this is wide-ranging, and your feedback makes a lot of sense. shall I start a new Issue, and describe this from scratch, and we iterate there? Or a design doc? I'm open to either (though I've done neither so bear with me!). |
|
@hvadehra re the --direct_dependencies point: I think this is the line that makes the rirst-level exports indistinguishable from the true directs: https://github.com/bazelbuild/rules_java/blob/35e8391a1c37dd5941ba594a01dcb08d3985cbd8/java/private/java_info.bzl#L612 re the 'some other attr than deps': I strongly agree. I'd really like a way to tell bazel that problematic dep X is actually there because it is present in attr A of target T, thus allowing users to guide it through macros and custom rules, in the end emitting a buildozer snippet that will actually fix their problem. We have some horrible tag-based contraptions in our codebase to make that work. I don't have any good ideas for what that API/implementation should look like, but I'm very up for trying to figure it out! |
A google doc might be best for everyone to able to contribute easily (https://bazel.build/contribute/design-documents#markdown-versus-gdocs) but I'm okay with any option tbh - even here on the PR works for me (if that's okay with everyone else).
Ah right. I guess we will need some sort mapping of Not sure if it's better to introduce new parameters or enhance existing ones. Just thinking out loud: the format of such flags (whose values must contain both |
|
Sorry for the radio silence, I got pulled on to other stuff, but I'm back! @hvadehra I added an MD file describing the approach. I really like your toolchain-based java_package_configuration, so I've dropped everything else in favour of that. I also think that that kills the need for warn, so that's gone too. Please ignore the current state of the code and focus on the markdown. Also I made the call that the nicer handling of deps should be a subsequent orthogonal change, is that right? (A thought - should I close this PR, and start a pristine new one with just the markdown and a back reference? or is it bete for everyone to have it all in one place?) |
| ### D. JavaBuilder Compiler Plugin (`StrictJavaDepsPlugin.java`) | ||
| Update the strict Java dependencies plugin in JavaBuilder to: | ||
| - Retrieve `--unused_deps` and `--target_declared_deps` from the options parser. | ||
| - Extract target labels from the manifests of the jars used during compilation. |
There was a problem hiding this comment.
I think this will run into the issue of embedded labels possibly not matching what is specified on the target's deps. I expect this will happen with external repo deps as well as aliases.
One idea could be to pass just the compile jars for a dep without the exports. IOW instead of looking at --direct_dependencies (i.e. dep[Javainfo].compile_jars) we pass in the new parameter --target_declared_deps with the value depest([output.compile_jar for dep in deps for output in dep[JavaInfo].java_outputs if output.compile_jar])
WDYT?
|
|
||
| ```java | ||
| // inside JavaCompileActionBuilder.java | ||
| if (unusedDepsMode != DepsCheckingMode.OFF) { |
There was a problem hiding this comment.
with only an off and error mode now, we can maybe get away with just --target_declared_deps instead of two flags/params?
|
|
||
| Core Bazel's native APIs and action builders must be updated to receive the metadata from `rules_java` and forward them to JavaBuilder. | ||
|
|
||
| ### A. Package Configuration Provider Update (`JavaPackageConfigurationProvider.java`) |
There was a problem hiding this comment.
this shouldn't be necessary. do you forsee any need to read these in the native Bazel code? IIUC the rules_java code need only read these fields and decide what parameters to pass to the compilation API.
|
@hvadehra - superb. I've addressed your comments, and it's looking so nice. Thank you! Let me know if you're good with this approach - I am going to implement it now as an optimistic branch prediction. |
63ae0f2 to
2816f47
Compare
…rs_to_verify is null/missing
…ct_dep_jars_to_verify
d75ae52 to
8491005
Compare
56d4deb to
c7ce9d7
Compare
|
@hvadehra since we decided in the other PR not to move all the code across to rules_java, I think this PR is ready for review, along with the rules_java side changes in bazelbuild/rules_java#359 |
|
|
||
| ```starlark | ||
| # Inside rules_java compilation helper: | ||
| direct_dep_jars_to_verify = [] |
There was a problem hiding this comment.
Any reason this isn't just a dict mapping jar to label?
| List<Artifact> directDepJarsToVerify, | ||
| List<String> directDepLabelsToVerify, |
There was a problem hiding this comment.
this does not match the current impl, but wouldn't a Dict<Artifact, Label> work here?
| result.addPath("--direct_dep_jar", directDepJarsToVerify.get(i)); | ||
| result.add("--direct_dep_label", directDepLabelsToVerify.get(i)); |
There was a problem hiding this comment.
I'm a little wary of having two flags that need to be in sync like this. What about a single flag that specifies the artifact and the label with some seperator?
Also, to avoid confusion with --direct_dependencies perhaps a different name? WDYT about --declared_deps
There was a problem hiding this comment.
the reason I didn't do artifact label was just because I wasn't sure what a guaranteed-never-used-separator-char would be. I'm happy to change that decision - what separator should we use?
| ``` | ||
|
|
||
| ### B. Compilation Action Builder Updates (`JavaCompileActionBuilder.java`) | ||
| Update the builder to receive the jar-to-label mapping and translate it into parallel command-line flags for JavaBuilder. The presence of these flags instructs JavaBuilder to execute the check: |
There was a problem hiding this comment.
Just thinking out loud here - wondering if we can sidestep any version skew issues and allow us to iterate on the rules_java <> JavaBuilder API more freely without involving Bazel. What we'd really like is for Bazel to just pass-through whatever it gets.
If done naively (by constructing the list of flags in rules_java), this would increase Bazel memory for the action (since we'd be expanding the artifact paths to strings during analysis). We can avoid that with some cleverness at the cost of some complexity. For example, we pass in a struct(obj, mapper_fn), and bazel invokes the mapper function on the obj to get the arguments to pass to the JavaBuilder command.
There was a problem hiding this comment.
@hvadehra are you thinking about doing this for all args or just these?
There was a problem hiding this comment.
I'm thinking if we add this one extra struct(obj, func) arg to this API with Bazel expanding it to a list of args by invoking the starlark function (+ cherrypick this change to 8.x, 9.x) we can be free of having to make any changes to Bazel in the future.
If we do want to go down this path, perhaps two functions might make sense. one that maps obj to [str] for arguments, and one to map obj to depset(file) for inputs. @cushon WDYT?
There was a problem hiding this comment.
@cushon - let me know which way you think we should go here please; Im happy to make the changes to the PR
Description
Motivation
Unused deps bloat the build graph, and while there is prior art in out-of-bazel detection, that requires extra repo-local build actions (via analysis-time-expensive aspects), extra CI tasks, or perhaps bot workflow to remediate the source code post-hoc. None of these are as attractive as bazel-native support.
Build API Changes
Yes this adds a new command line arg (marked experimental) to control default behaviour, plus a tag-based API to allow incremental adoption.
1 - No
2 - Yes, it's opt-in
3 - Not breaking (unless we eventually want to default to true)
Checklist
Release Notes
RELNOTES[NEW]: Add detection of unused direct dependencies for java targets.