feat: add runtime path support for foreign_cc outputs

[jsun-splunk]

This PR does 2 things:

1. Adds a runtime_library_search_directories feature to enable rpath on outputs within bazel.
2. Adds a runtime_execution rule to wrap foreign_cc outputs into an executable target that is compatible with FilesToRunProvider.

runtime_library_search_directories feature

Add opt-in runtime library search directory derivation for foreign_cc
outputs. This lets foreign-built binaries and shared libraries resolve
shared libraries from deps, dynamic_deps, and this rule's own declared
shared-library outputs without relying on loader environment variables such
as LD_LIBRARY_PATH.

Public attrs:

• runtime_library_search_directories
• additional_dynamic_runtime_library_search_origins
• additional_executable_runtime_library_search_origins

Example:

configure_make(
    name = "python",
    out_binaries = ["python3.10"],
    out_shared_libs = ["libpython3.10.so"],
    runtime_library_search_directories = "enabled",
    additional_dynamic_runtime_library_search_origins = [
        "lib/python3.10/lib-dynload",
    ],
    deps = [":openssl"],
)

runtime_library_search_directories accepts auto, enabled, and
disabled. The global build setting defaults to disabled, so existing
targets keep the previous behavior unless a target opts in or the build
setting is enabled.

Default origins are derived from declared output File.short_path values.
Shared-library link actions use declared shared-library output directories.
Executable link actions use declared binary output directories. Additional
origins are install-tree-relative paths joined under this rule's INSTALLDIR.

The implementation derives dependency origins from LibraryToLink dynamic
libraries and adds Bazel _solib sibling search paths for solib symlink
layouts. Runtime search derivation is skipped for Windows C++ toolchains.

Wire the runtime search path flags as common implementation flags.
However, support is limited to cmake, configure_make, make and
meson. Ninja and boost support is currently disabled.

runtime_executable rule to used with FilesToRunProvider compatible consumers.

Introduce a runtime_executable rule for turning selected foreign_cc binary outputs into executable Bazel targets. This cannot live directly on the existing foreign_cc rules because foreign_cc outputs are not always expected to contain a binary, so those rules cannot always expose an executable.

The adapted executable may require runtime_library_search_directories = "enabled" on the producing foreign_cc target when it depends directly or transitively on foreign_cc-produced shared libraries.

runtime_executable is a more Bazel-native version of runnable_binary because it exposes the executable through DefaultInfo.files_to_run for downstream consumers. The API is intentionally kept similar to runnable_binary. For now, it cannot completely replace runnable_binary because runtime_library_search_directories defaults to "disabled" to minimize blast radius while these features mature.

[matt-sm]

This creates a raw Bash script as the rule executable. Existing runnable_binary uses rules_shell's sh_binary wrapper, and this repo runs examples on Windows, so can we confirm this works for bazel run and tools = [...] on Windows? If not, this may need a rules_shell launcher, a .cmd-compatible wrapper, or an explicit target compatibility restriction.

[jsun-splunk]

I've refactored to use sh_binary as the frontend for runtime_executable.

runtime_executable is now a macro that wraps both the private _runtime_executable_wrapper and the public sh_binary.

[matt-sm]

Just confirming - runtime_executable tests are run on windows?

[jsun-splunk]

They are current enabled on window. This includes analysis test and sh_tests.

The purpose of runtime_executable is to expose a files_to_run provider for a foreign_cc built binary. There isn't a restriction for that binary to be unix only. The rule will collect the generated binary, its runfiles and its deps all in one place as a files_to_run provider and expose that. The problem is since windows cannot use rpath, it doesn't actually guarantee everything will work together. However, that probably shouldn't stop the rule from running on windows since there are other ways to make things work.

[matt-sm]

Design question: should this be a warning, or should enabling runtime_library_search_directories with out_shared_libs require shared_ldflags_vars?

As written, the feature can be explicitly enabled but still silently produce shared libraries without the intended rpaths unless the user also knows the build-system- specific flag variable. That seems like a sharp edge for a feature that otherwise looks like a single opt-in switch.

[jsun-splunk]

I've considered this, it is a problem of wrapping bazel around another build system. If we force runtime_library_search_directories to be tied to shared_ldflags_vars when out_shared_lib is define, we will need the upstream to have a clear separation for exe and shared flags via makevars. Otherwise, the user cannot enabled runtime_library_search_directories at all, even though by default it will still work for its out_binaries.

[matt-sm]

Could we make this strict only for explicit per-target enablement?

• runtime_library_search_directories = "enabled": fail if out_shared_libs are present, shared runtime search flags were derived, and shared_ldflags_vars is unset.
• runtime_library_search_directories = "auto": keep the warning when enabled by the global build setting, so broad migrations can still proceed.

That keeps the executable-only/migration case possible, but avoids presenting explicit "enabled" as a single switch when shared outputs are only partially covered.

[jsun-splunk]

This essentially means:

• If enabled via attr in rule, use fail
• if enabled via build setting, use warn

which I think confuses more than clarifies.

I still feel like we should either:

1. tighten the API to use the feature by explicitly failing instead of warn or
2. keep the looser API and try to add better documentations.

[jsun-splunk]

Ok so I've landed on a variation of what you suggested and tightened the API.

The description on the attr reads:

    "runtime_library_search_directories": attr.string(
        doc = (
            "Controls whether this target derives runtime library search " +
            "directories. Use 'auto' to follow the global " +
            "@rules_foreign_cc//foreign_cc/settings:runtime_library_search_directories " +
            "build setting, 'enabled' to force it on for this target, or " +
            "'disabled' to force it off. Global enablement is a strict Unix " +
            "conformance switch: Unix targets that declare shared-library " +
            "outputs must support the rule-specific shared-link flag hook or " +
            "set runtime_library_search_directories = 'disabled'. This is " +
            "not supported for Windows C++ toolchains; explicit per-target " +
            "'enabled' fails on Windows, while 'auto' with global enablement " +
            "is ignored on Windows. On macOS, derived runtime search paths also " +
            "require compatible Mach-O @rpath install names and load " +
            "commands. This feature passes runtime search directories to " +
            "link actions, but does not rewrite installed dylib " +
            "IDs or load commands after the upstream install step. Upstream " +
            "builds may need to emit or preserve @rpath/... install names."
        ),
        mandatory = False,
        values = ["auto", "enabled", "disabled"],
        default = "auto",
    ),

Effectively this becomes:

Platform	runtime_library_search_directories attr	Global setting	Effective runtime-search state	Additional rule/output checks
Unix/macOS	disabled	any	Off	Fails only if additional_dynamic_runtime_library_search_origins or additional_executable_runtime_library_search_origins is set
Unix/macOS	auto	disabled	Off	Fails only if additional_dynamic_runtime_library_search_origins or additional_executable_runtime_library_search_origins is set
Unix/macOS	auto	enabled	On	For make / configure_make: if out_shared_libs is non-empty and shared linker flags exist, shared_ldflags_vars must be set. For meson: shared_ldflags_option must be set. ninja / boost_build fail because runtime search is unsupported.
Unix/macOS	enabled	any	On	Same as above: make / configure_make require shared_ldflags_vars; meson requires shared_ldflags_option; ninja / boost_build fail.
Windows	disabled	any	Off	No runtime-search checks
Windows	auto	disabled	Off	No runtime-search checks
Windows	auto	enabled	Off, silently ignored	No runtime-search checks
Windows	enabled	any	Unsupported explicit request	Always fails: Windows does not support explicit per-target runtime search

Rule	Runtime search on + out_shared_libs non-empty + shared linker flags exist	Required attr
make	Fails if missing	shared_ldflags_vars
configure_make	Fails if missing	shared_ldflags_vars
meson	Fails if missing	shared_ldflags_option
ninja	Always fails on Unix/macOS when runtime search is on	Unsupported
boost_build	Always fails on Unix/macOS when runtime search is on	Unsupported

[jsun-splunk]

Also pushed changeset, to refactor the analysis tests to runtime into 2 separate files.

[jsun-splunk]

and then another refactor of runtime_library_search_directories.bzl into 2 libraries instead.

[matt-sm]

That table above is useful - is it included in any docs?

[jsun-splunk]

added a doc here.

[matt-sm]

great - this needs a dedicated page

[matt-sm]

Its very difficult figuring out what these tests actually do because they are split between the func, declaration and partial.make.

Would make it easier to understand if they were self-contained. The paths tests didn't seem to have this same issue.

[jsun-splunk]

did a refactor so most of the relevant test info is in partial.make.

having clearer test structure was easier to do in unittest on the path tests than the analysistest here.

[matt-sm]

Two cases untested:

1. auto + global=disabled (non-Windows) → False/False. The only non-Windows auto test runs with global=enabled, so it proves auto can be on but not that it actually tracks the global setting down. A regression that hardcoded auto to on would still pass today.

2. Explicit enabled (non-Windows) → True/True. Every enabled scenario is the Windows failure path; the success path is never exercised, and nothing proves explicit enabled ignores a disabled global.

[jsun-splunk]

thanks, tests added in changeset.

[matt-sm]

There's no equivalent check for executable_ldflags_vars?

For make/configure_make, flags.cxx_linker_executable reaches the link only via executable_ldflags_vars (make_env_vars.bzl:179) — should out_binaries + runtime search enabled also enforce?

[jsun-splunk]

I don't think so.

The default linker flags used by foreign_cc are from cxx_linker_executable, so the rpath in those flags will never be silently dropped.

[matt-sm]

The matrix only covers "enabled" + plain out_shared_libs/out_binaries. "auto" mode, additional_*_runtime_library_search_origins, and out_data_dirs/out_data_files triggers are unit-tested over a fake ctx but never built+run — so wrong link-wiring (vs. correct derivation) wouldn't be caught.

Suggest adding at least an auto and an additional_*_origins target that run with cleared LD_LIBRARY_PATH.

[jsun-splunk]

I don't think adding "auto" mode will add much more coverage than what the analysistest in runtime_search_policy_tests.bzl offers.

out_data_dirs/out_data_files are used as anchors in runtime path to determine the installdir. This is also tested in unittest. I'm not convinced it warrants the additional test harness (to do a build with just out_data_* and not out_binaries/shared) required to test the e2e flow.

I've updated runtime_search_paths_test() to account for additional_*_runtime_library_search_origins and assert the expected results with them declared. However, I have not added tests to run or load binaries and shared library in those additional paths. The path derivation for these are already covered by unittests, the additional test harnessing required doesn't seem worth it at the moment.

[jsun-splunk]

slightly relevant. I've refactored the sh_tests using runtime_search_test.sh to a new runtime_search_test macro. This sits nicely with the existing runtime_search_paths_test macro. See changeset.

[matt-sm]

This override block — discard inherited RUNFILES_DIR/MANIFEST_FILE so a Bazel-built parent tool's runfiles don't shadow the wrapper's — has no execution coverage.

[jsun-splunk]

has no execution coverage.

do you mean testing coverage?

[matt-sm]

yes

[jsun-splunk]

Add a test for this was more complicated than I thought.

The scenario for which we need the overriding block is a little complicated. We need to replicate a scenario where a parent runfiles exists and is exported by the env var and the wrapper's runfiles also exists (adjacent to the wrapper) but not exported by the env var.

This happened when using a foreign_cc built python, expose with the executable_runtime rule to create a python toolchain and expose it as a files_to_run provider. Then the consuming rule need to use the files_to_run provider as a input or tool to an action instead of the the executable. I've basically try to replicate the parent and child runfile structure inside a temp directory.

Here is the changeset.

[matt-sm]

thats more complicated than I first thought

[jsun-splunk]

As discussed, this has been removed. The test setup was too complex and not worth the overhead.