Resource creation race condition with OTEL_RESOURCE_ATTRIBUTES env var

[xrmx]

Users cannot assume that you can use OTEL_RESOURCE_ATTRIBUTES to override some resource attributes created by another resource detector because all the attributes collections by resource detectors are delegated to a thread pool, OTEL_RESOURCE_ATTRIBUTES included. The attributes are collected from the env var by OTELResourceDetector (otel entry point).

To make things simpler for users (like being able to override a value collected by a resource detector) we can probably put OTELResourceDetector out of the thread pool since it's not doing anything i/o bound or cpu intensitive and have it override any eventual duplicated collected attributes.
This may clash with the idea that all stable attributes are collected by OTELResourceDetector so another option would be to just move the collection of attributes from env vars out of resource detectors. Doing that will remove the ability to ignore env vars though.

[ravitheja4531-cell]

Thanks for filing this! I looked into the code and would love to work on a fix.

From get_aggregated_resources, the issue is clear — OTELResourceDetector is prepended to the detectors list but then submitted to the thread pool along with all other detectors simultaneously, so there's no deterministic guarantee it wins on attribute conflicts.

I'd like to go with Option 1: pull OTELResourceDetector out of the thread pool entirely, run it synchronously after all other detectors complete, and apply it as a final override merge. Since it only reads an env var, there's no I/O concern and no need for a timeout. This would also mean removing the current detectors = [OTELResourceDetector()] + detectors prepend inside get_aggregated_resources.

The key assumption is that Resource.merge() is called as otel_resource.merge(final_resource) so that OTEL_RESOURCE_ATTRIBUTES keys take precedence — wanted to confirm that's the intended merge semantics here.

Happy to open a PR with a fix and tests if this approach looks good to you. Let me know if you'd prefer Option 2 or have a different direction in mind.

[aabmass]

Can you clarify the bug? IIUC you're saying the OTEL_RESOURCE_ATTRIBUTES should always take priority but they are not today or it is nondeterministic.

I don't see what the thread pool has to do with it, since get_aggregated_resources looks like it maintains the order

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/resources/__init__.py

Lines 522 to 523 in e301732

	futures = [executor.submit(detector.detect) for detector in detectors]
	for detector_ind, future in enumerate(futures):

[herin049]

get_aggregated_resources submits all the detector.detect functions to the thread pool at once which is what leads to the non-deterministic ordering. It also looks like there is also an additional source of non-determinism in the loading of the entrypoints due to the use of a set:

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/resources/__init__.py

Lines 198 to 205 in e301732

	otel_experimental_resource_detectors: Set[str] = {"otel"}.union(
	{
	otel_experimental_resource_detector.strip()
	for otel_experimental_resource_detector in environ.get(
	OTEL_EXPERIMENTAL_RESOURCE_DETECTORS, ""
	).split(",")
	if otel_experimental_resource_detector
	}

I think the easiest is to just always execute the OTELResourceDetector after the detectors in the threadpool. Alternatively, if we want to we could add a precedence function that can be optionally overridden by resource detectors to allow user-defined resource detectors to override others, although this is probably overkill.

[aabmass]

get_aggregated_resources submits all the detector.detect functions to the thread pool at once which is what leads to the non-deterministic ordering.

I could be missing something, but it iterates over the futures in order #5054 (comment), right? So I think they are merged in the same order as they were originally passed.

I think the easiest is to just always execute the OTELResourceDetector after the detectors in the threadpool.

SGTM. Maybe we can also sort the other entry point loaded ones by name so that it's at least reproducible.