Add Python example for thread-safe Define using rdfslot_ by gayatripadalia · Pull Request #21798 · root-project/root

gayatripadalia · 2026-04-05T17:57:56Z

Summary

This PR adds a Python tutorial demonstrating thread-safe patterns in ROOT RDataFrame using rdfslot_.
In C++, ROOT provides DefineSlot and RedefineSlot for thread-safe operations by exposing the slot index to user-defined callables. These APIs are currently not available in PyROOT.

This tutorial shows how to reproduce the same behavior in Python by explicitly forwarding rdfslot_, enabling safe and lock-free access to per-slot resources in multi-threaded workflows.

Motivation

Issue #20839 highlights the lack of Python examples for thread-safe operations similar to DefineSlot.

This contribution provides a practical Python example of slot-based computation and demonstrates safe usage of mutable state in multi-threaded RDataFrame workflows, reflecting the design and intent of DefineSlot using existing PyROOT features.

Related Issue

Closes #20839

Adds a tutorial demonstrating a thread-safe pattern in Python using rdfslot_, as a workaround for DefineSlot which is not available in Python. Closes #20839

vepadulano

Dear @gayatripadalia ,

Thanks for attempting to increase the coverage of our suite of tutorials! The need to have more thread-safe examples is understood. Nonetheless, this PR does not contribute enough in that direction. Your example does not use the rdfslot_ implicit column in any meaningful manner, and it does not include examples of DefineSlot with comments motivating why that operation would be needed.

Improve rdfslot_ tutorial with thread-safe use cases.

gayatripadalia · 2026-04-08T15:26:06Z

Dear @vepadulano,
Thank you for your helpful feedback.
I’ve updated the PR to better demonstrate meaningful usage of rdfslot_, including clearer examples of per-slot resource handling and explanations motivating thread-safe patterns similar to DefineSlot.
I wanted to check if the current version now aligns with what you had in mind, or if you would suggest any further improvements, especially regarding the examples or level of detail.
Thanks again for your guidance.

gayatripadalia · 2026-04-18T14:43:31Z

Dear @vepadulano and @couet
I’ve updated the tutorial to use rdfslot_ more meaningfully, with clear examples of per-slot RNG and histograms to reflect DefineSlot-like thread-safe patterns.
Could you please take another look and let me know if further changes are needed?
Thank you.

vepadulano

Dear @gayatripadalia ,

I appreciate your effort, but I am not sure I see the direction you want to take. I have two major concerns about the tutorial you are proposing:

Have you tried actually running this code? From a cursory look, most of it is wrong and will probably not work as you expect. Please send back a full reproducer showing that the code works, including a recipe on the exact steps to run the code and the platform you ran it on (with screenshots showing the correct output)
How is this tutorial improving/different from the other one proposed at #20898 ? That tutorial is already showing usage of thread-safe RNGs with DefineSlot.

vepadulano · 2026-04-21T07:00:31Z

+## random-number generator or one histogram per thread — eliminating data races
+## without requiring a mutex.
+##
+## These APIs are currently not available in PyROOT.


This is false

vepadulano · 2026-04-21T07:01:14Z

+## In short:
+## Shared resource (unsafe): one RNG / histogram shared across threads
+## Per-slot resource (safe): one RNG / histogram per slot via rdfslot_


This summary is redundant

vepadulano · 2026-04-21T07:01:43Z

+# Helper: print a section banner so the terminal output is easy to follow
+def banner(title: str) -> None:
+    width = 72
+    print("\n" + "=" * width)
+    print(f"  {title}")
+    print("=" * width)


This helper is really redundant, in tutorials we favour less lines of code to ease the reader's experience.

vepadulano · 2026-04-21T07:01:52Z

+    print("\n" + "=" * width)
+    print(f"  {title}")
+    print("=" * width)
+# Background: DefineSlot in C++ vs the Python workaround


Unwarranted comment

vepadulano · 2026-04-21T07:02:38Z

+# In C++, you would write the following to safely smear values per-thread:
+#
+#   // One RNG per slot — constructed before the event loop
+#   unsigned int nSlots = df.GetNSlots();
+#   std::vector<TRandom3> rngs(nSlots);
+#   for (unsigned int i = 0; i < nSlots; ++i) rngs[i].SetSeed(i + 1);
+#
+#   auto df_smeared = df.DefineSlot(
+#       "smeared_pt",
+#       [&rngs](unsigned int slot, double pt) {
+#           // `slot` is guaranteed unique per thread — no data race
+#           return pt + rngs[slot].Gaus(0.0, 0.01 * pt);
+#       },
+#       {"true_pt"}
+#   );


Too many details, if anything there should be a full C++ tutorial to link to as an example. Also, beware of false-sharing.

vepadulano · 2026-04-21T07:04:57Z

+    rng_init = np.random.default_rng(seed=42)
+    true_pt  = rng_init.normal(loc=50.0, scale=10.0, size=N_EVENTS).astype(np.float64)
+    true_eta = rng_init.normal(loc=0.0,  scale=2.0,  size=N_EVENTS).astype(np.float64)
+    # Wrap numpy arrays as ROOT RVecs so RDataFrame can consume them directly


Redundant details

Suggested change

# Wrap numpy arrays as ROOT RVecs so RDataFrame can consume them directly

# Read the numpy arrays directly with RDataFrame

vepadulano · 2026-04-21T07:05:36Z

+    banner(f"Implicit MT enabled  |  slots used = {n_slots}")
+    print(f"\n  Dataset: {N_EVENTS:,} events  |  columns: true_pt, true_eta")


See double use of banner and then print, doesn't really help the tutorial.

vepadulano · 2026-04-21T07:06:25Z

+    # Wrong approach (DO NOT DO THIS):
+    #   shared_rng = np.random.default_rng(seed=0)          # one RNG for all threads
+    #   df.Define("smeared_pt",
+    #             lambda pt: pt + shared_rng.normal(0, 0.01*pt),  # RACE CONDITION
+    #             ["true_pt"])


Do not show wrong approaches in tutorials

vepadulano · 2026-04-21T07:08:47Z

+    banner("Summary")
+    print("""
+  The implicit column rdfslot_ is the Python equivalent of the `slot`
+  argument provided by C++ DefineSlot / RedefineSlot.  By listing
+  "rdfslot_" first in the column list and receiving it as the first
+  parameter of any callable (Define or Foreach), you can:
+    • Index into per-slot RNG instances → lock-free random smearing
+    • Fill per-slot histograms          → lock-free histogram filling
+    • Accumulate per-slot partial sums  → lock-free custom aggregations
+  In every case the pattern is:
+      per_slot_resource = [Resource(seed=s) for s in range(n_slots)]
+      def my_func(slot, *columns):
+          # per_slot_resource[slot] is owned by exactly one thread
+          return per_slot_resource[slot].compute(*columns)
+      df.Define("result", my_func, ["rdfslot_", "col_a", "col_b", ...])
+  This pattern serves as a practical replacement for DefineSlot until that API
+  becomes available in PyROOT.
+""")


Absolutely irrelevant

vepadulano · 2026-04-21T07:08:58Z

+    display_a = df_smeared.Define(
+        "slot_id", "rdfslot_"          # expose slot index as a named column
+    ).Display(["slot_id", "true_pt", "smeared_pt"], nRows=8)
+    display_a.Print()


This is the wrong way of using RDataFrame in general, you're triggering the computation graph twice when requesting the Display.

gayatripadalia · 2026-04-21T08:17:16Z

Dear @vepadulano,
Thank you for your detailed feedback and guidance, and for taking the time to review my contribution.
After reviewing your comments, I realize that this contribution needs significant rework and may not be aligned with the current direction or existing tutorials. I apologize for any inconvenience caused during the review.
Since this is my first contribution, I would prefer to step back, learn more about the codebase and guidelines, and come back with a better approach.
I will close this PR for now and revisit the idea after further understanding.
Thank you again for your time and suggestions.

Add Python example for thread-safe Define using rdfslot_

89dbcf9

Adds a tutorial demonstrating a thread-safe pattern in Python using rdfslot_, as a workaround for DefineSlot which is not available in Python. Closes #20839

gayatripadalia requested a review from couet as a code owner April 5, 2026 17:57

dpiparo requested a review from vepadulano April 6, 2026 10:06

dpiparo assigned vepadulano Apr 6, 2026

vepadulano requested changes Apr 8, 2026

View reviewed changes

Update df_python_threadsafe_example.py

c96eec1

Improve rdfslot_ tutorial with thread-safe use cases.

gayatripadalia requested a review from vepadulano April 8, 2026 15:18

vepadulano requested changes Apr 21, 2026

View reviewed changes

gayatripadalia marked this pull request as draft April 21, 2026 07:57

gayatripadalia closed this Apr 21, 2026

vepadulano mentioned this pull request Jun 8, 2026

Add Python example for thread-safe Define using rdfslot_ #22518

Closed

	# Wrap numpy arrays as ROOT RVecs so RDataFrame can consume them directly
	# Read the numpy arrays directly with RDataFrame

		banner(f"Implicit MT enabled \| slots used = {n_slots}")
		print(f"\n Dataset: {N_EVENTS:,} events \| columns: true_pt, true_eta")

Conversation

gayatripadalia commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Related Issue

Uh oh!

vepadulano left a comment

Choose a reason for hiding this comment

Uh oh!

gayatripadalia commented Apr 8, 2026

Uh oh!

gayatripadalia commented Apr 18, 2026

Uh oh!

vepadulano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gayatripadalia commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gayatripadalia commented Apr 5, 2026 •

edited

Loading