Skip to content

Add element_type/name_pattern/level filters to get_elements, get_element_names, get_elements_dataframe#1401

Open
meyersrl wants to merge 3 commits into
cubewise-code:masterfrom
meyersrl:feat/element-filtering-on-get-elements
Open

Add element_type/name_pattern/level filters to get_elements, get_element_names, get_elements_dataframe#1401
meyersrl wants to merge 3 commits into
cubewise-code:masterfrom
meyersrl:feat/element-filtering-on-get-elements

Conversation

@meyersrl
Copy link
Copy Markdown
Contributor

Summary

Adds three optional, AND-composable filter kwargs — element_type, name_pattern, level — to get_elements, get_element_names, and get_elements_dataframe. Translates to OData $filter server-side. Refactors get_elements_by_level and get_elements_filtered_by_wildcard to delegate to the new path (behavior preserved).

Motivation

Today TM1py can count elements by type (get_number_of_leaf_elements, get_number_of_consolidated_elements, etc.) but can't list them without pulling the entire dimension and filtering in Python, or hand-writing MDX. This closes the gap with a small, additive API.

# Before — pull everything, filter in Python:
all_elements = tm1.elements.get_elements(dim, hier)
numeric_leaves = [e for e in all_elements if e.element_type == Element.Types.NUMERIC]

# After:
numeric_leaves = tm1.elements.get_elements(dim, hier, element_type="numeric")

# Or compose all three:
tm1.elements.get_element_names(
    dim, hier,
    element_type=["numeric", "consolidated"],
    name_pattern="Region*",
    level=1,
)

API

element_type accepts Element.Types enum, str ('numeric'/'string'/'consolidated', case-insensitive), int (1/2/3), or an iterable of any of those (OR-combined).

name_pattern is a glob with * wildcard, case- and space-insensitive (matching the semantics of the existing get_elements_filtered_by_wildcard). Translates to startswith, endswith, contains, or eq depending on * placement. ? raises ValueError.

level is exact match on the hierarchy level (0 = leaf).

All three default to None — purely additive, no breaking changes.

Design notes worth flagging

1. Incidental fix in get_elements URL. The original get_elements URL used ?select=Name,Type (missing the $ prefix), which TM1's OData endpoint silently ignored — the response always carried the full element payload. This PR corrects it to ?$select=Name,Type. Element.from_dict handles UniqueName, Index, and Attributes via .get(..., None) so no existing caller breaks. Same latent bug exists at get_edges (line 706) but is out of scope for this PR.

2. element_type overrides skip_consolidations. In get_elements_dataframe, when element_type is explicitly set alongside skip_consolidations=True, element_type is authoritative. The two kwargs are not AND-combined — this is a deliberate design choice documented in the docstring. (Setting element_type=['numeric','consolidated'] while leaving skip_consolidations=True would otherwise be incoherent: you asked for consolidations but the default flag drops them.)

3. Empty-match dataframe schema. When the trio filter matches zero elements, get_elements_dataframe constructs an empty MDX set via Tm1FilterByLevel({Members}, 9999) rather than the bare {}. The bare empty set loses the dimension column in the downstream pd.merge, breaking the DataFrame schema for callers that index by attribute or level column. The 9999-level filter produces an empty result while preserving the full column schema. A regression test (test_dataframe_trio_empty_match_preserves_schema) guards this.

Test plan

  • ~48 pure-function unit tests for the two new private helpers (_coerce_element_types, _build_elements_filter) in Tests/ElementService_filtering_helpers_test.py, covering translation, type coercion, glob-to-OData conversion, single-quote escaping, and every validation error path. No TM1 connection required.
  • ~37 integration tests in Tests/ElementService_test.py::TestElementFiltering against a fixture dimension covering each kwarg alone, in combination, with case/space variations, quote-escaped names (O'Brien), and validation passthrough.
  • Regression tests confirming get_elements_by_level and get_elements_filtered_by_wildcard produce identical output to master (snapshots committed under Tests/fixtures/element_filtering_snapshots/).
  • Regression test confirming get_elements_dataframe produces an identical DataFrame to master when no trio kwargs are passed.

Total ~85 new tests. Verified against a live TM1 11.8.x server: all green, no regressions in the existing 115 TestElementService tests.

Backwards compatibility

Purely additive. All new kwargs default to None. No signatures removed, no return types changed, no error semantics changed for any existing path.

Files touched

  • TM1py/Services/ElementService.py — two new private helpers, kwargs added to three public methods, two existing methods refactored to thin delegations.
  • Tests/ElementService_filtering_helpers_test.py — new file, unit tests for the helpers.
  • Tests/ElementService_test.py — new TestElementFiltering class.
  • Tests/fixtures/element_filtering_snapshots/ — regression oracle snapshots (9 files).

…element_types)

Adds two private module-level helpers in ElementService.py that translate
optional element_type / name_pattern / level filters into OData $filter
clauses. No public API change yet — subsequent commits wire these into
get_elements, get_element_names, and get_elements_dataframe.

Includes ~50 pure-function unit tests covering translation, type coercion,
glob-to-OData conversion (startswith/endswith/contains/multi-segment),
OData single-quote escaping, and all validation error paths.
…d get_element_names

Adds three optional, AND-composable filter kwargs to the two list-returning
element retrieval methods. Filters translate to OData $filter server-side
via _build_elements_filter (introduced in the previous commit).

element_type accepts Element.Types enum, str ('numeric'/'string'/
'consolidated', case-insensitive), int (1/2/3), or an iterable of those
(OR-combined). name_pattern is a glob with '*' wildcard, case- and
space-insensitive (matching the semantics of get_elements_filtered_by_wildcard).
level is an exact match on the hierarchy level integer.

All three default to None, purely additive, no breaking changes. Includes
integration tests against a fixture dimension covering each kwarg alone,
in combination, with case/space variations, quote-escape, and validation
error paths.
… trio kwargs to get_elements_dataframe

get_elements_by_level and get_elements_filtered_by_wildcard are now thin
delegations to get_element_names with the appropriate kwargs. Behavior is
preserved (verified by regression tests against snapshots captured from
master before the refactor). Net effect: single source of truth for OData
$filter construction across the four element-listing methods.

get_elements_dataframe gains element_type / name_pattern / level kwargs.
When any of the trio is set while elements is None, the method resolves the
selection via get_element_names and feeds it into the existing MDX path.
The trio is authoritative and overrides skip_consolidations (documented in
the docstring).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants