Expose resolved data version from DataOps load APIs

[damonbayer]

It would be helpful if DataOps exposed the resolved version used by load operations such as BlobEndpoint.get_dataframe().

Right now callers can pass a version_spec like <=2026-07-09T00-00-00, and DataOps resolves that internally via _get_version_blobs(...). However, the public return value is only the loaded dataframe. Downstream pipeline code that needs to record data freshness or provenance has to reimplement version resolution separately, or call lower-level/private APIs, to know which version was actually selected.

A backwards-compatible way to support this would be to add an optional flag that returns metadata alongside the existing result.

df, metadata = endpoint.load.get_dataframe(
    output="pl",
    version_spec="<=2026-07-09T00-00-00",
    include_metadata=True,
)

Then metadata might look like

{
    "version": "2026-07-08T00-00-00",
    "version_spec": "<=2026-07-09T00-00-00",
    "selection": "newest",
    ...
}

Alternatively, we could alter the return type for include_metadata=True. Then you could access the object with

result = endpoint.load.get_dataframe(..., incldue_metadata=True)

result.dataframe
result.version
result.version_spec
result.blob_names

[damonbayer]

Another idea: expose the version resolver as a separate function so users can query the versions without loading the data.

[ryanraaschCDC]

I like the first option out of the three, but I could implement whichever would be most helpful. The first one makes the most sense to me though. If that works for you I'll implement that.

[dylanhmorris]

Another idea: expose the version resolver as a separate function so users can query the versions without loading the data.

This is my preference. I'm not strongly opposed to having a convenient interface to annotate a retrieved dataframe with its version. That said:

1. it should be in addition to allowing users to query metadata without loading data and should use that query function itself under the hood.
2. I'd lean toward the metadata being stored in the data frame's own column(s) for tidyness (as opposed to in a separate dict). So if we have

metadata = endpoint.load.resolve_version(version_spec="<=2026-07-09T00-00-00")

then

endpoint.load.get_dataframe(
    output="pl",
    version_spec="<=2026-07-09T00-00-00",
    with_metadata=True
)

is equivalent to

metadata = endpoint.load.resolve_version(version_spec="<=2026-07-09T00-00-00")
endpoint.load.get_dataframe(
    output="pl",
    version_spec="<=2026-07-09T00-00-00",
    with_metadata=True
).with_columns(
   pl.struct(metadata).alias('metadata') # or unpack into individual columns
)

You'd need some interface for custom column names in case of a column name clash, but that's not too bad, and I personally prefer it to something named get_dataframe returning a tuple.

[ryanraaschCDC]

what is all the metadata you want included? Is it primarily the version used? Or do you want all the associated parameters when querying the dataframe?

[damonbayer]

2. I'd lean toward the metadata being stored in the data frame's own column(s) for tidyness (as opposed to in a separate dict). So if we have

Agree with your idea, except for this. The metadata applies to the dataframe itself, not each row. It's totally redundant to store it as a column.

[damonbayer]

what is all the metadata you want included? Is it primarily the version used? Or do you want all the associated parameters when querying the dataframe?

Maybe a blob storage url? If we truly want just the version, we could simplify the output.

[damonbayer]

While we do this, we should also make printing the version optional from get_dataframe.

[dylanhmorris]

Agree with your idea, except for this. The metadata applies to the dataframe itself, not each row. It's totally redundant to store it as a column.

I take your points but

• tidy data isn't necessarily normalized anyway!
• you might imagine wanting to row-bind two dfs containing two different vintages

If the metadata is not going to be truly attached to the df itself (and polars doesn't support this), then in some sense I prefer to make the user do

df_meta = endpoint.load.resolve_version(
    output="pl",
    version_spec="<=2026-07-09T00-00-00")
df = endpoint.load.get_dataframe(version_spec="<=2026-07-09T00-00-00")

but I'm also open to having a helper function like

df, df_meta = endpoint.load.get_dataframe_and_metadata(version_spec="<=2026-07-09T00-00-00")

[damonbayer]

If the metadata is not going to be truly attached to the df itself (and polars doesn't support this), then in some sense I prefer to make the user do

df_meta = endpoint.load.resolve_version(version_spec="<=2026-07-09T00-00-00")
df = endpoint.load.get_dataframe(version_spec="<=2026-07-09T00-00-00")

This is satisfactory for my needs.

[ryanraaschCDC]

I opened this PR to address these comments. I was able to find ways to attach metadata directly to the pandas or polars dataframes as attributes. Let me know if this is what you're thinking
#119