Skip to content

Let merge_datasets_as_delayed merge >2 datasets, filter by predicates and subset column#239

Closed
mlondschien wants to merge 4 commits into
JDASoftwareGroup:masterfrom
mlondschien:issue-235
Closed

Let merge_datasets_as_delayed merge >2 datasets, filter by predicates and subset column#239
mlondschien wants to merge 4 commits into
JDASoftwareGroup:masterfrom
mlondschien:issue-235

Conversation

@mlondschien

Copy link
Copy Markdown
Contributor

Description:

This Draft PR tries to implement changes as suggested in #235. The current philosophy is to

  • Implement an additional function (e.g. merge_many_datasets_as_delayed) not to break the existing API
  • rewrite align_datasets to efficiently cope with multiple datasets
  • Implement an additional merge_many_dataframes or similar to the MetaPartition class to allow for merging of dataframes.

The new merge_many_datasets_as_delayed can be supplied with predicates to subset rows and columns to subset columns. To account for the case where a column is defined in multiple datasets, columns needs to be a dictionary of type {uuid: table: [columns]}.

@mlondschien

Copy link
Copy Markdown
Contributor Author

Closing this as its superseded by #243 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant