feat!: Auto decouple and partition#23
Merged
Merged
Conversation
* Migrated changes from toy_examples branch * Added python analysis scripts * Create separate crate for hydro_optimize_examples, fix compile errors, cargo fmt + clippy * Struct must have all public fields
* Bug fix part 2 * Migrate toy_examples from main repo * Change dependencies to be local, merge with hydro main (add support for ChainFirst, remove delta, duplicate private functions), change simple_graphs to be time-based * Fixes for cases where no send/recv overhead is recorded, added noop experiment * Adopting changes HydroNode, HydroRoot, and HydroIrMetadata. WIP * Fixed compile-time errors * HydroRoot input() and input_metadata() no longer return vecs * Remove toy examples * Point to git repo instead of local * update Cargo.lock * cargo clippy * Fix tests, remove partition_node_analysis tests that rely on checking the taints of all nodes (since there's Cast, Batch, etc, too many to track now)
* Switch ILP solver, fix Tee inner no ID decoupling bug, add SingletonSource as a source * Insta fixes * Use HiGHS solver, much faster * cargo lock
* Fix bug where ILP solver would attempt to decouple at the network. New HydroNodes (such as YieldConcat) were not having their cardinalities calculated correctly. * cargo fmt * one last cargo fmt * Hide results.txt files
…later passes would check the hashmap of counters for that machine only and erase cardinalities for nodes on other machines. (#8) Give counter to SingletonSource (which may have a singleton per tick).
…d, assuming EDBs are never decoupled)
…ut" of a component is the actual input. Partitioning ILP WIP on field constraints
…lysis permits * deference
Fix: For operators with 2 parents, dependencies on 1 parent might reveal fields in the other. For example, AntiJoins pos side is completely dependent (no field info) but the 0th field is also completely dependent on the neg side. Therefore, the 0th field must also exist on the pos side. In the general case, fields on either side must be "projected" onto the other.
…o-project/hydro-optimize into auto_decouple_and_partition
…o-project/hydro-optimize into auto_decouple_and_partition
…o-project/hydro-optimize into auto_decouple_and_partition
…he wrong type after Map. 2. Tee had wrong location set after decoupling (the original tee needs to take its inner's location). 3. swap_root should use the parent's ID if a decoupling is required. 4. Partition divide by 0 bug
…s all analysis and optimizations together with single flag, untested
…sed usage table, incorporated into ILP solver (but times out for high budgets). CAS benchmark setup
…fields, decoupling checks is serializable (but not bounded, which is too strict), commutative reduce can be partitioned, op_id_to_parents filters op's own location as well, effective network cost add before partitioning
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.