Description
Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed.
Use case/motivation
Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved.
Describe the solution
Extended support for a new master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster.
Subtasks
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed.
Use case/motivation
Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved.
Describe the solution
Extended support for a new master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster.
Subtasks
Related issues
No response
Are you willing to submit a PR?
Code of Conduct