It would be valuable to have a more balanced sharding to make workers to run about same amount of time.
One solution:
Write a submitter script
- list all files
- Remove files already done.
- group them into n-shards looking on size
- Create an indata file that is consumed by packager.
It would be valuable to have a more balanced sharding to make workers to run about same amount of time.
One solution:
Write a submitter script