FEAT: reconnect and reconstrcut model replicas after restart supervisor

[leslie2046]

close #2402

Problem solved:

Resolving the issue that workers must restart and the model must be redeployed after the supervisor restarts.

Technical solution:

After the supervisor restarts, the worker automatically reconnects to the supervisor and uploads its own status and model replica status information.

[leslie2046]

@qinxuye

[nasircsms]

@leslie2046 can you please have a look at #2402 (comment)

[qinxuye]

@leslie2046 can you please have a look at #2402 (comment)

Maybe it's a different issue, the solution might be if some replica failed, it should be blacklisted.

[nasircsms]

@leslie2046 can you please have a look at #2402 (comment)

Maybe it's a different issue, the solution might be if some replica failed, it should be blacklisted.

Yes.. I have tested it with v2.4.0 as well. As soon as a worker becomes unavailable then whole cluster fails. Even if that worker joins again on the same port but still cluster remains unavailable. Need to restart supervisor and all the workers to make the cluster available again.

[qinxuye]

@leslie2046 can you please have a look at #2402 (comment)

Maybe it's a different issue, the solution might be if some replica failed, it should be blacklisted.

Yes.. I have tested it with v2.4.0 as well. As soon as a worker becomes unavailable then whole cluster fails. Even if that worker joins again on the same port but still cluster remains unavailable. Need to restart supervisor and all the workers to make the cluster available again.

I guess the worker is not dead right? Maybe just because the heartbeat is not sent just in time.

[qinxuye]

LGTM

[nasircsms]

@leslie2046 can you please have a look at #2402 (comment)

Maybe it's a different issue, the solution might be if some replica failed, it should be blacklisted.

Yes.. I have tested it with v2.4.0 as well. As soon as a worker becomes unavailable then whole cluster fails. Even if that worker joins again on the same port but still cluster remains unavailable. Need to restart supervisor and all the workers to make the cluster available again.

I guess the worker is not dead right? Maybe just because the heartbeat is not sent just in time.

In my case worker machine is powered-off or restarted. Although I am running xinference as service so on boot it auto-connect to supervisor again on same port, but even if worker comes back or not the cluster is done for.

[qinxuye]

@leslie2046 can you please have a look at #2402 (comment)

Maybe it's a different issue, the solution might be if some replica failed, it should be blacklisted.

Yes.. I have tested it with v2.4.0 as well. As soon as a worker becomes unavailable then whole cluster fails. Even if that worker joins again on the same port but still cluster remains unavailable. Need to restart supervisor and all the workers to make the cluster available again.

I guess the worker is not dead right? Maybe just because the heartbeat is not sent just in time.

In my case worker machine is powered-off or restarted. Although I am running xinference as service so on boot it auto-connect to supervisor again on same port, but even if worker comes back or not the cluster is done for.

You can try this PR, it might address this issue.