HDDS-14188. Include DN states that do not accept writes in NodesOutOfSpace metric

[sarvekshayr]

What changes were proposed in this pull request?

SCM metric NodesOutOfSpace only considers DN space availability. We should include DNs that are not in NodeState#HEALTHY and NodeOperationalState#IN_SERVICE states as non-writable too. Renamed the metric to NonWritableNodes.

What is the link to the Apache JIRA

HDDS-14188

How was this patch tested?

CI: https://github.com/sarvekshayr/ozone/actions/runs/20300694429
Before

    "InServiceHealthyNodes" : 5,
    "EnteringMaintenanceStaleNodes" : 0,
    "EnteringMaintenanceHealthyReadonlyNodes" : 0,
    "EnteringMaintenanceHealthyNodes" : 0,
    "EnteringMaintenanceDeadNodes" : 0,
    "AllNodes" : 5,
    "NonWritableNodes" : 0,

After putting a node in maintenance mode

    "InServiceHealthyNodes" : 4,
    "EnteringMaintenanceStaleNodes" : 0,
    "EnteringMaintenanceHealthyReadonlyNodes" : 0,
    "EnteringMaintenanceHealthyNodes" : 1,
    "EnteringMaintenanceDeadNodes" : 0,
    "AllNodes" : 5,
    "NonWritableNodes" : 1,

[rich7420]

thanks for the patch!

[adoroszlai]

Thanks @sarvekshayr for the patch.

[adoroszlai]

nit: I think NodesOutOfSpace may need to be renamed to reflect the change. Or maybe have a separate metric for non-writable nodes?

[sarvekshayr]

nit: I think NodesOutOfSpace may need to be renamed to reflect the change. Or maybe have a separate metric for non-writable nodes?

We need ONE metric that considers both space and state of the nodes. I was thinking about renaming it to ReadableNodes or NonWritableNodes. What do you suggest?

[adoroszlai]

NonWritableNodes seems OK.

[adoroszlai]

Thanks @sarvekshayr for the patch, @rich7420 for the review.