HDDS-13945. Show datanode reserved space in StorageDistributionEndpoint

[priyeshkaratha]

What changes were proposed in this pull request?

This change displays datanode reserved space in the response of StorageDistributionEndpoint in Recon.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13945

How was this patch tested?

curl -X GET http://localhost:9888/api/v1/storageDistribution | jq .

{
  "globalStorage": {
    "totalUsedSpace": 13148160,
    "totalFreeSpace": 3054489243648,
    "totalCapacity": 3242976054744
  },
  "globalNamespace": {
    "totalUsedSpace": 0,
    "totalKeys": 0
  },
  "usedSpaceBreakdown": {
    "openKeyBytes": 0,
    "committedKeyBytes": 0,
    "preAllocatedContainerBytes": 0
  },
  "dataNodeUsage": [
    {
      "datanodeUuid": "ea347fcc-78ba-4066-a6e5-95ba802389c9",
      "hostName": "ozone-datanode-1.ozone_default",
      "capacity": 1080992018248,
      "used": 4382720,
      "remaining": 1018163081216,
      "committed": 0,
      "minimumFreeSpace": 104857600,
      "reserved": 108110008
    },
    {
      "datanodeUuid": "8368ebbe-efa1-48d0-af67-7036dda471e1",
      "hostName": "ozone-datanode-3.ozone_default",
      "capacity": 1080992018248,
      "used": 4382720,
      "remaining": 1018163081216,
      "committed": 0,
      "minimumFreeSpace": 104857600,
      "reserved": 108110008
    },
    {
      "datanodeUuid": "699f66dd-e8d0-49f5-b371-6cfff3f63fba",
      "hostName": "ozone-datanode-2.ozone_default",
      "capacity": 1080992018248,
      "used": 4382720,
      "remaining": 1018163081216,
      "committed": 0,
      "minimumFreeSpace": 104857600,
      "reserved": 108110008
    }
  ]
}

[devmadhuu]

Thanks @priyeshkaratha for the patch. Changes LGTM , but just a minor comment and also I think , better to write both unit and integration tests.

[devmadhuu]

Though very rare, but this getTotalReserved can throw NodeNotFoundException, so in that case, what will be the behavior ?

[ChenSammi]

Can we retrieve the reservedSpace from SCMNodeMetric too? Retrieving all data from same source is better from both data consistent and performance point.

[priyeshkaratha]

Currently, SCMNodeMetric does not have this information. Since the command-line tool already relies on NodeManager, I thought it would be better to reuse the same approach.

Adding this to SCMNodeMetric would require redefining multiple classes, and since SCM does not use this value for any other purpose, do we really need to introduce it into SCMNodeMetric?

[ChenSammi]

StorageReportProto has the reserved space value, so SCM has this information(#9338). Currently SCM just doesn't populate the info to SCMNodeStat and SCMNodeMetric.

ozone admin datanode usageinfo command gets all info from the scmClient.getDatanodeUsageInfo(). You can check if Recon has use the same way. Otherwise populate the reserved space value in SCMNodeStat and SCMNodeMetric is preferred. It's not an issue if multiple classes are required to update. The only thing matters is whether it's the right/better way.

[priyeshkaratha]

@ChenSammi @devmadhuu Addressed the changes and added integration test to validate the values.

[devmadhuu]

Thanks @priyeshkaratha for the patch. Just a nit:. pls handle.

[adoroszlai]

Thanks @priyeshkaratha for the patch, @ChenSammi, @devmadhuu for the review.