Skip to content

Commit ebf7ba3

Browse files
committed
updated docs
1 parent 4f03700 commit ebf7ba3

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

docs/modules/hdfs/pages/usage-guide/operations/rack-awareness.adoc

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
= HDFS Rack Awareness
22
:rack-awareness-docs: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/RackAwareness.html
3-
:hdfs-topology-provider: https://github.com/stackabletech/hdfs-topology-provider
3+
:hdfs-topology-provider: https://github.com/stackabletech/hdfs-utils/blob/main/src/main/java/tech/stackable/hadoop/StackableTopologyProvider.java
44

55
{rack-awareness-docs}[Rack awareness] is a feature in Apache Hadoop that allows users to define a cluster's node topology.
66
Hadoop uses that topology to distribute block replicas in a way that maximizes fault tolerance.
@@ -35,3 +35,7 @@ This creates an internal topology label by combining the values of the `topology
3535
In order to enable gathering this information the Hadoop images contain the {hdfs-topology-provider}[hdfs-topology-provider] on the classpath, which can be configured to read labels from Kubernetes objects.
3636

3737
The operator deploys ClusterRoles and ServicesAccounts with the relevant RBAC rules to allow the Hadoop Pod to access the necessary Kubernetes objects.
38+
Topologies and other metadata such as Node- and Pod-IPs and endpoints are held in separate caches so that they can be refeshed independently of one another.
39+
The {hdfs-topology-provider}[hdfs-topology-provider] is namespace-scoped and pods in the active namespace are watched so that changes can be propagated to the internal cache to minimise cache misses.
40+
41+
NOTE: Rack awareness will not work on clusters such as `kind` or `k3s` that configure IP-masquerading differently to production-ready distributions.

0 commit comments

Comments
 (0)