Skip to content

Commit 85fab38

Browse files
roehrich-hperoehrich-hpe
authored
Merge release v0.1.14
Release v0.1.14
2 parents f5f55a8 + 82b779f commit 85fab38

6 files changed

Lines changed: 279 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,3 @@ INFO - Documentation built in 0.22 seconds
3030
INFO - [10:59:28] Watching paths for changes: 'docs', 'mkdocs.yml'
3131
INFO - [10:59:28] Serving on http://127.0.0.1:8000/
3232
```
33-

‎docs/guides/index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,8 @@
2727

2828
* [Disable or Drain a Node](node-management/drain.md)
2929
* [Debugging NVMe Namespaces](node-management/nvme-namespaces.md)
30+
31+
## Monitoring the Cluster
32+
33+
* [Auditing](monitoring-cluster/auditing.md)
34+
* [API Priority and Fairness](monitoring-cluster/api-priority-and-fairness.md)

‎docs/guides/monitoring-cluster/api-priority-and-fairness.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Kubernetes API Priority And Fairness
2+
3+
Kubernetes [API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/) (APF) allows requests to the Kubernetes API server to be classified, isolated, and queued in a fine-grained way.
4+
5+
The APF metrics can be monitored to determine how well the API servers are handling the workload. The metrics are intended to be interpreted by tools like [Prometheus](https://prometheus.io/) or [VictoriaMetrics](https://victoriametrics.com/). This document will use them in their raw form.
6+
7+
The metrics covered by this document are **counter** type. Counters are incremented, never decremented. While sampling counters in raw form, they will appear to bounce but on an idle system a given counter should make its current high value known after it appears in 3-5 samples.
8+
9+
## Concepts
10+
11+
Requests coming into the API server are classified by `FlowSchemas` and assigned to priority levels. The FlowSchema assigns the request to a **flow** and gives it a **flow distinguisher**. The flow distinguisher indicates the origin of the request--a user, service account, controller, namespace, or nothing. A priority level may take requests from multiple flows. The priority level attempts to give equal response time to each flow.
12+
13+
To view FlowSchemas and their assigned priority levels:
14+
15+
```console
16+
kubectl get flowschemas
17+
```
18+
19+
Flowschema sample output:
20+
21+
```bash
22+
NAME PRIORITYLEVEL MATCHINGPRECEDENCE DISTINGUISHERMETHOD AGE MISSINGPL
23+
[...]
24+
system-leader-election leader-election 100 ByUser 112d False
25+
endpoint-controller workload-high 150 ByUser 112d False
26+
workload-leader-election leader-election 200 ByUser 112d False
27+
system-node-high node-high 400 ByUser 112d False
28+
system-nodes system 500 ByUser 112d False
29+
[...]
30+
```
31+
32+
To view priority levels:
33+
34+
```console
35+
kubectl get prioritylevelconfiguration
36+
```
37+
38+
Priority level sample output:
39+
40+
```bash
41+
NAME TYPE NOMINALCONCURRENCYSHARES QUEUES HANDSIZE QUEUELENGTHLIMIT AGE
42+
[...]
43+
global-default Limited 20 128 6 50 112d
44+
leader-election Limited 10 16 4 50 112d
45+
node-high Limited 40 64 6 50 112d
46+
system Limited 30 64 6 50 112d
47+
workload-high Limited 40 128 6 50 112d
48+
workload-low Limited 100 128 6 50 112d
49+
[...]
50+
```
51+
52+
## Metric types
53+
54+
As noted earlier, the metrics will be viewed in their raw form and they are all of **counter** type. An individual counter must be sampled multiple times before its current high value can be clearly identified.
55+
56+
To view a counter's type:
57+
58+
```console
59+
kubectl get --raw /metrics | grep flowcontrol_rejected | grep '^#'
60+
```
61+
62+
The output will describe the counter and its type:
63+
64+
```bash
65+
# HELP apiserver_flowcontrol_rejected_requests_total [BETA] Number of requests rejected by API Priority and Fairness subsystem
66+
# TYPE apiserver_flowcontrol_rejected_requests_total counter
67+
```
68+
69+
## Examples
70+
71+
A quick way to get a summary of requests by priority level:
72+
73+
```console
74+
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
75+
```
76+
77+
From here one can drill down into the `Flowschemas` that feed a given priority level to see which one is generating the traffic.
78+
79+
View activity that uses the **nnf-clientmount** credentials:
80+
81+
```console
82+
kubectl get --raw /metrics | grep 'flow_schema=\"nnf-clientmount\"' | head -6
83+
```
84+
85+
View activity that uses the **viewer** user credential:
86+
87+
```console
88+
kubectl get --raw /metrics | grep 'flow_schema=\"nodediag-kubectls\"' | head -6
89+
```
90+
91+
## Resources
92+
93+
### Kubernetes
94+
95+
A description of APF:
96+
[API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/)
97+
98+
Debugging guide:
99+
[Flow Control](https://kubernetes.io/docs/reference/debug-cluster/flow-control/)
100+
101+
### Other sources
102+
103+
An excellent, though dated, description of tunables:
104+
[Kubernetes API and flow control: Managing request quantity and queuing procedure](https://blog.palark.com/kubernetes-api-flow-control-management/)
105+
106+
Slide deck that gets into the algorithms:
107+
[Kubernetes API Priority and Fairness](https://speakerdeck.com/ladicle/kubernetes-api-priority-and-fairness)

‎docs/guides/monitoring-cluster/auditing.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Kubernetes Auditing
2+
3+
Auditing provides records of each request that arrives in the kube-apiserver. The audit record will indicate what happened and who requested it.
4+
5+
## Enable Auditing
6+
7+
Enable auditing by installing an audit policy configuration file on each k8s master, creating a directory on the master to hold the audit logs, and providing the appropriate commandline options to kube-apiserver.
8+
9+
### Install an audit policy
10+
11+
The audit policy file will be installed on each k8s master node as `/etc/kubernetes/policies/audit-policy.yaml`.
12+
13+
The following is an example audit policy file that captures events for the NNF stack. Other examples can be found later in this document.
14+
15+
```bash
16+
apiVersion: audit.k8s.io/v1
17+
kind: Policy
18+
19+
omitStages:
20+
- RequestReceived
21+
22+
rules:
23+
- level: Metadata
24+
verbs: ["get", "list", "watch", "create", "patch", "update"]
25+
resources:
26+
27+
- group: lus.cray.hpe.com
28+
- group: dataworkflowservices.github.io
29+
- group: nnf.cray.hpe.com
30+
- group: dm.cray.hpe.com
31+
```
32+
33+
### Create a log directory
34+
35+
Create a directory on each k8s master to contain the audit logs.
36+
37+
```console
38+
mkdir /var/log/kubernetes
39+
```
40+
41+
### Configure the kube-apiserver
42+
43+
The following is an example patch to apply to the `/etc/kubernetes/manifests/kube-apiserver.yaml` file on each k8s master node. The arguments in this patch refer to the audit policy file location and audit log location used earlier in this document.
44+
45+
**Do not copy the `kube-apiserver.yaml` file to other master nodes. It contains IP addresses that are specific to one master node.**
46+
47+
After applying this patch to `kube-apiserver.yaml`, clear any extra patch or backup files out of `/etc/kubernetes/manifests` because kubelet will read all of them, regardless of the file suffix.
48+
49+
The kubelet on that master will detect the change to the `kube-apiserver.yaml` file and will restart the kube-apiserver.
50+
51+
```bash
52+
--- a/kube-apiserver.yaml-orig 2024-05-13 12:18:48.256680095 -0700
53+
+++ b/kube-apiserver.yaml 2024-05-28 13:39:50.342694448 -0700
54+
@@ -41,6 +41,9 @@
55+
- --service-cluster-ip-range=10.96.0.0/12
56+
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
57+
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
58+
+ - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
59+
+ - --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log
60+
+ - --audit-log-maxsize=100
61+
image: registry.k8s.io/kube-apiserver:v1.29.3
62+
imagePullPolicy: IfNotPresent
63+
livenessProbe:
64+
@@ -86,6 +89,12 @@
65+
- mountPath: /etc/kubernetes/pki
66+
name: k8s-certs
67+
readOnly: true
68+
+ - mountPath: /etc/kubernetes/policies/audit-policy.yaml
69+
+ name: k8s-policies
70+
+ readOnly: true
71+
+ - mountPath: /var/log/kubernetes/
72+
+ name: k8s-log
73+
+ readOnly: false
74+
hostNetwork: true
75+
priority: 2000001000
76+
priorityClassName: system-node-critical
77+
@@ -105,4 +114,12 @@
78+
path: /etc/kubernetes/pki
79+
type: DirectoryOrCreate
80+
name: k8s-certs
81+
+ - hostPath:
82+
+ path: /etc/kubernetes/policies/audit-policy.yaml
83+
+ type: File
84+
+ name: k8s-policies
85+
+ - hostPath:
86+
+ path: /var/log/kubernetes/
87+
+ type: DirectoryOrCreate
88+
+ name: k8s-log
89+
status: {}
90+
```
91+
92+
## Disable auditing
93+
94+
Disable auditing by editing the `/etc/kubernetes/manifests/kube-apiserver.yaml` on each master to remove the `--audit-*` commandline options from the kube-apiserver configuration. The kubelet on that master will detect the change to the `kube-apiserver.yaml` file and will restart the kube-apiserver.
95+
96+
Clear any extra patch or backup files out of `/etc/kubernetes/manifests` because kubelet will read all of them, regardless of the file suffix.
97+
98+
## Auditing in KIND
99+
100+
The KIND environment that is created by the tools in nnf-deploy already has auditing enabled. See the notes in nnf-deploy's [audit-policy.yaml](https://github.com/NearNodeFlash/nnf-deploy/blob/master/config/audit-policy.yaml) to access the audit log.
101+
102+
## Reading the audit log
103+
104+
The `jq(1)` command can be used to make sense of the audit logs. The following `jq` commands have proven useful to the NNF project:
105+
106+
Pretty-print the log events:
107+
108+
```console
109+
jq -M . kube-apiserver-audit.log | less
110+
```
111+
112+
Dump a quick-to-digest summary of the log events:
113+
114+
```console
115+
jq -M '[.auditID,.verb,.requestURI,.user.username,.responseStatus.code,.stageTimestamp]' kube-apiserver-audit.log | less
116+
```
117+
118+
Extract a specific event record from the log:
119+
120+
```console
121+
jq -M '. | select(.auditID=="d1053ee5-0734-4b40-815f-3f6831f82bac")' kube-apiserver-audit.log | less
122+
```
123+
124+
## Example audit policies
125+
126+
Log all activity from the clientmountd daemon. Extract records from the log with:
127+
128+
```console
129+
jq -M '.|select(.user.username=="system:serviceaccount:nnf-system:nnf-clientmount")' kube-apiserver-audit.log
130+
```
131+
132+
This could also be adjusted to isolate any other ServiceAccount.
133+
134+
```bash
135+
apiVersion: audit.k8s.io/v1
136+
kind: Policy
137+
138+
omitStages:
139+
- RequestReceived
140+
141+
rules:
142+
143+
- level: Metadata
144+
users: ["system:serviceaccount:nnf-system:nnf-clientmount"]
145+
resources:
146+
- group: "" # core
147+
- group: lus.cray.hpe.com
148+
- group: dataworkflowservices.github.io
149+
- group: nnf.cray.hpe.com
150+
- group: dm.cray.hpe.com
151+
```
152+
153+
A more complex [audit-policy.yaml](https://github.com/NearNodeFlash/nnf-deploy/blob/master/config/audit-policy.yaml) can be found in the nnf-deploy configuration for KIND environments.
154+
155+
## References
156+
157+
### Kubernetes
158+
159+
[Auditing](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)
160+
161+
### nnf-deploy
162+
163+
Nnf-deploy contains a more complex audit policy:
164+
[audit-policy.yaml](https://github.com/NearNodeFlash/nnf-deploy/blob/master/config/audit-policy.yaml)

‎external/nnf-dm‎

Submodule nnf-dm updated 129 files

‎mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ nav:
2121
- 'Lustre External MGT': 'guides/external-mgs/readme.md'
2222
- 'Global Lustre': 'guides/global-lustre/readme.md'
2323
- 'Disable or Drain a Node': 'guides/node-management/drain.md'
24+
- 'Auditing': 'guides/monitoring-cluster/auditing.md'
25+
- 'API Priority and Fairness': 'guides/monitoring-cluster/api-priority-and-fairness.md'
2426
- 'Debugging NVMe Namespaces': 'guides/node-management/nvme-namespaces.md'
2527
- 'Directive Breakdown': 'guides/directive-breakdown/readme.md'
2628
- 'System Storage': 'guides/system-storage/readme.md'

0 commit comments

Comments
 (0)