Skip to content

Commit 944459a

Browse files
eprice555CopilotCopilotjc-clark
authored
Add article: Analyzing Git traffic on your GHES instance (#62231)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Joe Clark <31087804+jc-clark@users.noreply.github.com>
1 parent fe1cb4f commit 944459a

3 files changed

Lines changed: 288 additions & 2 deletions

File tree

‎content/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -974,7 +974,7 @@ ghe-btop [ <port number> | --help | --usage ]
974974
975975
#### ghe-governor
976976
977-
This utility helps to analyze Git traffic. It queries _Governor_ data files, located under `/data/user/gitmon`. {% data variables.product.company_short %} holds one hour of data per file, retained for two weeks. For more information, see [Analyzing Git traffic using Governor](https://github.com/orgs/community/discussions/34220) in {% data variables.product.prodname_github_community %}.
977+
This utility helps to analyze Git traffic. It queries _Governor_ data files, located under `/data/user/governor/`. {% data variables.product.company_short %} holds one hour of data per file, retained for two weeks. For more information, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/monitoring-your-instance/analyze-git-traffic).
978978
979979
```bash
980980
ghe-governor <subcommand> <column> [options]

‎content/admin/monitoring-and-managing-your-instance/monitoring-your-instance/analyze-git-traffic.md‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
---
2+
title: Analyzing Git traffic on your {% data variables.product.prodname_ghe_server %} instance
3+
shortTitle: Analyze Git traffic
4+
intro: 'Use Governor to identify Git traffic patterns that are driving load on your instance, so you can troubleshoot slow Git operations and reduce performance impact.'
5+
versions:
6+
ghes: '*'
7+
contentType: how-tos
8+
category:
9+
- Monitor and audit your enterprise
10+
---
11+
12+
## About Governor
13+
14+
Governor is a built-in monitor for Git activity on your instance. Use the `ghe-governor` command to see which repositories, users, IP addresses, and Git operations are creating load.
15+
16+
Use this data when CPU, memory, or disk usage increases and you need to confirm whether Git traffic is the cause. For command syntax and subcommands, see [AUTOTITLE](/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities#ghe-governor).
17+
18+
Governor records Git operations only. It does not include API or web traffic. To inspect operations currently running, see the `ghe-btop` utility in [AUTOTITLE](/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities#ghe-btop).
19+
20+
### About Governor data files
21+
22+
Governor stores its data in files under `/data/user/governor/`. Each file holds one hour of data and is retained for two weeks. The file names contain Unix timestamps that indicate the time period each file covers.
23+
24+
> [!NOTE]
25+
> On {% data variables.product.prodname_ghe_server %} 3.13 and earlier, Governor data files are located under `/data/user/gitmon/` and use the naming pattern `gitmon.<timestamp>.db`.
26+
27+
To confirm the range of data currently held on disk, convert the timestamps in the earliest and latest file names to human-readable dates.
28+
29+
```shell
30+
for epoch in $(sudo ls /data/user/governor/ 2>/dev/null | grep '^governor\.' | sort | sed -n '1p;$p' | cut -f2 -d.); do echo "${epoch} = $(date -d @${epoch})"; done
31+
```
32+
33+
The output shows the start time of the earliest and latest data files.
34+
35+
```text
36+
1551186000 = Tue Feb 26 13:00:00 UTC 2019
37+
1552392000 = Tue Mar 12 12:00:00 UTC 2019
38+
```
39+
40+
## Before you start
41+
42+
Before you run Governor queries, gather this information:
43+
44+
* **SSH access to the appliance.** You need access to the administrative shell to run `ghe-governor`. See [AUTOTITLE](/admin/administering-your-instance/administering-your-instance-from-the-command-line/accessing-the-administrative-shell-ssh).
45+
* **Which node to query.** For standalone or high-availability deployments, start on the primary node. For clusters, run queries on each `git-server` node because no single node contains all Git traffic.
46+
* **The impact window.** Note the start and end times of reported degradation so you can scope queries with `-t` and `-u`.
47+
* **Repository or organization names**, if users have reported problems with specific repositories or organizations.
48+
49+
## Identify whether Git is contributing to load
50+
51+
Start with a broad summary, then attribute load to specific repositories or programs.
52+
53+
* Summarize all recent Git activity on your instance:
54+
55+
```shell
56+
ghe-governor health
57+
```
58+
59+
* Find repositories with the slowest average response time:
60+
61+
```shell
62+
ghe-governor aggregate repo avg_rt
63+
```
64+
65+
* Find repositories with the longest single operation:
66+
67+
```shell
68+
ghe-governor aggregate repo max_rt
69+
```
70+
71+
* Find the slowest individual operations (not grouped):
72+
73+
```shell
74+
ghe-governor top rt -n 50
75+
```
76+
77+
* Find the repositories consuming the most CPU time:
78+
79+
```shell
80+
ghe-governor aggregate repo cpu
81+
```
82+
83+
* For a repository that stands out, find which Git subprogram is responsible for its CPU usage:
84+
85+
```shell
86+
ghe-governor aggregate program cpu -r OWNER/REPOSITORY
87+
```
88+
89+
* Find the repositories where `pack-objects` consumed the most CPU time. The `pack-objects` program assembles data sent to clients during clones and fetches; a high count places significant CPU and memory pressure on the appliance:
90+
91+
```shell
92+
ghe-governor aggregate repo cpu -P pack-objects
93+
```
94+
95+
* Find the individual operations that used the most CPU time (not grouped):
96+
97+
```shell
98+
ghe-governor top cpu -n 50
99+
```
100+
101+
* Find the repositories driving the most disk writes during a specific time interval:
102+
103+
```shell
104+
ghe-governor aggregate repo disk_write_kb -t START-TIME -u END-TIME
105+
```
106+
107+
* Check for high concurrency, which indicates many Git operations running simultaneously:
108+
109+
```shell
110+
ghe-governor aggregate repo max_parallelism
111+
```
112+
113+
If the results point to a specific repository or program, continue with the relevant section below.
114+
115+
## Identify top clone and fetch traffic
116+
117+
The `upload-pack` program handles data served to clients during clones and fetches. Use these queries to find which repositories, users, and IP addresses are driving the most clone and fetch activity.
118+
119+
* Count clone and fetch operations by repository:
120+
121+
```shell
122+
ghe-governor aggregate repo count -P upload-pack
123+
```
124+
125+
* Identify users running clones and fetches and count their operations:
126+
127+
```shell
128+
ghe-governor aggregate user_id count -P upload-pack
129+
```
130+
131+
* Identify the IP addresses generating the most clone and fetch requests. A small set of IP addresses with a high count often indicates a CI runner fleet:
132+
133+
```shell
134+
ghe-governor aggregate ip count -P upload-pack
135+
```
136+
137+
* Measure the total volume of data served per repository:
138+
139+
```shell
140+
ghe-governor aggregate repo uploaded_kb -P upload-pack
141+
```
142+
143+
* Measure the average volume of data uploaded per user:
144+
145+
```shell
146+
ghe-governor aggregate user_id avg_uploaded -P upload-pack
147+
```
148+
149+
* Find peak clone and fetch concurrency per repository. The `MAXPL` column shows the highest number of simultaneous operations recorded for each repository. A high value for a small number of repositories suggests a thundering herd:
150+
151+
```shell
152+
ghe-governor aggregate repo max_parallelism -P upload-pack
153+
```
154+
155+
* Find the largest individual clone or fetch operations (not grouped):
156+
157+
```shell
158+
ghe-governor top uploaded -P upload-pack -n 50
159+
```
160+
161+
## Identify push-heavy traffic
162+
163+
The `receive-pack` and `spokes-receive-pack` programs handle data received from clients during pushes. Use these queries to find which repositories, users, and IP addresses are generating the most push activity.
164+
165+
* Count push operations by repository:
166+
167+
```shell
168+
ghe-governor aggregate repo count -P receive-pack -P spokes-receive-pack
169+
```
170+
171+
* Identify users pushing to an organization and count their push operations:
172+
173+
```shell
174+
ghe-governor aggregate user_id count -o ORGANIZATION -P receive-pack -P spokes-receive-pack
175+
```
176+
177+
* Measure the total volume of data received per repository:
178+
179+
```shell
180+
ghe-governor aggregate repo received_kb -P receive-pack -P spokes-receive-pack
181+
```
182+
183+
* Identify IP addresses sending the most push data:
184+
185+
```shell
186+
ghe-governor aggregate ip received_kb -P receive-pack -P spokes-receive-pack
187+
```
188+
189+
* Find the largest individual push operations (not grouped):
190+
191+
```shell
192+
ghe-governor top received -P receive-pack -P spokes-receive-pack -n 50
193+
```
194+
195+
## Narrow the analysis
196+
197+
You can add any combination of the following options to a `ghe-governor top` or `ghe-governor aggregate` command to focus the query on a specific time window, repository, owner, or program.
198+
199+
### Time filters
200+
201+
| Option | Description |
202+
| --- | --- |
203+
| `-t <timespec>` | Consider only operations since a given start time. The default is 48 hours ago. |
204+
| `-u <timespec>` | Consider only operations up to a given end time. The default is the current time. |
205+
206+
The following formats are accepted for `<timespec>`.
207+
208+
| Format | Example | Meaning |
209+
| --- | --- | --- |
210+
| Unix timestamp | `-t 1371614483` | Seconds since January 1, 1970 |
211+
| Java timestamp | `-t 1371614483637` | Milliseconds since January 1, 1970 |
212+
| Relative days | `-t 1d` | The last day |
213+
| Relative hours | `-t 2h` | The last two hours |
214+
| Relative minutes | `-t 20m` | The last twenty minutes |
215+
216+
### Scope filters
217+
218+
| Option | Description |
219+
| --- | --- |
220+
| `-r <owner>/<repository>` | Consider only operations matching a given owner and repository. Specify this option multiple times to match several repositories. |
221+
| `-o <owner>` | Consider only operations matching a given owner, such as a user or organization. Specify this option multiple times to match several owners. |
222+
| `-P <program>` | Consider only operations that ran a given Git subprogram, such as `upload-pack`, `receive-pack`, `rev-list`, or `pack-objects`. Specify this option multiple times to match several programs. |
223+
| `-I <address>` | Consider only operations from a specific IP address. Specify this option multiple times to match several addresses. |
224+
225+
### Output options
226+
227+
| Option | Description |
228+
| --- | --- |
229+
| `-j` | Set the output format to JSON instead of an ASCII table. |
230+
| `-n <N>` | Limit the output to N records. The default is 20 for aggregate queries and 200 for top queries. |
231+
| `--count-only` | Show only the `KEY` and `COUNT` columns. Applies to aggregate queries only. |
232+
233+
### Interpreting result columns
234+
235+
The following abbreviations appear in `ghe-governor` result tables.
236+
237+
| Column | Meaning |
238+
| --- | --- |
239+
| `AVG RT` | Average time, in seconds, that Git invocations took |
240+
| `MAX RT` | Running time, in seconds, of the longest-running invocation, per host |
241+
| `MAXPL` / `AVGPL` | Maximum and average parallelism: how many Git invocations were outstanding at one time |
242+
| `CPU/SEC` | Seconds of CPU time used by Git per second of wall-clock time. Divide by the number of CPU cores and multiply by 100 to get the Git-specific CPU percentage. This value cannot exceed the number of CPU cores. |
243+
| `UPL` | Data the server uploaded to clients, such as during fetches and clones |
244+
| `RECV` | Data the server received from clients, such as during pushes |
245+
246+
The `READ`, `WRITE`, `UPL`, and `RECV` columns are reported in gigabytes (GB), and the corresponding rate is reported in megabytes per second (MB/s).
247+
248+
## Interpret common patterns
249+
250+
The following table maps common Governor output patterns to their likely causes.
251+
252+
| Pattern | Likely cause |
253+
| --- | --- |
254+
| High `upload-pack` count from a small set of IP addresses | A CI runner fleet is repeatedly cloning one or more repositories instead of reusing local checkouts |
255+
| High `max_parallelism` for a small number of repositories | Thundering herd: many runners triggering concurrent clones at the same time, often because scheduled jobs start simultaneously |
256+
| Large `uploaded_kb` per repository combined with high `avg_uploaded` per user | Repeated full clones; clients are not reusing local checkouts or are not using shallow or partial clones |
257+
| High `received_kb` for a repository or organization | Push-intensive workload; may involve large binary files or frequent commits to a monorepo |
258+
| High `pack-objects` CPU with moderate operation counts | Expensive object packing; the repository may benefit from a maintenance run or from client-side use of partial clones with `--filter` |
259+
| High `avg_rt` or `max_rt` for a repository | Slow operations; often caused by large `pack-objects` runs, high concurrency, or resource contention from another workload |
260+
261+
## Recommended actions for Git load reduction
262+
263+
If Governor data shows that a repository, user, or runner fleet is generating excessive load, the following actions can reduce the impact.
264+
265+
* **Reuse local checkouts.** Replace fresh clones with `git fetch` in automated workflows. This avoids transferring the full object graph on each run.
266+
* **Use shallow clones.** For workflows that do not require full commit history, pass `--depth` to the clone command, for example `git clone --depth 1`. This reduces both the data transferred and the work `pack-objects` must do.
267+
* **Use partial clones.** For repositories with large binary objects or many files not needed in a given workflow, use `--filter` to request only the objects the workflow requires, for example `git clone --filter=blob:none`.
268+
* **Add a local mirror or caching proxy.** For large runner fleets that clone the same repositories repeatedly, a pull-through proxy or local mirror can absorb fetch traffic and reduce load on your instance.
269+
* **Stagger scheduled jobs.** Offset CI/CD workflows triggered on a schedule so that not all runners start at the same time, reducing the size of concurrent clone bursts.
270+
* **Reduce workflow parallelism.** Workflows that clone multiple repositories in parallel can be reconfigured to limit concurrency and spread the load over time.
271+
272+
## When to contact {% data variables.product.company_short %} Support
273+
274+
Contact {% data variables.contact.contact_ent_support %} and include a support bundle if any of the following apply.
275+
276+
* Performance remains degraded after reducing clone pressure or staggering scheduled jobs.
277+
* Your instance is serving widespread Git errors or connection failures that are not explained by client workload.
278+
* Governor shows persistent `max_parallelism` values suggesting the instance is at or near capacity, and workload changes alone are unlikely to resolve the issue.
279+
* You see a clear pattern in Governor output but are uncertain how to interpret it or what action to take.
280+
281+
For more information about generating a support bundle, see [AUTOTITLE](/support/contacting-github-support/providing-data-to-github-support#creating-and-sharing-support-bundles).
282+
283+
## Further reading
284+
285+
For more information about monitoring system resources, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/monitoring-your-instance/about-the-monitor-dashboards).

‎content/admin/monitoring-and-managing-your-instance/monitoring-your-instance/index.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ versions:
1414
children:
1515
- /about-monitoring-your-instance
1616
- /about-the-monitor-dashboards
17+
- /analyze-git-traffic
1718
- /recommended-alert-thresholds
1819
- /opentelemetry-metrics
1920
- /collectd-metrics
@@ -24,4 +25,4 @@ children:
2425
- /generating-a-health-check-for-your-enterprise
2526

2627
shortTitle: Monitor your instance
27-
---
28+
---

0 commit comments

Comments
 (0)