Skip to content

Commit dc51aed

Browse files
peffgitster
authored andcommitted
t/lib-httpd: bump apache timeout
Since enabling more tests with 7a094d6 (ci: run expensive tests on push builds to integration branches, 2026-05-08), we sometimes see test failures or timeouts in GitHub CI. The culprit seems to be the "enormous ref negotiation" test in t5551, which creates ~100k tag refs in our http server-side repo. Iterating through the loose refs of this repo to generate a ref advertisement can take a long time, especially on a platform with slow I/O. On my otherwise unloaded local machine, a cold cache ref advertisement takes ~10s. On a busy CI machine running tests in parallel, it can presumably top 60s, which runs afoul of Apache's default CGI timeout. The result in t5551 is a test failure, where Apache simply hangs up the connection and the client reports an error. But worse, t5559 runs the same test with HTTP/2, and a bug in Apache causes the connection to hang indefinitely! We eventually see this as a CI timeout after 6 hours. Let's bump Apache's timeout to something much larger: 600 seconds. This doesn't eliminate the possibility of a timeout, but it makes it much less likely. It should eliminate both the test failures and the CI timeouts in practice, and it protects us from running into similar problems with other tests in the future. There are two counter-arguments to consider. One, could/should we just make the test faster? Probably yes. The biggest mistake here is having such an absurd number of unpacked refs on a system which is bottle-necked on I/O. But I think it's worth bumping the timeout so that we can fix this (and possibly other) correctness issues, and then consider performance separately (which we'll do in subsequent patches). And two, is this just papering over a problem that users might see in the real world? We could teach Git to handle this case more gracefully with optimizations or keep-alives. But I think it's really an artificial situation. You need a combination of this silly number of loose refs, plus a very heavily loaded system. If you were trying to run a real server and it took more than 60s to generate the ref advertisement, I don't think the timeout is your biggest problem. Your crappy service is, and you should adjust your resources to match your load. I.e., it is probably reasonable for Git to assume that advertisements happen fast-ish and don't need protocol-level keepalives. Though the patch here is small, tons of work went into analyzing the problem. Many thanks to the contributors credited below. Helped-by: Michael Montalbo <mmontalbo@gmail.com> Helped-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
1 parent a6876b2 commit dc51aed

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

t/lib-httpd/apache.conf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ DocumentRoot www
44
LogFormat "%h %l %u %t \"%r\" %>s %b" common
55
CustomLog access.log common
66
ErrorLog error.log
7+
Timeout 600
78
<IfModule !mod_log_config.c>
89
LoadModule log_config_module modules/mod_log_config.so
910
</IfModule>

0 commit comments

Comments
 (0)