Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1432988	dotnet/roslyn	Checkout dotnet/roslyn@main to s	Log
1432954	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128521/merge to s	Log	dotnet/runtime#128521
1432885	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128522/merge to s	Log	dotnet/runtime#128522
1432820	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127989/merge to s	Log	dotnet/runtime#127989
1432826	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126947/merge to s	Log	dotnet/runtime#126947
1432790	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1432774	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1432768	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1432757	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/83397/merge to s	Log	dotnet/roslyn#83397
1432751	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/83761/merge to s	Log	dotnet/roslyn#83761
1432716	dotnet/performance	Checkout dotnet/performance@refs/pull/5220/merge to s/performance	Log	dotnet/performance#5220
1432687	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128519/merge to s	Log	dotnet/runtime#128519
2981947	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61470/merge to s	Log	#61470
2981946	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61469/merge to s	Log	#61469
2981945	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61469/merge to s	Log	#61469
2981923	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61466/merge to s	Log	#61466
2981897	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61465/merge to s	Log	#61465
1431680	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128487/merge to s	Log	dotnet/runtime#128487
1431217	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53953/merge to s	Log	dotnet/sdk#53953
1431124	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54409/merge to s	Log	dotnet/sdk#54409
1430745	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66793/merge to s	Log	dotnet/aspnetcore#66793
2981311	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61439/merge to s	Log	#61439
1429746	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128442/merge to s	Log	dotnet/runtime#128442
1428924	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128347/merge to s	Log	dotnet/runtime#128347
1428393	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128409/merge to s	Log	dotnet/runtime#128409
2980017	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61351/merge to s	Log	#61351
1427854	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6318/merge to s	Log	dotnet/arcade-services#6318
1427691	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128394/merge to s	Log	dotnet/runtime#128394
2979270	dotnet-maui	Checkout dotnet-maui@inflight/current to s	Log
2979009	dotnet-installer	Clone dotnet/installer	Log	#61281
2978769	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr7 to s	Log
1426660	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124173/merge to s	Log	dotnet/runtime#124173
1426463	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123252/merge to s	Log	dotnet/runtime#123252
2978282	dotnet-sdk	Checkout dotnet-sdk@refs/pull/61258/merge to s	Log	#61258
1425412	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128337/merge to s	Log	dotnet/runtime#128337
1424819	dotnet/maui	Checkout dotnet/maui@refs/pull/34758/merge to s	Log	dotnet/maui#34758
1424687	dotnet/maui	Checkout dotnet/maui@refs/pull/35362/merge to s	Log	dotnet/maui#35362
1423818	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66712/merge to s	Log	dotnet/aspnetcore#66712
1423733	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128298/merge to s	Log	dotnet/runtime#128298
1422707	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128264/merge to s	Log	dotnet/runtime#128264
1422251	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54318/merge to s	Log	dotnet/sdk#54318
1417748	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54266/merge to s	Log	dotnet/sdk#54266
2975817	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61122/merge to s	Log	#61122
1421013	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128213/merge to s	Log	dotnet/runtime#128213
1421076	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
2975280	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61099/merge to s	Log	#61099
1421037	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128154/merge to s	Log	dotnet/runtime#128154
1420498	dotnet/maui	Checkout dotnet/maui@refs/pull/32874/merge to s	Log	dotnet/maui#32874
1420351	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128186/merge to s	Log	dotnet/runtime#128186
1420270	dotnet/runtime	Checkout dotnet/runtime@release/9.0-staging to s	Log
1419317	dotnet/maui	Checkout dotnet/maui@refs/pull/35362/merge to s	Log	dotnet/maui#35362
1419034	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1417428	dotnet/maui	Checkout dotnet/maui@refs/pull/35362/merge to s	Log	dotnet/maui#35362
1417483	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6289/merge to s	Log	dotnet/arcade-services#6289
2973006	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
1417411	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16804/merge to s	Log	#16804
1417288	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16796/merge to s	Log	#16796
1417258	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16793/merge to s	Log	#16793
1417239	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16792/merge to s	Log	#16792
1416576	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128056/merge to s	Log	dotnet/runtime#128056
2972521	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60874/merge to s	Log	#60874
1416509	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128055/merge to s	Log	dotnet/runtime#128055
2972469	dotnet-sdk	Checkout dotnet-sdk@refs/pull/60944/merge to s	Log	#60944
1416300	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128049/merge to s	Log	dotnet/runtime#128049
1415831	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6275/merge to s	Log	dotnet/arcade-services#6275
1415050	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128004/merge to s	Log	dotnet/runtime#128004
1412613	dotnet/performance	Checkout dotnet/performance@refs/pull/5216/merge to s	Log	dotnet/performance#5216
1412228	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1412130	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127912/merge to s	Log	dotnet/runtime#127912
1411994	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127450/merge to s	Log	dotnet/runtime#127450
2969365	dotnet-maui	Checkout dotnet-maui@main to s	Log
1411902	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66613/merge to s	Log	dotnet/aspnetcore#66613
1411011	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125895/merge to s	Log	dotnet/runtime#125895
1409828	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127819/merge to s	Log	dotnet/runtime#127819
1409469	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127826/merge to s	Log	dotnet/runtime#127826
2967543	dotnet-maui	Checkout dotnet-maui@net11.0 to s	Log
1408595	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53946/merge to s	Log	dotnet/sdk#53946
1408639	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53954/merge to s	Log	dotnet/sdk#53954
1408359	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6257/merge to s	Log	dotnet/arcade-services#6257
1408354	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6253/merge to s	Log	dotnet/arcade-services#6253
1407663	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127754/merge to s	Log	dotnet/runtime#127754
1407647	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127760/merge to s	Log	dotnet/runtime#127760
1407588	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127697/merge to s	Log	dotnet/runtime#127697
1407679	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127763/merge to s	Log	dotnet/runtime#127763
1407587	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127697/merge to s	Log	dotnet/runtime#127697
1407481	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127748/merge to s	Log	dotnet/runtime#127748
1407609	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127755/merge to s	Log	dotnet/runtime#127755
1407715	dotnet/maui	Checkout dotnet/maui@refs/pull/34621/merge to s	Log	dotnet/maui#34621
1407637	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127744/merge to s	Log	dotnet/runtime#127744
1407405	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127016/merge to s	Log	dotnet/runtime#127016
1407554	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127743/merge to s	Log	dotnet/runtime#127743
1407595	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127749/merge to s	Log	dotnet/runtime#127749
1407644	dotnet/sdk	Checkout dotnet/sdk@dev/nikolam/aot.fix to s	Log
1407563	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127112/merge to s	Log	dotnet/runtime#127112
1407556	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127743/merge to s	Log	dotnet/runtime#127743
1407372	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6251/merge to s	Log	dotnet/arcade-services#6251
1406665	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117261/merge to s	Log	dotnet/runtime#117261
2965456	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60696/merge to s	Log	#60696
2965400	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60692/merge to s	Log	#60692
1405511	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127492/merge to s	Log	dotnet/runtime#127492
Displaying 100 of 123 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	37	123

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?