windows garm runner fails to install gitea-runner on Invoke-FastWebRequest

[hulto]

Issue

When installing the gitea runner from gitea.com the Invoke-FastWebRequest function errors with 403 (Forbidden).

Reference

Invoke-FastWebRequest fails with 403.
Invoke-WebRequest works
Invoke-FastWebRequest to non-gitea domain works.

Digging into the function seems to be around the $client.GetStreamAsync function.

Tested hosting the runner (v1.0.7 diff than past testing) with python3 -m http.server and file was grabbed successfully with Invoke-FastWebRequest

Workaround

I created a custom template and replaced calls to Invoke-FastWebRequest with Invoke-WebRequest

Template .ps1

function Invoke-FastWebRequest {
	[CmdletBinding()]
	Param(
		[Parameter(Mandatory=$True,ValueFromPipeline=$true,Position=0)]
		[System.Uri]$Uri,
		[Parameter(Position=1)]
		[string]$OutFile,
		[Hashtable]$Headers=@{},
		[switch]$SkipIntegrityCheck=$false
	)
	PROCESS
	{
		if(!([System.Management.Automation.PSTypeName]'System.Net.Http.HttpClient').Type)
		{
			$assembly = [System.Reflection.Assembly]::LoadWithPartialName("System.Net.Http")
		}

		if(!$OutFile) {
			$OutFile = $Uri.PathAndQuery.Substring($Uri.PathAndQuery.LastIndexOf("/") + 1)
			if(!$OutFile) {
				throw "The ""OutFile"" parameter needs to be specified"
			}
		}

		$fragment = $Uri.Fragment.Trim('#')
		if ($fragment) {
			$details = $fragment.Split("=")
			$algorithm = $details[0]
			$hash = $details[1]
		}

		if (!$SkipIntegrityCheck -and $fragment -and (Test-Path $OutFile)) {
			try {
				return (Test-FileIntegrity -File $OutFile -Algorithm $algorithm -ExpectedHash $hash)
			} catch {
				Remove-Item $OutFile
			}
		}

		$client = new-object System.Net.Http.HttpClient
		foreach ($k in $Headers.Keys){
			$client.DefaultRequestHeaders.Add($k, $Headers[$k])
		}
		$task = $client.GetStreamAsync($Uri)
		$response = $task.Result
		if($task.IsFaulted) {
			$msg = "Request for URL '{0}' is faulted. Task status: {1}." -f @($Uri, $task.Status)
			if($task.Exception) {
				$msg += "Exception details: {0}" -f @($task.Exception)
			}
			Throw $msg
		}
		$outStream = New-Object IO.FileStream $OutFile, Create, Write, None

		try {
			$totRead = 0
			$buffer = New-Object Byte[] 1MB
			while (($read = $response.Read($buffer, 0, $buffer.Length)) -gt 0) {
				$totRead += $read
				$outStream.Write($buffer, 0, $read);
			}
		}
		finally {
			$outStream.Close()
		}
		if(!$SkipIntegrityCheck -and $fragment) {
			Test-FileIntegrity -File $OutFile -Algorithm $algorithm -ExpectedHash $hash
		}
	}
}

[gabriel-samfira]

I wonder if there are any users that need to spin up windows and are doing so with anything older than Windows 10 or the server equivalent. Because starting with Windows 10 (I think), curl.exe is available by default. We could just replace the powershell functions with curl.exe and save a lot of lines of code.

The Invoke-FastWebRequest commandlet probably fails to follow redirects, or after a redirect discards the object storage token it needs to actually download the file. The Invoke-WebRequest commandlet has been historically really slow for no good reason. It should work, but expect longer download times.

I will have a look as soon as I can. Thanks for the debugging and suggested fix!

[gabriel-samfira]

turns out the issue was due to the fact that no user-agent was set by the the HTTP client. This should be fixed in #781 .