Http authentication

[tpetricek]

It would be nice if the Http methods also allowed specifying credentials (so that we can nicely solve something like this).

(But on the other hand, we do not need to have tooltip that does not fit on a screen :-))

[veikkoeeva]

This is a follow-up from my question in Stackoverflow as to how to use JsonProvider with authorization.

I'm currently on a phone (it can take a few days I'll get properly to check this), but to facilitate things, how shoud one go about adding authorization to JsonProvider? Glancing through the sources I see some Http.fs methods accept headers (AsyncRequest/Request -> InnerRequest mainly) and this looks like being the place to add headers. Would this be a correct assumption to make?

Then it looks like the user name and password parameters should be exposed in JsonProvider.fs for user consumption. Would this be a correct assumption to make?

Assuming these are the main places to grok, I didn't see how to the call chain goes to Http.fs and where the headers come to play.

As a side note, there could be some slight crinkles
http://stackoverflow.com/questions/1702426/httpwebrequest-not-passing-credentials
http://devproj20.blogspot.fi/2008/02/assigning-basic-authorization-http.html (for some more elaborate explanation related to the SO link).

[ovatsus]

I would approach in the following way:

• Also support for using something like http://user:password@url in the url parameter of Http.Request, so XmlProvider, JsonProvider & CsvProvider automatically support that also. Do that parsing somewhere around here: https://github.com/fsharp/FSharp.Data/blob/master/src/Net/Http.fs#L185
• When the sample url contains authentication, disable the generation of the GetSample methods for security reasons (so that the username password are not embedded in the generated code without the user realizing that). See here: https://github.com/fsharp/FSharp.Data/blob/master/src/CommonProviderImplementation/Helpers.fs#L259

[ovatsus]

And don't forget to update the docs here: https://github.com/fsharp/FSharp.Data/tree/master/docs/content/library

[veikkoeeva]

Thanks for the nudge, I wonder why I didn't think of the standard URI scheme in the first place and besides, should other ways be supported?

There's some C# code below, apologies for that, and if I'm rehashing something that's in the code already. I'm currently just documenting my current flow of thoughts here (just in case) and charting the territory, so to speak.

I think the very basic way to get basic authentication supported would be to just add

string credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes(new Uri(uri).UserInfo))
client.Headers[HttpRequestHeader.Authorization] = "Basic " + credentials

I'm fairly positive that at least on Windows one wouldn't need to touch the URI (say, to remove credentials or anything) in any way and concerning my small code snippet, only a string.IsNullOrEmpty check should be added, if even that.

But it looks there could be subtle issues left lurking going down this route. Two of which are that authorization with negotiated authentication or with proxies wouldn't work. So, an improved change could be something like

var cc = new CredentialCache();
cc.Add(new Uri(uri), "BASIC", new NetworkCredential("username", "password"));
cc.Add(new Uri(uri), "NEGOTIATE", CredentialCache.DefaultNetworkCredentials);
var client = new WebClient();
client.Credentials = cc;
if(client.Proxy != null) client.Proxy.Credentials = cc;
//client.PreAuthenticate = true;

Where username and password are obtained from Uri.UserInfo by splitting it on colon (other code and creating credentials of course with proper checks).

I'm not sure how I could write tests for the authorization code, but for what I understand from the documentation CredentialCache would be in charge of choosing the correct authorization credentials and would take care, for instance, they aren't accidentaly exposed in presence of redirects.

I used the following resources to gleanse more information on how WebClient authorization is supposed to work

• Network Class Library Team (System.Net): Custom HTTP Authentication Schemes
• Network Class Library Team (System.Net): System.Uri FAQ
• Rick Strahl's Web Log: .NET WebRequest.PreAuthenticate – not quite what it sounds like
• Buck Hodges: Authentication in web services with HttpWebRequest

[ovatsus]

I though of a using the uri just for helping passing the parameters, but it's probably safer to remove that from the uri before constructing it, and handling separately.

Ideally, instead of that, we'd add specific parameters for Http.Request instead, but that wouldn't work at compile time when getting the schema for JsonProvider or XmlProvider, as you only have the url, unless you add user and password parameters to all type providers, which will be a bit of parameter creep, they already have a lot of them.

Testing will be a tricky bit, we currently have zero testing of Http.Request except for it's basic usage throught passing url's in the sample parameter of the type providers, but it would be nice to improve on that

Do what you think it's best and then we'll review and iterate

[ovatsus]

Hi @veikkoeeva, are you getting along with this, or do you need help with something?

[veikkoeeva]

Hi!

I've made progress and I have code that allows amongst other things one to authenticate via HTTP Basic. It has just been a bit bumpy at work, so I haven't had the time to clean things properly and write rationale for the actions I've taken. I'm planning to do something this evening (kids and wife allowing :)), and I expect to submit pull requests no later than during the coming weekend.

Currently in my code there is HTTP Basic, Digest and NTLM/Kerberos (negotiate), of which I'm able to test HTTP Basic (I need to export the code to a domain joined work machine to see if negotiate can be get to work that way). Basically it's done as you can see in the previous snippet on using CredentialCache -- the username and password aren't added to negotiation based authorization currently. I also scrub the URL from the credentials (I haven't gone yet to disable samples). In case of HTTP Basic, though, the credentials are passed as plaintext already (as mandaged by "some" RFC, so scrubbing them from the URL wouldn't provide more security, I believe. Still, perhaps it's good hygiene to remove the credentials from the URI. For security reasons it doesn't seem reasonable to take the credentials in the URI and use them to build negotiation credentials.

Then I have done some research (on more authoritative sources than previous links) as how do (Windows) proxies work and perhaps what kind of authorizations scenarios could be supported and the problems people encounter with authorization. For instance, there is a "double hop issue" when accessing SharePoint, which could be solved by adding two sets of credentials. This particular problem is described succintly at Pass credentials along to new WebRequest, as well as solutions (also in comments). Then there are various issues with proxies, something what looks like being a bug in Digest authorization and so forth, but I'll see if I can write a more coherent memo about things I've come across.

Oh, I believe one could also leverage Windows Credential Manager (or major browser credential sotres), albeit they look like being rather insecure. I suspect I just mention these in the memo and drop them for further consideration at this point.

[veikkoeeva]

This feature lacks documentation, but I'll try to update it today (it's about 17 o'clock here) or the latest tomorrow. I seem to lack time currently, so I pushed it here so we can iterate as discussed.

One place to test the code is at http://eu.httpbin.org/ . Upon reading the code written before me I suspect I semi-accidentally recreated some of the functionality already present in Helpers.fs in some form, mainly regarding on how to recognize if the provided "uri" is a valid one (especially a valid URL).

Perusing some discussion I noticed I probably could hook myself (I'm on Windows 8.1) to the build engine before committing and syncing, but that'd be taken some time to do. Hmm, what else, anything to improve my F# or how to better use GitHub or "behave" in FSharp.Data is fair game too.

[ovatsus]

Yes, we already have the docs using httpbin, I think that's a good option. But these things are always hard to test, so If you only build and test manually it's fine.
I'll wait for you to update the pr with some docs, as I'm busy with other stuff right now, and then I'll look into it and merge in the next couple of days.
Thanks for your work!

[veikkoeeva]

I added some documentation.

You are welcome, and thank you for lending a hand and having patience. It looks like you'll need it, as Travis says the build fails with a message regarding SecureString. I have no clue how to fix that (other than using plain strings, which might be just as secure in this case) as the code compiles just fine on my box. Additionally GitHub says I want to merge master in my pull request, but Travis wants to me belive my changes are already in the master.

In any event, I'll jot down some notes here regarding authorization.

[ovatsus]

Maybe SecureString is not available on mono? I'll have a look later

[veikkoeeva]

It looks like being there in Mono. It looks like Mono's version of NetworkCredentials doesn't support SecureString. For comparison, here the version of MS flavor of .NET. I need to run now, but I'll file a bug later in the evening/niught (unless someone does it before me). Who knows, maybe I will eventually even add the overloads. In the meantime, how should we proceed?
<edit 2014-02-24: It looks like the Mono version lacks only the constructor overloads, but provides a property setter for SecureString password. So, perhaps the course of action is to set the credentials via properties.

Oh, I can iterate this more in case it's not a problem my commits are somewhat spotty and concentrate to weekends. Maybe it's clear, but I also don't mind if someone makes the code better or rearranges the various parts of code I added.

To add more context on what I did and why: Currently I just added support for parsing the userinfo part of the URL, added those credentials to the WebRequest the library will use and removed the credentials from the URL used. I think this scenario needs to be supported since it's a part of the standard URI scheme. However, whilst pondering – unfortunately rather Windows centrically, please, bear with me – what scenarios should and could be supported and what kind of troubles people encounter, I learned more on the authentication story and I believe there is enough of a case to consider exposing a user supplied CredentialCache parameter and/or perhaps some factory methods to build them. Regarding this particular ticket, one could just write

let credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes((new Uri(uri)).UserInfo))
client.Headers[HttpRequestHeader.Authorization] <- "Basic " + credentials

and get the HTTP basic authentication to work. However, if redirects were allowed, the same credentials would be sent to the new address. In that case

• It may not be desired as the redirected address may not be deemed trustworthy.
• The resource may not accept those particular credentials, but some other ones.

By using the CredentialCache one gains control on how the credentials are used. There's a succinct summary on this at Network Class Library Team (System.Net): Custom HTTP Authentication Schemes

WebRequest by default facilitates automatic request redirection from site to site. However if a
server you have logged into decides to redirect you elsewhere, WebRequest will remove its Credentials property before automatically redirecting to prevent your username and password from being exposed. If you are aware that redirects will take place, you can use the CredentialCache class to manage what servers are allowed access to your credentials. A CredentialCache pairs credentials with the URIs and authentication schemes they’re allowed to be used with.

(As a side note, the just mentioned blog writing has a code example on how to create credentials for Google logon, where a token needs to be retrieved first. Maybe those ideas could be used to unify the interface (thinking of Twitter etc.) if one ever decides to support passing CredentialCache as a parameter.)

Adding a set of credentials to CredentialCache and passing it to the WebRequest looks like is also the correct way to handle proxies that require authentication, redirects and what comes to possible mixed authentication schemes. I suspect this is so, I’m not entirely sure. By default WebRequest looks like is using the same proxy settings as IE, which is probably what is generally preferred. The settings can also be altered with the proxy element in .config files. As for proxies in Windows in general at least two authoritative looking posts on the matter look like being

• IEInternals: Proxy-Authentication breaks many applications
• IEInternals: Web Proxy Configuration and IE11 Changes
  A perhaps worthy point is that there’s a user voice ticket on making proxies work better with .NET at Fix it so that .NET apps can access HTTP thru authenticating proxy server by default.

Further ideas (probably bad ones, but I just slap them here too), and why I also brought up the "Google credentials" are elaborated from a post by Jeff Sanders on Using HttpWebRequest with Credential Manager. That is, the user is prompted to enter credentials and they can be saved to the system. This idea could be elaborated on using browser stored credentials, as shown in How Browsers Store Your Passwords (and Why You Shouldn't Let Them) (or rubbing Credential Vault still further Saving Credentials on Windows Computers). These latter points and the Google case mentioned in parenthesis are something I think could be considered as factory methods.