Respect robots.txt by default

[fawkesley]

And probably provide a context manager with ignore_robots for rare occasions when we encounter a broken robots.txt file.

[fawkesley]

See robotparser in Python standard library: http://docs.python.org/2/library/robotparser.html