URLHeadBear: fake the user agent#2879
Conversation
|
Comment on 5d00e26. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
Akhelesh
force-pushed
the
URLHeadBear
branch
from
5d00e26 to
87fa05e
Compare
|
Comment on 87fa05e. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
1 similar comment
|
Comment on 87fa05e. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
Akhelesh
force-pushed
the
URLHeadBear
branch
from
6f10567 to
258838f
Compare
frextrite
left a comment
frextrite
left a comment
There was a problem hiding this comment.
After looking at the said requirements you may also want to squash the commits.
| dennis~=0.9 | ||
| docutils-ast-writer~=0.1.2 | ||
| eradicate~=0.1.6 | ||
| fake-useragent~=0.1.11 |
There was a problem hiding this comment.
headers present in fake-useragent are pretty old. Do we really want to use it?
| @staticmethod | ||
| def get_head_response(url, timeout): | ||
| try: | ||
| headers = {'User-Agent': UserAgent().random} |
There was a problem hiding this comment.
And why are we using a random UserAgent? Using a specific browser header like firefox/chrome would be much better.
On that note, why do we even need to generate a user-agent. Why not just hardcode the latest chrome header?
There was a problem hiding this comment.
@frextrite I tried using a random user agent since some websites block the user if it makes many requests but I Googled more to find out that it would be useless unless IP addresses are rotated too. So in this case your solution suits better.
Akhelesh
force-pushed
the
URLHeadBear
branch
from
a50aec0 to
258838f
Compare
A random fake user agent is used to request webpages that do not allow bot visits. Fixes coala#1203
Akhelesh
force-pushed
the
URLHeadBear
branch
from
258838f to
76cb8cc
Compare
frextrite
left a comment
frextrite
left a comment
There was a problem hiding this comment.
A few more changes,
- change the commit body according to updated changes
- read, follow and tick the items in PR body
- look at Travis CI build logs and fix the failing tests,
and you'll be good to go.
|
You could run |
Yeah I have created an issue for the same to keep it separate from this one. |
frextrite
left a comment
frextrite
left a comment
There was a problem hiding this comment.
I think instead of creating a separate issue you may want to work on the tests in this PR directly. Since the changes in this PR are directly affecting the tests, they should be resolved in this PR.
5 participants
A random fake user agent is used to request webpages
that do not allow bot visits.
Fixes #1203
For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!
Checklist
them.
individually. It is not sufficient to have "fixup commits" on your PR,
our bot will still report the issues for the previous commit.) You will
likely receive a lot of bot comments and build failures if coala does not
pass on every single commit!
After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.
Please consider helping us by reviewing other peoples pull requests as well:
corobo mark wip <URL>to get it outof the review queue.
The more you review, the more your score will grow at coala.io and we will
review your PRs faster!