According to the global Internet accessibility audit report released by authoritative organizations, about three quarters of websites in the world have content or part of their content generated dynamically through JavaScript. This means that when we use "View Page Source" in the browser window, we cannot find these contents in the HTML code, and the way we used before to get data cannot work normally anymore. There are basically two solutions to this problem. One is to get the data interface that provides the dynamic content. This way is also suitable for getting data from mobile apps. The other is to run a browser through the automated testing tool Selenium and get the rendered dynamic content. For the first solution, we can use the browser's Developer Tools or more professional packet-capture tools, such as Charles, Fiddler, or Wireshark, to get the data interface. The later operations are the same as the way talked about in the previous lesson for getting the data of the 360 Image website, so we will not repeat them here. In this chapter, we mainly explain how to use the automated testing tool Selenium to get the dynamic content of websites.
Selenium is an automated testing tool. By using it, we can drive the browser to carry out specific actions, and finally help crawler developers get the dynamic content of web pages. Simply speaking, as long as we can see the content in the browser window, Selenium can get it. For those websites that use JavaScript dynamic rendering technology, Selenium is an important choice. Below, we still use the Chrome browser as an example to explain the use of Selenium. Everyone needs to first install Chrome and download its driver. Chrome's driver program can be downloaded from the ChromeDriver official website. The version of the driver needs to match the version of the browser. If there is no exactly matching version, choose the one whose version number is closest.
We can first install Selenium through pip, and the command is shown below.
pip install seleniumNext, we use the code below to drive Chrome to open Baidu.
from selenium import webdriver
# Create a Chrome browser object
browser = webdriver.Chrome()
# Load the specified page
browser.get('https://www.baidu.com/')If you do not want to use Chrome, you can also change the code above to control other browsers. You only need to create the matching browser object, such as Firefox or Safari. If you run the program above and see the error message shown below, it means we have not yet added the Chrome browser driver to the PATH environment variable, and we also did not specify the location of the Chrome browser driver in the program.
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
There are three ways to solve this problem:
-
Put the downloaded ChromeDriver into a path that is already under the PATH environment variable. It is recommended to directly put it in the same directory as the Python interpreter, because when Python was installed before, we already put the path of the Python interpreter into the PATH environment variable.
-
Put ChromeDriver into the
binfolder under the project virtual environment, and the matching directory on Windows isScripts. In this way, ChromeDriver is in the same place as the Python interpreter under the virtual environment, and it can definitely be found. -
Modify the code above. When creating the Chrome object, configure a
Serviceobject through theserviceparameter, and specify the location of ChromeDriver through theexecutable_pathparameter when creating theServiceobject, as shown below.from selenium import webdriver from selenium.webdriver.chrome.service import Service browser = webdriver.Chrome(service=Service(executable_path='venv/bin/chromedriver')) browser.get('https://www.baidu.com/')
Next, we can try to simulate the user typing a search keyword into the text box on the Baidu home page and clicking the "Baidu Search" button. After the page is loaded, we can get page elements through the find_element and find_elements methods of the Chrome object. Selenium supports many ways of getting elements, including CSS selectors, XPath, element name, meaning tag name, element ID, class name, and so on. The former gets a single page element, a WebElement object, and the latter gets a list made up of multiple page elements. After getting a WebElement object, we can simulate user input behavior through send_keys, and simulate user click operations through click. The code is shown below.
from selenium import webdriver
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
browser.get('https://www.baidu.com/')
# Get an element through its ID
kw_input = browser.find_element(By.ID, 'kw')
# Simulate user input behavior
kw_input.send_keys('Python')
# Get an element through a CSS selector
su_button = browser.find_element(By.CSS_SELECTOR, '#su')
# Simulate user click behavior
su_button.click()If you need to perform a series of actions, such as simulating drag-and-drop, you can create an ActionChains object. Interested readers can study it by themselves.
There is one more detail everyone needs to know here. Elements on a page may be generated dynamically. When we use the find_element or find_elements methods to get them, rendering may not be finished yet, and then a NoSuchElementException error is raised. To solve this problem, we can use implicit wait by setting a waiting time so the browser can finish rendering page elements. Besides that, we can also use explicit wait. By creating a WebDriverWait object and setting the waiting time and conditions, if the conditions are not met, we can wait first and then try later operations. The specific code is shown below.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
browser = webdriver.Chrome()
# Set browser window size
browser.set_window_size(1200, 800)
browser.get('https://www.baidu.com/')
# Set the implicit waiting time to 10 seconds
browser.implicitly_wait(10)
kw_input = browser.find_element(By.ID, 'kw')
kw_input.send_keys('Python')
su_button = browser.find_element(By.CSS_SELECTOR, '#su')
su_button.click()
# Create an explicit wait object
wait_obj = WebDriverWait(browser, 10)
# Set the waiting condition, wait for the search result div to appear
wait_obj.until(
expected_conditions.presence_of_element_located(
(By.CSS_SELECTOR, '#content_left')
)
)
# Take a screenshot
browser.get_screenshot_as_file('python_result.png')The waiting condition set above, presence_of_element_located, means waiting for the specified element to appear. The table below lists common waiting conditions and their meanings.
| Waiting Condition | Meaning |
|---|---|
title_is / title_contains |
The title is the specified content / the title contains the specified content |
visibility_of |
The element is visible |
presence_of_element_located |
The located element has finished loading |
visibility_of_element_located |
The located element becomes visible |
invisibility_of_element_located |
The located element becomes invisible |
presence_of_all_elements_located |
All located elements have finished loading |
text_to_be_present_in_element |
The element contains the specified content |
text_to_be_present_in_element_value |
The value attribute of the element contains the specified content |
frame_to_be_available_and_switch_to_it |
Load and switch to the specified inner window |
element_to_be_clickable |
The element can be clicked |
element_to_be_selected |
The element is selected |
element_located_to_be_selected |
The located element is selected |
alert_is_present |
An alert popup appears |
For a page that uses waterfall loading, if you want to load more content in the browser window, you can do it by running JavaScript code through the execute_scripts method of the browser object. Similar operations may also be needed in some more advanced crawling actions. If your crawler code needs JavaScript support, it is recommended to learn JavaScript properly first, especially the BOM and DOM operations in JavaScript. If we add the code below before taking the screenshot in the code above, then JavaScript can be used to scroll the page to the very bottom.
# Run JavaScript code
browser.execute_script('document.documentElement.scrollTop = document.documentElement.scrollHeight')Some websites set anti-crawler measures specifically for Selenium, because in a browser driven by Selenium, the value of the webdriver property shown in the console is true, as shown below. If you want to bypass this check, you can first change it to undefined by running JavaScript code before loading the page.
On the other hand, we can also hide the text "Chrome is being controlled by automated test software" on the browser window. The full code is shown below.
# Create a Chrome options object
options = webdriver.ChromeOptions()
# Add experimental parameters
options.add_experimental_option('excludeSwitches', ['enable-automation'])
options.add_experimental_option('useAutomationExtension', False)
# Create a Chrome browser object and pass in the parameters
browser = webdriver.Chrome(options=options)
# Run a Chrome DevTools Protocol command
# execute the specified JavaScript code when the page is loaded
browser.execute_cdp_cmd(
'Page.addScriptToEvaluateOnNewDocument',
{'source': 'Object.defineProperty(navigator, "webdriver", {get: () => undefined})'}
)
browser.set_window_size(1200, 800)
browser.get('https://www.baidu.com/')Many times, when we are collecting data, we do not need to see the browser window. As long as there is Chrome and its matching driver program, our crawler can run. If you do not want to see the browser window, we can set a headless browser in the way below.
options = webdriver.ChromeOptions()
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)There is still much more Selenium knowledge, and we will not repeat it one by one here. Below, we list some common properties and methods of browser objects and WebElement objects. For the exact content, everyone can also refer to the Chinese translation of the Selenium official documentation.
Table 1. Common properties
| Property Name | Description |
|---|---|
current_url |
URL of the current page |
current_window_handle |
Handle, meaning reference, of the current window |
name |
Name of the browser |
orientation |
Direction of the current device, landscape or portrait |
page_source |
Source code of the current page, including dynamic content |
title |
Title of the current page |
window_handles |
Handles of all windows opened by the browser |
Table 2. Common methods
| Method Name | Description |
|---|---|
back / forward |
Go backward / forward in browser history |
close / quit |
Close the current browser window / exit the browser instance |
get |
Load the page of the specified URL into the browser |
maximize_window |
Maximize the browser window |
refresh |
Refresh the current page |
set_page_load_timeout |
Set the page loading timeout |
set_script_timeout |
Set the JavaScript execution timeout |
implicit_wait |
Set waiting for elements to be found or target actions to finish |
get_cookie / get_cookies |
Get the specified cookie / get all cookies |
add_cookie |
Add cookie information |
delete_cookie / delete_all_cookies |
Delete the specified cookie / delete all cookies |
find_element / find_elements |
Find a single element / find a series of elements |
Table 1. Common properties of WebElement
| Property Name | Description |
|---|---|
location |
Position of the element |
size |
Size of the element |
text |
Text content of the element |
id |
ID of the element |
tag_name |
Tag name of the element |
Table 2. Common methods
| Method Name | Description |
|---|---|
clear |
Clear the content in a text box or text area |
click |
Click the element |
get_attribute |
Get the attribute value of the element |
is_displayed |
Judge whether the element is visible to the user |
is_enabled |
Judge whether the element is usable |
is_selected |
Judge whether the element, meaning a radio button or checkbox, is selected |
send_keys |
Simulate text input |
submit |
Submit a form |
value_of_css_property |
Get the specified CSS property value |
find_element / find_elements |
Get one child element / get a series of child elements |
screenshot |
Generate a snapshot for the element |
The example below shows how to use Selenium to search and download images from the 360 Image website.
import os
import time
from concurrent.futures import ThreadPoolExecutor
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
DOWNLOAD_PATH = 'images/'
def download_picture(picture_url: str):
"""
Download and save the picture
:param picture_url: URL of the picture
"""
filename = picture_url[picture_url.rfind('/') + 1:]
resp = requests.get(picture_url)
with open(os.path.join(DOWNLOAD_PATH, filename), 'wb') as file:
file.write(resp.content)
if not os.path.exists(DOWNLOAD_PATH):
os.makedirs(DOWNLOAD_PATH)
browser = webdriver.Chrome()
browser.get('https://image.so.com/z?ch=beauty')
browser.implicitly_wait(10)
kw_input = browser.find_element(By.CSS_SELECTOR, 'input[name=q]')
kw_input.send_keys('Cang Laoshi')
kw_input.send_keys(Keys.ENTER)
for _ in range(10):
browser.execute_script(
'document.documentElement.scrollTop = document.documentElement.scrollHeight'
)
time.sleep(1)
imgs = browser.find_elements(By.CSS_SELECTOR, 'div.waterfall img')
with ThreadPoolExecutor(max_workers=32) as pool:
for img in imgs:
pic_url = img.get_attribute('src')
pool.submit(download_picture, pic_url)Run the code above, and check whether the images found by the keyword search have been downloaded under the specified directory.