Selenium

Although most websites on the internet can be crawled using simple crawling techniques, some websites require more sophisticated crawling methods to extract data from them. For example, some data is hidden behind buttons that need to be clicked first. Sometimes, filters need to be set first before data is shown. Also, some websites are created using JavaScript instead of HTML, making it much more difficult to crawl data. To solve this problem, a Python library called Selenium can be used.

Using Selenium

In order to use Selenium, we need to first initialize the Chrome driver and create a Selenium object.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless') # This allows you to crawl without opening a separate chrome window
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)

Then, we can use the driver to make a request to a web page.

url = "https://www.amazon.co.uk/"
driver.get(url)

Just like normal crawling, Selenium can also extract HTML code from a web page. Here is an example:

One advantage of Selenium is that it can parse through the HTML code to interact with the web page. For example, I can find a search bar and write some text using the following code:

element = driver.find_element(By.ID, "twotabsearchtextbox")
from selenium.webdriver.common.keys import Keys
element.send_keys("shoes")
element.send_keys(Keys.ENTER)
time.sleep(10)

Then, we can parse through the HTML code again to extract the data we need.

from selenium.common.exceptions import NoSuchElementException
names = []
prices = []
links = []
names, prices, links = add_items(names, prices, links)
crawling = True
while crawling:
    try:
        driver.find_element(By.XPATH, ".//a[@class = 's-pagination-item s-pagination-next s-pagination-button s-pagination-separator']").send_keys(Keys.ENTER)
        time.sleep(10)
        names, prices, links = add_items(names, prices, links)
    except NoSuchElementException:
        print("There is no more next page.")
        crawling = False
        
df = pd.DataFrame(zip(names, prices, links), columns=['Item Name', 'Price', 'Link'])

Running the following code will extract data and save it as a Pandas DataFrame