Crawling refers to following the links on a page and continuing to find and follow new pages to other pages. The main reason we use crawling in data science or machine learning is to extract data from websites. For example, if we need “news” data, then we can crawl a news website and gather articles from that website.
Manually getting the data from online sources is not a feasible option, so we can write some code that will automate this process for us.
To get information from a certain website, we first need to request that website. Although there are many types of requests, we learned about two of the important ones.
GET requests include all required data in the URL, while POST requests add additional data in the message body when making requests. The type of request that needs to be used is decided by the target website. Therefore, when making a request, we first need to identify whether we need to make a GET request or a POST request.
Below is an example code for making a GET request to a website about inspirational quotes:
import requests # Use Python's requests library to make requests
url = "https://www.passiton.com/inspirational-quotes"
response = requests.get(url)
After making a request, the data can be returned to use in various forms such as HTML, JSON, XML, etc. The most common form is HTML. However, HTML contains tags, which we do not need when we are crawling for data. So, we need to parse the HTML code and extract the data that we need.
To do this, we used a Python library called BeautifulSoup that will help us parse through the HTML code and target certain sections of the code that is holding our data.
Below is some example code of BeautifulSoup that extracts all quotes from the website:
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.content, 'html5lib')
table = soup.find('div', attrs={'id': 'all_quotes'})
quotes = []
for row in table.findAll('div', attrs={'class': 'col-6 col-lg-4 text-center margin-30px-bottom sm-margin-30px-top'}):
quote = {}
quote['theme'] = row.h5.text
quote['url'] = row.a['href']
quote['img'] = row.img['src']
quote['inspirational_quote'] = row.img['alt'].split(' #')[0]
quote['author'] = row.img['alt'].split(' #')[1]
quotes.append(quote)
The result of this code looks like this:
Now, the data that we crawled can be converted into a Pandas DataFrame or exported into a CSV file so that it can be used for further data analysis.