What is a Dataset – And How to Build Your Own with Web Scraping

What is a Dataset?

A dataset is a collection of related data organized in a structured manner. It can take various forms, such as tabular data – with columns representing different variables and rows representing individual records. In other cases, a data set may consist of documents or files.

Essentially, a dataset brings together pieces of information that are grouped together based on their relationship or purpose. It allows for efficient management and access, enabling us to analyze and work with the data individually or collectively.

Think of a data set as a container holding valuable information, like a virtual filing cabinet or a database. It provides a structured framework that helps us store, organize, and make sense of the data.

Public vs. Private Datasets

A public dataset is a collection of information that is available to anyone who wants to access it. It can be used for research, analysis, or any other purpose. Researchers and data scientists often use public datasets to develop new algorithms and models.

A good example of public datasets is those shared by Kaggle.

(These are available for everyone to download and use as they please.)

On the other hand, a private dataset is a collection of information that is only available to a specific group of people or organizations. Companies and organizations often use private datasets to store sensitive information that they do not want to share with the public.

Something to keep in mind is that not because a dataset is paid means it is private. This is an important distinction because access to data can give a business a clear advantage over the competition.

For example, if many businesses buy the same datasets from the same provider, the difference in outcomes would come down to the skill of the analysts and how fast they can act on the insights.

However, when businesses create their own datasets from alternative sources, they can access a wealth of unique insights no other organization or business can.

How to Build a Dataset

Any machine learning (ML) or data analysis project requires information, and there are many ways professionals and businesses can gather it:

• Surveys and Questionnaires – People fill out forms providing the necessary information researchers need.
• Publicly Available Data – Some governments and institutions provide public datasets you can download and use in your projects.
• Transactional Data – Things like sales records provide a lot of useful information businesses can use to better understand the demand for products and services, specific details about their audiences, and more.
• Data APIs – Some websites, data providers, and applications provide open-source and paid APIs that allow us to retrieve information from their databases.

However, in a lot of cases, the data we need is scattered across the web with no direct access to it via APIs or structured formats – a few examples are product data, public social media conversations and posts, forums, etc.

In those cases, the only solution is using web scraping to access, gather and format the data into usable datasets.

Building a Dataset with Web Scraping

Web scraping is the process of programmatically extracting publicly available web data from websites and online platforms. This allows us to have access to millions of data points we can later use to build applications, draw insights to make business decisions, and more.

In other words, we can easily build our own private, structured datasets using web scraping.

For this tutorial, we’ll be using the following five technologies:

• Python
• Requests – to make our HTTP requests
• ScraperAPI – to avoid IP bans and bypass anti-scraping mechanisms
• BeautifulSoup – to parse the HTML response
• Pandas – to build our DataFrame

Note: In Python, a DataFrame is a structure that fits our dataset definition.

Our objective is to scrape book data from https://books.toscrape.com/index.html and build a DataFrame with the name, price, description, stock availability, universal product number (UPC), and category.

TL;DR – Full Code

For those in a hurry or experience, here’s the full codebase that we’re building in this tutorial.

import requests
from bs4 import BeautifulSoup
import pandas as pd

scraperapi = 'https://api.scraperapi.com?api_key=YOUR_API_KEY&url='
all_books = []

for x in range(1, 5):
   response = requests.get(scraperapi + f'https://books.toscrape.com/catalogue/page-{x}.html')
   soup = BeautifulSoup(response.content, 'html.parser')
   onpage_books = soup.find_all('li', class_='col-xs-6')
   for books in onpage_books:
        r = requests.get(scraperapi + 'https://books.toscrape.com/catalogue/' + books.find('a')['href'])
        s = BeautifulSoup(r.content, 'html.parser')
        all_books.append({
             'Name': s.find('h1').text,
             'Description': s.select('p')[3].text,
             'Price': s.find('p', class_='price_color').text,
             'Category': s.find('ul', class_='breadcrumb').select('li')[2].text.strip(),
             'Availability': s.select_one('p.availability').text.strip(),
             'UPC': s.find('table').find_all('tr')[0].find('td').text
        })

df = pd.DataFrame(all_books)
df.to_csv('book_data.csv', index=False)

Before running the script, create a free ScraperAPI account and add your API key to the script within the scraperapi variable.

1. Creating our scraping plan

After getting a clear objective, the next step is navigating to the website to plan how we’ll get the data out.

The website itself is a combination of paginated pages, category pages, and book pages. To get detailed information for each book, we’ll have to navigate to each of those book pages and extract the elements.

However, if we click on any of these books, you’ll notice there’s no way to predict what the URL will be – besides using the name of the book as part of the URL:

https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

That said, upon a closer look, in the paginated series, the site displays 20 books from a total of 1000 books, and if we click on the next button at the bottom of the page, the URL changes to:

https://books.toscrape.com/catalogue/page-2.html

And the number just keeps increasing with each next page:

In other words, by programmatically increasing the number in the URL, we’re able to navigate to the paginated series.

With this information, we can now write our plan down:

• We’ll navigate through the paginated series and gather all books’ URLs
• Then, we can send a request to download each individual book page’s HTML and gather the details we’re interested in
• Lastly, we’ll format all this information into a DataFrame we can use directly or export it into another format if we’d like

Let’s now translate all of this into code!

2. Setting Up Our Project

Let’s create a new project directory and a book_scraper.py file inside. Then, import all our dependencies at the top of the file.

import requests #pip install requests

from bs4 import BeautifulSoup #pip install beautifulsoup4

import pandas as pd #pip install pandas

Note: If you don’t have any of these packages installed, just use the pip command next to it in the code snippet above.

We’ll also need to have access to a ScraperAPI key. Just create a free ScraperAPI account, and you’ll find your key on your dashboard.

Instead of sending all requests from our machine, we’ll send them through ScraperAPI servers. This way, ScraperAPI will smartly rotate our IP between requests and choose the right HTTP headers to ensure a near 100% success rate.

This is important because most websites would quickly identify your scraper and ban your IP, blocking any traffic coming from your machine – potentially forever.

Although there are many ways to do this, we’ll use a quite simple one today. From your dashboard, copy the Sample API Code and paste it into a variable called scraperapi – but deleting the value within the url parameter:

scraperapi = 'https://api.scraperapi.com?api_key=YOUR_API_KEY&url='

This way, we can add scraperapi to our get requests and concatenate it with the target URL. Just to keep it clean.

3. Testing our requests

The first thing to do is test one important hypothesis: if we change the number in https://books.toscrape.com/catalogue/page-2.html to a 1, can we get the homepage? This is critical because it is going to be our initial request.

Two ways we can test this:

1. We change it directly on the address bar on our browser (which works)

2. Send a request through our script (which also works)

response = requests.get('https://books.toscrape.com/catalogue/page-1.html')
print(response.status_code)

It print a 200 status code.

For one last test, let’s add ScraperAPI to the mixed:

response = requests.get(scraperapi + 'https://books.toscrape.com/catalogue/page-1.html')

It returns a 200 status code. Perfect!

4. Navigating through the paginated series

To change the number within our URL, we need to set a range through a range() function and then use an f-string to add the variable to our URL string in the proper place.

for x in range(1, 6):
response = requests.get(scraperapi + f'https://books.toscrape.com/catalogue/page-{x}.html')

Note: The first number in the range specified the initial count (we have to specify it’s 1 because the count starts at 0 by default), and the second the last number of the rage – in this case, the rage will be from 1 to 6 without including 6.

We’re using such a short range to limit our request to five and ensure it is working. Let’s run it with the following print() statement for visual feedback.

print('this is request number ' + str(x) + '. Status code: ' + str(response.status_code))

Result:

this is request number 1. Status code: 200
this is request number 2. Status code: 200
this is request number 3. Status code: 200
this is request number 4. Status code: 200
this is request number 5. Status code: 200

5. Parsing the HTML response

Before we can extract any element, we first need to process the returned HTML with BeautifulSoup and turn it into a parse tree.

soup = BeautifulSoup(response.content, 'html.parser')

This way, we can now use HTML tags and CSS selectors to pick specific elements within the parse tree. But what element first? Well, if we take a look at the page, each book is inside a card-like element.

With that premise in mind, we should be able to find the right element to target in order to store all of them into a single variable.

Bingo! Each element is represented as a <li> tag with a unique CSS class we can easily target using the .find_all() method.

onpage_books = soup.find_all('li', class_='col-xs-6')

As we said before, each page has 20 books, so let’s print the length of onpage_books to ensure we’re getting them all.

print(len(onpage_books))

We got them!