How to Scrape Amazon using Python and Beautiful Soup

If you inspect the html variable, you will see that this time you received the complete search result page as desired.

While we are at it, let’s load the response in a BeautifulSoup object as well:

	from bs4 import BeautifulSoup

	soup = BeautifulSoup(html.text)

Sweet! Now that you have the complete response in a BeautifulSoup object, you can start extracting relevant data from it.

Scraping Amazon Product Attributes

The easiest way to figure out how to scrape the required data is by using the Developer Tools, which are available in almost all famous browsers.

The Developer Tools will let you explore the DOM structure of the page.

The main goal is to figure out which HTML tag attributes can let you uniquely target an HTML tag.

Most of the time, you will rely on the id and class attributes. Then you can supply these attributes to BeautifulSoup and ask it to return whatever text/attribute you want to extract from that particular tag.

Extracting the Product Name

Let’s take a look at how you can extract the product name. This will give you a good understanding of the general process.

Right-click on the product name and click on Inspect:

This will open up the developer tools:

As you can observe in the screenshot above, the product name is nested in a span with the following classes: a-size-medium a-color-base a-text-normal.

At this point, you have a decision to make: you can either ask BeautifulSoup to extract all the spans with these classes from the page, or you can extract each result div and then loop over those divs and extract data for each product.

I generally prefer the latter method as it helps identify products that might not have all the required data. This tutorial will also showcase this same method.

Therefore, now you need to identify the div that wraps each result item:

According to the screenshot above, each result is nested in a div tag with the data-component-type attribute of s-search-result.

Let’s use this information to extract all the result divs and then loop over them and extract the nested product titles:

	results = soup.find_all('div', attrs={'data-component-type': 's-search-result'})
	for r in results:
	   print(r.select_one('.a-size-medium.a-color-base.a-text-normal').text)

Here’s a simple breakdown of what this code is doing:

• It uses the find_all() method provided by BeautifulSoup
• It returns all the matching elements from the HTML
• It then uses the select_one() method to extract the first element that matches the CSS Selector passed in

Notice that here we append a dot (.) before each class name. This tells BeautifulSoup that the passed-in CSS Selector is a class name. There is also no space between the class names. This is important as it informs BeautifulSoup that each class is from the same HTML tag.

If you are new to CSS selectors, you should read our CSS selectors cheat sheet, we go over the basics of CSS selectors and provide you with an easy-to-use framework to speed up the process.

Extracting the Product Price

Now that you have extracted the product name, extracting the product price is fairly straightforward.

Follow the same steps from the last section and use the Developer Tools to inspect the price:

The price can be extracted from the span with the class of a-offscreen. This span is itself nested in another span with a class of a-price. You can use this knowledge to craft a CSS selector:

	for r in results:
        # -- truc --
    	print(r.select_one('.a-price .a-offscreen').text)

As you want to target nested spans this time, you have to add a space between the class names.

Extracting the Product Image

Try following the steps from the previous two sections to come up with the relevant code on your own. Here is a screenshot of the image being inspected in the Developer Tools window:

The img tag has a class of s-image. You can target this img tag and extract the src attribute (the image URL) using this code:

	for r in results:
    	# -- truc --
    	print(r.select_one('.s-image').attrs['src'])

Note: Extra points if you do it on your own!

Complete Scraper Code

You have all the bits and pieces to put together the complete scraper code.

Here is a slightly modified version of the scraper that appends all the product results into a list at the very end:

	import requests
	from bs4 import BeautifulSoup
	
	url = "https://www.amazon.com/s?k=iphone+15+charger&s=price-asc-rank"
	headers = {
		'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
		'Accept-Language': 'en-US, en;q=0.5'
	}
	html = requests.get(url, headers=headers)
	soup = BeautifulSoup(html.text)
	results = soup.find_all('div', attrs={'data-component-type': 's-search-result'})
	
	result_list = []
	for r in results:
	   result_dict = {
		  "title": r.select_one('.a-size-medium.a-color-base.a-text-normal').text,
		  "price": r.select_one('.a-price .a-offscreen').text,
		  "image": r.select_one('.s-image').attrs['src']
	   }
	   result_list.append(result_dict)

You can easily repurpose this result_list to power different APIs or store this information in a spreadsheet for further analysis.

Using Structured Data Endpoint

This tutorial did not focus too much on the gotchas of scraping Amazon at scale.

Amazon is notorious for banning scrapers and making it difficult to scrape data from their websites.

You already saw a glimpse of this during the very beginning of the tutorial, where a request without the proper headers was blocked by Amazon.

Luckily, there is an easy solution to this problem. Instead of sending a request directly to Amazon, you can send the request to ScraperAPI’s Structured Data endpoint, and ScraperAPI will respond with the scraped data in nicely formatted JSON.

This is a very powerful feature offered by ScraperAPI, as by using this, you do not have to be worried about getting blocked by Amazon or keeping your scraper updated with the never-ending changes in Amazon’s anti-bot techniques.

The best part is that ScraperAPI provides 5,000 free API credits for 7 days on a trial basis and then provides a generous free plan with recurring 1,000 API credits to keep you going. This is enough to scrape data for general use.

You can quickly get started by going to the ScraperAPI dashboard page and signing up for a new account:

After signing up, you will see your API Key:

Now you can use the following code to access the search results from Amazon using the Structured Data endpoint:

	import requests

	payload = {
	   'api_key': 'API_KEY,
	   'query': 'iphone 15 charger',
	   's': 'price-asc-rank'
	}
	 
	response = requests.get('https://api.scraperapi.com/structured/amazon/search', params=payload)
	print(response.json())

Note: Don’t forget to replace API_KEY in the code above with your own ScraperAPI API key.

As you might have already observed, you can pass in most query params that Amazon accepts as part of the payload. This means all the following sorting values are valid for the s key in the payload:

1. Price: High to low = price-desc-rank
2. Price: Low to high = price-asc-rank
3. Featured = rerelevanceblender
4. Avg. customer review = review-rank
5. Newest arrivals = date-desc-rank

If you want the same result_list data as from the last section, you can add the following code at the end:

	result_list = []
	for r in response.json()['results']:
	   result_dict = {
		  "title": r['name']
		  "price": r['price_string'],
		  "image": r['image']
	   }
	   result_list.append(result_dict)

You can learn more about this endpoint over at the ScraperAPI docs.

Wrapping Up

This tutorial was a quick rundown of how to go about scraping data from Amazon.

• It taught you a simple bypass method for the bot detection system used by Amazon.
• It showed you how to use the various methods provided by BeautifulSoup to extract the required data from the HTML document.
• Lastly, you learned about the Structured Data endpoint offered by ScraperAPI and how it solves quite a few problems.

If you are ready to take your data collection from a couple of pages to thousands or even millions of pages, our Business Plan is a great place to start.

Need 10M+ API credits? Contact sales for a custom plan, including all premium features, premium support, and an account manager.

Until next time, happy scraping!