How to Scrape Amazon Product Data: Prices, Search Page, Etc.

How to Approach Amazon Scraping

The first method for scraping data from Amazon is to crawl each keyword’s category or shelf list, then request the product page for each one before moving on to the next. This is best for smaller scale, less-repetitive scraping.

The second option is to create a database of products you want to track by having a list of products or ASINs (unique product identifiers), then have your Amazon web scraper scrape each of these individual pages every day/week/etc. This is the most common method among scrapers who track products for themselves or as a service.

Scraping Amazon Product Data with Scrapy

For this tutorial, we’ll focus on the second method, making our script first do an Amazon search and then collecting the product data using the ASIN number from the products listed on the page.

Also, we’ll be using ScraperAPI to help us bypass Amazon’s anti-bot measures, allowing us to scale our project without having to build a complete scraping infrastructure from scratch and for a fraction of the cost of using residential proxies.

Step 1: Setup Your Project

To build our scraper, we’ll be using Python’s Scrapy, a web crawling and data extraction platform that can be used for a variety of applications such as data mining, information retrieval and historical archiving

if you haven’t already, install Python before you can using the following pip command to install Scrapy:

pip install scrapy

Then go to the folder where your project is saved (Scrapy automatically creates a web scraping project folder for you) and run the startproject command along with the project name, amazon_scraper.

Scrapy will construct a web scraping project folder for you, with everything already set up:

scrapy startproject amazon_scraper
<p>

The result should look like this:

</p>
├── scrapy.cfg # deploy configuration file
└── tutorial # project's Python module, you'll import your code from here
	├── __init__.py
	├── items.py # project items definition file
	├── middlewares.py # project middlewares file
	├── pipelines.py # project pipeline file
	├── settings.py # project settings file
	└── spiders # a directory where spiders are located
		├── __init__.py
		└── amazon.py # spider we just created
<p>

Scrapy creates all of the files you’ll need, and each file serves a particular purpose:

1. Items.py – Can be used to build your base dictionary, which you can then import into the spider.
2. Settings.py – All of your request settings, pipeline, and middleware activation happens in settings.py. You can adjust the delays, concurrency, and several other parameters here.
3. Pipelines.py – The item yielded by the spider is transferred to Pipelines.py, which is mainly used to clean the text and bind to databases (Excel, SQL, etc).
4. Middlewares.py – When you want to change how the request is made and scrapy manages the answer, Middlewares.py comes in handy.

Step 2: Create an Amazon Spider

You’ve established the project’s overall structure, so now you’re ready to start working on the spiders that will do the scraping.

Scrapy has a variety of spider species, but we’ll focus on the most popular one: the Generic Spide

Simply run the genspider command to make a new spider:

</p>
# syntax is --> scrapy genspider name_of_spider website.com 
scrapy genspider amazon amazon.com
<p>

Scrapy now creates a new file with a spider template, and you’ll gain a new file called amazon.py in the spiders folder.

Your code should look like the following:

</p>
import scrapy
class AmazonSpider(scrapy.Spider):
	name = 'amazon'
	allowed_domains = ['amazon.com']
	start_urls = ['http://www.amazon.com/']
	def parse(self, response):
		pass
<p>

Delete the default code (allowed_domains, start_urls, and the parse() function) and replace it with your own, which should include these four functions:

1. start_requests – sends an Amazon search query with a specific keyword.
2. parse_keyword_response – extracts the ASIN value for each product returned in an Amazon keyword query, then sends a new request to Amazon for the product listing. It will also go to the next page and do the same thing.
3. parse_product_page – extracts all of the desired data from the product page.
4. get_url – sends the request to the ScraperAPI, which will return an HTML response.

Step 3: Send a Search Query to Amazon

You can now scrape Amazon for a particular keyword using the following steps, with an Amazon spider and ScraperAPI as the proxy solution. This will allow you to scrape all of the key details from the product page and extract each product’s ASIN.

All pages returned by the keyword query will be parsed by the spider. Try using these fields for the spider to scrape from the Amazon product page:

• ASIN
• Product name
• Price
• Product description
• Image URL
• Available sizes and colors
• Customer ratings
• Number of reviews
• Seller ranking

Important Update:

Using our Amazon Search structured data endpoint (SDE), you can now collect all mentioned product details like ASIN, name, price, images, descriptions, and more in JSON format from Amazon search results with a simple API call.

Just send you requests to our endpoint, alongside your API key and the query you want to collect data from.

Here’s a code snippet to get you started:

import requests

payload = {
   'api_key': 'APIKEY',
   'query': 'QUERY',
   'country': 'COUNTRY',
   'tld': 'TLD'
}

r = requests.get('https://api.scraperapi.com/structured/amazon/search', params=payload)
print(r.text)

Don’t have an API key? Create a free ScraperAPI account to start your 7-day free trial with 5,000 API credits and all our tools available.

The first step is to create a start_requests() function to send our Amazon search requests containing our keywords.

Outside of AmazonSpider, you can easily identify a list variable using our search keywords. Input the keywords you want to search for in Amazon into your script:

</p>
queries = [‘tshirt for men’, ‘tshirt for women’]
<p>

Inside the AmazonSpider, you can build your start_requests feature, which will submit the requests to Amazon. Submit a search query k=SEARCH KEYWORD to access Amazon’s search features via a URL:

</p>
https://www.amazon.com/s?k=<SEARCH_KEYWORD>
<p>

It looks like this when we use it in the start_requests() function:

</p>
## amazon.py
queries = ['tshirt for men', ‘tshirt for women’]
class AmazonSpider(scrapy.Spider):
	def start_requests(self):
		for query in queries:
			url = 'https://www.amazon.com/s?' + urlencode({'k': query})
			yield scrapy.Request(url=url, callback=self.parse_keyword_response)
<p>

You will urlencode each query in your queries list so that it is secure to use as a query string in a URL, and then use scrapy.Request to request that URL.

Use yield instead of return since Scrapy is asynchronous, so the functions can either return a request or a completed dictionary.

If a new request is received, the callback method is invoked. If an object is yielded, it will be sent to the data cleaning pipeline.

The parse_keyword_response callback function will then extract the ASIN for each product when scrapy.Request activates it.

Step 4: Scrape Amazon Products Using ASINs

One of the most popular methods to scrape Amazon includes extracting data from a product listing page.

Important Update

You can now use ScraperAPI’s Amazon Product SDE to scrape any product listing using a list of ASIN.

Just send your request alongside your API key and ASIN (or list of ASINs) and get detailed information like name, price, reviews, rating, and more.

Here’s a code snippet to get you started:

import requests

payload = {
   'api_key': 'APIKEY',
   'asin': 'ASIN',
   'country': 'COUNTRY',
   'tld': 'TLD'
}

r = requests.get('https://api.scraperapi.com/structured/amazon/product', params=payload)
print(r.text)

Want to learn more? Check our tutorial on scraping Amazon ASINs at scale.

Using an Amazon product page ASIN ID is the simplest and most common way to retrieve this data as very product on Amazon has a unique ASIN assigned.

We may use this ID in our URLs to get the product page for any Amazon product, such as the following:

</p>
https://www.amazon.com/dp/<ASIN>
<p>

Using Scrapy’s built-in XPath selector extractor methods, we can extract the ASIN value from the product listing tab. You can build an XPath selector in Scrapy Shell that captures the ASIN value for each product on the product listing page and generates a url for each product:

</p>
products = response.xpath('//*[@data-asin]')
		for product in products:
			asin = product.xpath('@data-asin').extract_first()
			product_url = f"https://www.amazon.com/dp/{asin}"
<p>

The function will then be configured to send a request to this URL and then call the parse_product_page() callback function when it receives a response.

Note: This request will also include the meta parameter, which is used to move items between functions or edit certain settings.

</p>
def parse_keyword_response(self, response):
		products = response.xpath('//*[@data-asin]')
		for product in products:
			asin = product.xpath('@data-asin').extract_first()
			product_url = f"https://www.amazon.com/dp/{asin}"
			yield scrapy.Request(url=product_url, callback=self.parse_product_page, meta={'asin': asin})
<p>

Step 5: Extract Specific Data From the Amazon Product Page

After the parse_keyword_response() function requests the product pages URL, it transfers the response it receives from Amazon along with the ASIN ID in the meta parameter to the parse product page callback function.

We now want to derive the information we need from a product page, such as a product page for a t-shirt.

To do this, you need to create XPath selectors to extract each field from the HTML response we get from Amazon:

</p>
def parse_product_page(self, response):
	asin = response.meta['asin']
	title = response.xpath('//*[@id="productTitle"]/text()').extract_first()
	image = re.search('"large":"(.*?)"',response.text).groups()[0]
	rating = response.xpath('//*[@id="acrPopover"]/@title').extract_first()
	number_of_reviews = response.xpath('//*[@id="acrCustomerReviewText"]/text()').extract_first()
	bullet_points = response.xpath('//*[@id="feature-bullets"]//li/span/text()').extract()
	seller_rank = response.xpath('//*[text()="Amazon Best Sellers Rank:"]/parent::*//text()[not(parent::style)]').extract()
<p>

Note: Try using a regex selector over an XPath selector for scraping the image URL if the XPath is extracting the image in base64.

When working with large websites like Amazon that have a variety of product pages, you’ll find that writing a single XPath selector isn’t always enough since it will work on certain pages but not others. To deal with the different page layouts, you’ll need to write several XPath selectors in situations like these.

When you run into this issue, give the spider three different XPath options:

</p>
def parse_product_page(self, response):
	asin = response.meta['asin']
	title = response.xpath('//*[@id="productTitle"]/text()').extract_first()
	image = re.search('"large":"(.*?)"',response.text).groups()[0]
	rating = response.xpath('//*[@id="acrPopover"]/@title').extract_first()
	number_of_reviews = response.xpath('//*[@id="acrCustomerReviewText"]/text()').extract_first()
	bullet_points = response.xpath('//*[@id="feature-bullets"]//li/span/text()').extract()
	seller_rank = response.xpath('//*[text()="Amazon Best Sellers Rank:"]/parent::*//text()[not(parent::style)]').extract()
	price = response.xpath('//*[@id="priceblock_ourprice"]/text()').extract_first()
	if not price:
		price = response.xpath('//*[@data-asin-price]/@data-asin-price').extract_first() or \
				response.xpath('//*[@id="price_inside_buybox"]/text()').extract_first()
<p>

If the spider is unable to locate a price using the first XPath selector, it goes on to the next.