Scraping review data from Target.com can be tricky. Like many modern websites, Target uses tools to block bots, such as rate limiting, IP blocking, and dynamic content that loads with JavaScript. This means that basic scraping methods often won’t work well or may get blocked quickly.
In this tutorial, I’ll walk you through scraping product reviews from Target.com using Python, Selenium, and ScraperAPI in proxy mode. ScraperAPI helps us avoid getting blocked by rotating IP addresses, making collecting the data we need easier.
By the end of this guide, you’ll have a solid scraping setup that can reliably pull product reviews from Target.com—great for research, analysis, or building tools that rely on honest customer opinions.
Ready? Let’s get started!
TL;DR: Scraping Target Product Reviews [Full Code]
If you’re in a hurry, here’s the full Target Product Scraper:
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import undetected_chromedriver as uc
from bs4 import BeautifulSoup
import json
from time import sleep
from dotenv import load_dotenv
import os
load_dotenv() # Loads the .env file
API_KEY = os.getenv("SCRAPERAPI_KEY")
target_url = "https://www.target.com/p/enclosed-cat-litter-box-xl-up-up/-/A-90310047?preselect=87059440#lnk=sametab"
# Setup proxy and driver
proxy_url = f"http://scraperapi.render=true.country_code=us:{API_KEY}@proxy-server.scraperapi.com:8001"
options = Options()
#options.add_argument("--headless") # Optional: enable for headless mode
seleniumwire_options = {
'proxy': {
'http': proxy_url,
'https': proxy_url,
},
'verify_ssl': False,
}
driver = uc.Chrome(options=options, seleniumwire_options=seleniumwire_options)
# Load page
print(f"Opening URL: {target_url}")
driver.get(target_url)
sleep(10)
# Scroll and click "Show more" if it appears
print("Scrolling and checking for 'Show more' buttons...")
for i in range(3):
try:
driver.execute_script("window.scrollTo(0, 3200);")
show_more = WebDriverWait(driver, 5).until(
EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/div[2]/main/div/div[3]/div/div[17]/div/div[5]/button"))
)
show_more.click()
show_more.click()
print(f"'Show more' clicked ({i + 1})")
sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(3)
except:
print(f"No more 'Show more' button after {i} click(s).")
break
# Wait for reviews to appear
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "div[data-test='reviews-list']"))
)
except:
print("Review section may not be fully loaded.")
driver.quit()
exit()
sleep(3)
# Parse reviews
soup = BeautifulSoup(driver.page_source, "lxml")
driver.quit()
data = []
review_divs = soup.find_all("div", attrs={"data-test": "review-card--text"})
rating_spans = soup.find_all("span", class_=["styles_ndsScreenReaderOnly__mcNC_", "styles_notFocusable__XkHOR"])
for review in review_divs:
review_text = review.get_text(strip=True)
data.append({"review": review_text})
for i, rating in enumerate(rating_spans):
if i < len(data):
data[i]["rating"] = rating.get_text(strip=True)
# Save to JSON
if data:
with open("target_reviews.json", "w", encoding="utf-8") as f:
json.dump(data, f, indent=4, ensure_ascii=False)
print(f"Saved {len(data)} reviews to target_reviews.json")
else:
print("No reviews found to save.")
print("Scraping complete.")
Want to dive deeper? Keep reading!
Scraping Target.com Reviews with Python and ScraperAPI (Proxy Mode)
In this section, we’ll build a Python script using Selenium, BeautifulSoup, and ScraperAPI in proxy mode to extract customer reviews from a product page. We’ll walk through loading the page, handling dynamic content like the “Show more” button, and saving the extracted reviews and ratings into a JSON file.
Prerequisites
Before we jump into the code, make sure you have the following installed:
- Python 3.7 or above. Due to compatibility errors between 3.12+ Python and undetected-chromedriver, we recommend sticking to Python 3.7-3.11. For a quick workaround on Python 3.12, you can pip install setuptools, which tends to temporarily fix the error.
undetected-chromedriver– helps bypass bot detectionselenium– to control the browserselenium-wire– for setting up proxies with Seleniumbeautifulsoup4– to parse HTML contentpython-dotenv– to load your credentials from your .env file- Google Chrome browser
- A ScraperAPI account and API key: Sign up for a free API key here if you don’t have one.
You can install the required Python packages using pip:
pip install undetected-chromedriver selenium selenium-wire beautifulsoup4 lxml python-dotenv
Step 1: Set Up Your Imports and API Key
Start by creating a new Python file, such as target_scraper.py. This will hold your scraping script.
Then, import the necessary libraries and set your ScraperAPI key and target URL:
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import undetected_chromedriver as uc
from bs4 import BeautifulSoup
import json
from time import sleep
In the root directory of your project, create a .env file. This file will contain your credentials (your API key in this case) and store them safely, so that you can be sure not making them public by mistake. You should also add this file to your .gitignore file to avoid pushing sensitive data to version control. Create it by running:
touch .env
In it, define your API key:
SCRAPERAPI_KEY= "YOUR_API_KEY" # Replace with your actual ScraperAPI key
Once your credentials are safely stored, we can import them in our Python file.
Also add the URL of the product page you want to scrape:
from dotenv import load_dotenv
import os
load_dotenv() # Loads the .env file
API_KEY = os.getenv("SCRAPERAPI_KEY")
target_url = "https://www.target.com/p/enclosed-cat-litter-box-xl-up-up/-/A-90310047?preselect=87059440#lnk=sametab"
Step 2: Configure ScraperAPI as a Proxy
To avoid getting blocked, we’ll use ScraperAPI in proxy mode. Here’s how to set that up and pass it into the Chrome driver.
proxy_url = f"http://scraperapi.render=true.country_code=us:{API_KEY}@proxy-server.scraperapi.com:8001"
options = Options()
options.add_argument("--headless") # Runs Chrome in headless mode (no browser window)
seleniumwire_options = {
'proxy': {
'http': proxy_url,
'https': proxy_url,
},
'verify_ssl': False,
}
driver = uc.Chrome(options=options, seleniumwire_options=seleniumwire_options)
This configures the driver to run headlessly (in the background) and use ScraperAPI’s rotating proxy for every request.
Step 3: Open the Target Product Page
Now that our browser is ready and using a proxy, we navigate to the Target product page:
print(f"Opening URL: {target_url}")
driver.get(target_url)
sleep(10)
We’re adding a sleep(10) here to give the page enough time to load all the dynamic content.
Step 4: Scroll and Click the “Show More” Button
Target hides some of the reviews behind a “Show more” button. If that button is available, we’ll scroll down and try clicking it up to three times.
print("Scrolling and checking for 'Show more' buttons...")
for i in range(3):
try:
driver.execute_script("window.scrollTo(0, 3200);")
show_more = WebDriverWait(driver, 5).until(
EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/div[2]/main/div/div[3]/div/div[17]/div/div[5]/button"))
)
show_more.click()
show_more.click()
print(f"'Show more' clicked ({i + 1})")
sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(3)
except:
print(f"No more 'Show more' button after {i} click(s).")
break
This loop mimics a real user by scrolling and clicking the button up to 3 times. It uses WebDriverWait and expected_conditions to wait for the button to become clickable.
Step 5: Wait for the Reviews to Load Fully
Before we parse the HTML, we must ensure the review list has fully rendered:
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "div[data-test='reviews-list']"))
)
except:
print("Review section may not be fully loaded.")
driver.quit()
exit()
sleep(3)
We use the presence_of_element_located to confirm that the main reviews container exists in the DOM. If it’s missing, we exit early.
Step 6: Parse the Reviews with BeautifulSoup
With the page fully loaded, we can parse it using BeautifulSoup and extract reviews and ratings.
soup = BeautifulSoup(driver.page_source, "lxml")
driver.quit() # Close the browser
data = []
review_divs = soup.find_all("div", attrs={"data-test": "review-card--text"})
rating_spans = soup.find_all("span", class_=["styles_ndsScreenReaderOnly__mcNC_", "styles_notFocusable__XkHOR"])
for review in review_divs:
review_text = review.get_text(strip=True)
data.append({"review": review_text})
for i, rating in enumerate(rating_spans):
if i < len(data):
data[i]["rating"] = rating.get_text(strip=True)
We’re extracting review texts from div tags with the data-test="review-card--text" attribute and star ratings from span tags containing specific class names. We then pair reviews and ratings by their order on the page to build a structured list of dictionaries.
Step 7: Save the Reviews to a JSON File
Finally, save the extracted data into a .json file.
if data:
with open("target_reviews.json", "w", encoding="utf-8") as f:
json.dump(data, f, indent=4, ensure_ascii=False)
print(f"Saved {len(data)} reviews to target_reviews.json")
else:
print("No reviews found to save.")
print("Scraping complete.")
The result is a JSON file like this:
[
{
"review": "I like everything about this property box. I have 2 cats and this box can handle boths",
"rating": ""
},
{
"review": "Large and sturdy. Easy to put together.",
"rating": ""
},
{
"review": "Perfect size. Even our 20+lbs chunky bum fits in it.",
"rating": "4.7 out of 5 stars with 972 reviews"
},
{
"review": "I love the size of this litter box. Plenty of space for my cat to stand up and move around inside it. I removed the flap and the opening is really easy for my cat to go in and out of. The lid locks on securely and I've had no issues with it sliding or moving as my cat goes in and out.",
"rating": "2 out of 5 stars with 1 ratings"
},
{
"review": "Sturdy litter box, tall enough to keep everything in the box (even for our tall male cat). The price is great",
"rating": "5 out of 5 stars with 1 ratings"
},
{
"review": "I am very pleased with this items. My cat loves it 🤩",
"rating": "5 out of 5 stars with 11 ratings"
},
{
"review": "Quality is good but disappointed in scratches the swing door has. Seems like they give you what's been tossed around at the store for pick up orders",
"rating": "4.4 out of 5 stars with 1437 ratings"
},
Wrapping Up
And that’s it! You now have a working scraper that pulls real customer reviews from Target.com using Python, Selenium, and ScraperAPI’s proxy mode.
This data can be helpful for sentiment analysis, market research, or just keeping tabs on how customers receive products. With the proper setup, you can expand this script to scrape multiple products, track changes over time, or even build your review analysis tool.
If you haven’t already, sign up for ScraperAPI and grab your free API key to start scraping without worrying about getting blocked. It’s an excellent tool for handling the behind-the-scenes complexity of proxies and IP rotation so you can focus on building.
Happy Scraping!