How to Scrape G2 Reviews Using Python

Step 2: Sending a Request and Parsing the Response

We create a payload object that includes our API key and the G2 URL we wish to scrape. This payload is then used to send a get() request to our Scraping API, which will handle the complexities of web scraping such as smart IP and header rotation, CAPTCHA handling, and more, using machine learning and statistical analysis.

	payload = {"api_key": API_KEY, "url": url}
	html = requests.get("https://api.scraperapi.com", params=payload)

Then, we can use BeautifulSoup to parse the HTML response and store it as a soup object – allowing us to then navigate the parsed tree using CSS selectors.

	soup = BeautifulSoup(html.text, "lxml")

Step 3: Extracting G2 Software Reviews

This step involves initializing a dictionary to store the results and extracting the product name, number of reviews, and user review data from the parsed HTML response.

We start by creating a dictionary called results to store the data we’ll extract.

	results = {"product_name": "", "number_of_reviews": "", "reviews": []}

Here, product_name and number_of_reviews are strings that will hold the name of the product and the total number of reviews, respectively, while reviews is a list containing dictionaries, each representing an individual review.

Once we have our dictionary ready, we extract the product name from the parsed HTML.

We use BeautifulSoup’s select_one() method to select the first element that matches the CSS selector .product-head__title a. We then get the text of this element and assign it to results['product_name'].

	product_name_element = soup.select_one(".product-head__title a")
	results["product_name"] = product_name_element.get_text(strip=True) if product_name_element else "Product name not found"

Similarly, we extract the number of reviews by selecting the element with the CSS selector li.list--piped__li and getting its text.

	reviews_element = soup.find("li", {"class": "list--piped__li"})
	results["number_of_reviews"] = reviews_element.get_text(strip=True) if reviews_element else "Number of reviews not found"

After that, we loop through all the reviews on the page. Each review is selected using the selector .paper.paper--white.paper--box. We extract the username, summary, full review, date, URL, and rating for each review, then proceed to store this information in a new dictionary called review_data.

Extracting the Username

The username is typically found within an anchor (a) tag. To extract it, we look for the anchor tag with a class that indicates it contains the username.

For example, if the class name is link--header-color, we then use BeautifulSoup to find this a tag and then get its text content.

	username_element = review.find("a", {"class": "link--header-color"})
	review_data["username"] = username_element.get_text(strip=True) if username_element else "No username found"

Extracting the Review Summary

The review summary is found within an h3 heading with a class set to m-0 l2.

	summary_element = review.find("h3", {"class": "m-0 l2"})
	review_data["summary"] = summary_element.get_text(strip=True).replace('"', "") if summary_element else "No summary found"

Extracting the Review Body

The full text of the review is usually contained within a div with an itemprop attribute set to reviewBody. To extract the review text, we find this div and retrieve its text content.

	user_review_element = review.find("div", itemprop="reviewBody")
	review_data["review"] = user_review_element.get_text(strip=True) if user_review_element else "No review found"

Extracting the Review Date

The date of the review is often located within a time element.

	review_date_element = review.find("time")
	review_data["date"] = review_date_element.get_text(strip=True) if review_date_element else "No date found"

Extracting the Review URL

To extract the URL of each review, we look for an anchor tag that contains the href attribute. Specifically, we are interested in the a tags with the class pjax, as this is the class used by G2 for the links to the individual reviews.

Here’s how we can include this in our extraction process:

	review_url_element = review.find("a", {"class": "pjax"})
	review_data["url"] = review_url_element["href"] if review_url_element else "No URL found"

Extracting the Rating

Ratings are found within a meta tag with an attribute itemprop set to ratingValue. We search for this tag and extract its content attribute to get the rating value.

	rating_element = review.find("meta", itemprop="ratingValue")
	review_data["rating"] = rating_element["content"] if rating_element else "No rating found"

Each piece of data is then stored in a dictionary called review_data. After extracting the data for each review, we append review_data to results['reviews'].

	results["reviews"].append(review_data)

By the end of this step, the results will be a dictionary containing the product name, number of reviews, and a list of dictionaries, each representing an individual review.