How to Scrape Redfin Property Data with Node.js

In this tutorial, we’ll show you how to extract property listings from Redfin using Node.js, automating the data collection process without getting blocked.

TL;DR: Full Redfin Scraper

For those in a hurry, here is the full Node.js code

	const axios = require('axios');
	const cheerio = require('cheerio');
	
	const REDFIN_PAGE_URL = 'https://www.redfin.com/city/5155/CO/Denver/apartments-for-rent';
	const API_URL = 'https://api.scraperapi.com';
	const API_KEY = '' // <--- Enter your API key here
	
	const webScraper = async () => {
	   console.log('Fetching data with ScraperAPI...');
	
	   const queryParams = new URLSearchParams({
		   api_key: API_KEY,
		   url: REDFIN_PAGE_URL,
		   country_code: 'us'
	   });
	
	   try {
		   const response = await axios.get(`${API_URL}?${queryParams.toString()}`);
	
		   const html = response.data;
	
		   const $ = cheerio.load(html);
		   const propertyList = [];
	
		   console.log('Extracting information from the HTML...');
	
		   $(".HomeCardContainer").each((_, el) => {
			   const price = $(el).find('.homecardV2Price span').text();
			   const beds = $(el).find('.HomeStatsV2 .stats:first-child').text();
			   const bath = $(el).find('.HomeStatsV2 .stats:nth-child(2)').text();
			   const space = $(el).find('.HomeStatsV2 .stats:last-child').text();
			   const address = $(el).find('.homeAddressV2 .fullAddress').text().substring(2);
			   const link = $(el).find('a:has(div.link-and-anchor)').attr('href');
			   const linkText = $(el).find('a:has(div.link-and-anchor)').text();
			   const contact = $(el).find('a.large-phone-cta').text();
	
			   if (!price) {
				   return
			   }
	
			   propertyList.push({
				   price,
				   address: address ?? linkText,
				   beds,
				   bath,
				   space,
				   link: link ? `https://www.redfin.com${link}` : null,
				   contact
			   });
		   });
	
		   console.log('JSON result:', propertyList);
	   } catch (error) {
		   console.log(error)
	   }
	};
	
	void webScraper();

Before running this code, install the dependencies and set your API key, which you can find in your ScraperAPI dashboard.

Don’t have an API key yet? Create a free ScraperAPI account and get 5,000 API credits with your 7-day free trial.

Scraping Redfin Property Listings

For this tutorial, we’ll write a scraper that finds properties for rent in the city of Denver, Colorado. For each property for sale, we will retrieve the following information:

• Address
• Price
• Number of beds
• Number of bath
• Surface
• Property link
• Contact phone number

The script will return the properties list extracted in JSON format so you can easily use it for other purposes.

Prerequisites

You must have these tools installed on your computer to follow this tutorial.

• Node.js 18+ and NPM
• Basic knowledge of JavaScript and Node.js API
• A ScraperAPI account; sign up and get 5,000 free API credits to get started

Step 1: Set Up Your Project

Let’s create a folder that will contain the source code of the Refin web scraper.

	mkdir redfin-scraper

Enter the folder and initialize a new Node.js project. The second command above will create a package.json file in the folder.

	cd redfin-scraper

	npm init -y	

Next, create a index.js file and add a simple JavaScript instruction inside.

	touch index.js

	echo "console.log('Hello world!');" > index.js	

Run the file index.js using the Node.js runtime.

	node index.js

This execution will print the message Hello world! in the terminal, verifying the installation was successful.

Step 2: Install the Dependencies

Before we can scrape Redfin pages, we must install two Node.js packages:

• Axios – to build the HTTP request (headers, body, query string parameters, etc…), send it to the ScraperAPI standard API, and download the HTML content.
• Cheerio – to extract the information from the HTML downloaded with the Axios request.

Run the command below to install these packages:

	npm install axios cheerio

Step 3: Inspect Redfin Result Page

Navigate to https://www.redfin.com. The property for sale is the default option; select the option “Rent” to target houses for rent. Type “Denver” in the search bar and press enter.

Once the property listings load, inspect the page to display the HTML structure and identify the DOM selector associated with the HTML tag wrapping the information we want to extract.

From the above picture, here are all the DOM selectors the Web scraper will target to extract the information of each property.

Be careful when writing the selector because a misspelling will prevent the script from retrieving the correct value.

Note: A good method to avoid errors when building your selectors is to try them with jQuery first. In the browser’s console, type your selector like $(".HomeCardContainer .homeAddressV2 .fullAddress"). if it returns the correct DOM element, then you are good to go.

Step 4: Scrape Redfin’s Property Page

To access the data, we need to send a get() request to download the page’s HTML content.

However, after sending just a couple of requests, Redfin will identify our bot and block all requests coming from our IP address.

To overcome this challege, we’ll use Axios to send our requests through ScraperAPI. This scraping API will rotate our IP and headers automatically when needed, handle CAPTCHAs, etc., allowing us to bypass Redfin’s anti-bot detection.

The request needs the following query parameters:

• The URL to scrape: it is the URL of the Redfin properties search page; you can copy it in the address bar of your browser.
• The API Key: to authenticate against the Scraping API and perform the scraping.

Note: Create a free ScraperAPI account to get your unique API key.

Pro Tip: Scraping Localized Redfin Data

In the real estate industry, location is one of the most critical factors for property buyers or renters. Redfin operates in the USA and Canada, and using geo-targeting in our web scraper helps gather precise data for areas of these two countries.

With the ScraperAPI’s API, you can specify the country code from where the request must come from. In the HTTP request query parameter, you must set the key country_code with the value us to indicate the request comes from the United States.

Note: Check out the Scraping API documentation to view all supported country codes.

Edit the index.js file to add the code below that builds the HTTP request, sends it, receives the response, and prints it in the terminal.

	const axios = require('axios');

	const REDFIN_PAGE_URL = 'https://www.redfin.com/city/5155/CO/Denver/apartments-for-rent';
	const API_URL = 'https://api.scraperapi.com';
	const API_KEY = ''
	
	const webScraper = async () => {
		console.log('Fetching data with ScraperAPI...');
	
		const queryParams = new URLSearchParams({
				api_key: API_KEY,
				url: REDFIN_PAGE_URL,
				country_code: 'us'
		});
	
		try {
				const response = await axios.get(`${API_URL}?${queryParams.toString()}`);
	
				const html = response.data;
	
				console.log("HTML content", html);
		} catch (error) {
				console.log(error)
		}
	};
	
	void webScraper();

Step 5: Parse Redfin’s Property Listings

Now that we have the HTML content of the page, let’s parse it with Cheerio to easily navigate through the DOM and extract all the information we want.

Cheerio provides functions to load HTML text, then navigate through the structure to extract information using the DOM selectors.

The code below goes through each element, extracts the information, and returns an array containing all the properties.

	const cheerio = require('cheerio');

	const $ = cheerio.load(html);
	const propertyList = [];
	
	console.log('Extract information from the HTML...');
	
	$(".HomeCardContainer").each((_, el) => {
	   const price = $(el).find('.homecardV2Price span').text();
	   const beds = $(el).find('.HomeStatsV2 .stats:first-child').text();
	   const bath = $(el).find('.HomeStatsV2 .stats:nth-child(2)').text();
	   const space = $(el).find('.HomeStatsV2 .stats:last-child').text();
	   const address = $(el).find('.homeAddressV2 .fullAddress').text().substring(2);
	   const link = $(el).find('a:has(div.link-and-anchor)').attr('href');
	   const linkText = $(el).find('a:has(div.link-and-anchor)').text();
	   const contact = $(el).find('a.large-phone-cta').text();
	
	   if (!price) {
		   return
	   }
	
	   propertyList.push({
		   price,
		   address: address ?? linkText,
		   beds,
		   bath,
		   space,
		   link: link ? `https://www.redfin.com${link}` : null,
		   contact
	   });
	});
	
	console.log('JSON result:', propertyList);

Note: At this point, we’ve built the same scraper shown in the TL;DR of this article.

Run the code with the command node index.js, and appreciate the result.