Web Scraping Proxies Made Simple

Getting Started

Using ScraperAPI is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the url you want to scrape.

API Key & Authentication

ScraperAPI uses API keys to authenticate requests. To use the API you need to sign up for an account and include your unique API key in every request.

Making Requests

You can use the API to scrape web pages, API endpoints, images, documents, PDFs or other files just as you would any other URL. Note: there is a 2MB limit per request.

There are three ways in which you can send GET requests to ScraperAPI:

• Via our API endpoint http://api.scraperapi.com?
• Via our Python SDK
• Via our proxy port http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001

Choose whichever option best suits your scraping requirements.

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip"

import requests
payload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/ip'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url]
def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url, self.parse)

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true"

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url +'&render=true']
def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url +'&render=true', self.parse)

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true', 'country_code': 'us'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text
# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url +'&render=true' + ‘&country_code=true’]
def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url +'&render=true' + ‘&country_code=true’, self.parse)

pip install scraperapi-sdk

from scraper_api import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
result = client.get(url = 'http://httpbin.org/ip').text
print(result)
# Scrapy users can simply replace the urls in their start_urls and parse function
# Note for Scrapy, you should not use DOWNLOAD_DELAY and
# RANDOMIZE_DOWNLOAD_DELAY, these will lower your concurrency and are not
# needed with our API
# ...other scrapy setup code
start_urls =[client.scrapyGet(url = 'http://httpbin.org/ip')]
def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request(client.scrapyGet(url = 'http://httpbin.org/ip'), self.parse)

from scraper_api import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
#Requests Users
result = client.get(url = 'http://httpbin.org/ip', render=true).text
# Scrapy users
yield scrapy.Request(client.scrapyGet(url = 'http://httpbin.org/ip', render=true), self.parse)

from scraper_api import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
#Requests Users
result = client.get(url = 'http://httpbin.org/ip', render=true, country_code=true).text
# Scrapy users
yield scrapy.Request(client.scrapyGet(url = 'http://httpbin.org/ip', render=true, country_code=true), self.parse)

"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"

import requests
proxies = {
  "http": "http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"
}
r = requests.get('http://httpbin.org/ip', proxies=proxies, verify=False)
print(r.text)
# Scrapy users can likewise simply pass their API key in headers.
# NB: Scrapy skips SSL verification by default.
# ...other scrapy setup code
start_urls = ['http://httpbin.org/ip']
meta = {
  "proxy": "http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"
}
def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request(url, callback=self.parse, headers=headers, meta=meta)

"http://scraperapi.render=true:APIKEY@proxy-server.scraperapi.com:8001"

"http://scraperapi.render=true.country_code=us:APIKEY@proxy-server.scraperapi.com:8001"

Making POST/PUT Requests

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/anything'}

r = requests.post('http://api.scraperapi.com', params=payload, data={'foo': 'bar'})
print(r.text)

API Status Codes

The API will return a specific status code after every request depending on whether the request was successful, failed or some other error occurred. ScraperAPI will retry failed requests for up to 60 seconds to try and get a successful response from the target url before responding with a 500 error indicating a failed request.

Note: To avoid timing out your request before the API has had a chance to complete all retries, remember to set your timeout to 60 seconds.

In cases where a request fails after 60 seconds of retying, you will not be charged for the unsuccessful request (you are only charged for successful requests, 200 and 404 status codes).

Errors will inevitably occur so on your end make sure you catch these errors. You can configure your code to immediately retry the request and in most cases it will be successful. If a request is consistently failing then check your request to make sure that it is configured correctly. Or if you are consistently getting bans messages from an anti-bot then create a ticket with our support team and we will try to bypass this anti-bot for you.

If you receive a successful 200 status code response from the API but the response contains a CAPTCHA then contact our support team and they will add it to our CAPTCHA detection database. Once included in our CAPTCHA database the API will treat it as a ban in future and automatically retry the request.

Below are the possible status code you will receive:

Customise API Functionality

ScraperAPI enables you to customise the APIs functionality by adding additional parameters to your requests. The API will accept the following parameters:

Rendering Javascript

If you are crawling a page that requires you to render the javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser. To render javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is only available on the Business and Enterprise plans.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text

# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + '&render=true']

def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url + '&render=true', self.parse)

Custom Headers

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

import requests
url = 'http://httpbin.org/anything'
headers = {
  'Accept': 'application/json'
  'X-MyHeader': '123',
}
payload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/ip', 'keep_headers': 'true'}
r = requests.get('http://api.scraperapi.com', params=payload, headers=headers)
print r.text

 # Scrapy users can simply replace the urls in their start_urls and parse function in their parse function
 # ...other scrapy setup code
headers = {
  'Accept': 'application/json'
  'X-MyHeader': '123',
}
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + '&keep_headers=true']

def parse(self, response):
# ...your parsing logic here
  scraper_url = 'http://api.scraperapi.com/?api_key=APIKEY&url=' + url + '&keep_headers=true'
  yield scrapy.Request(scraper_url, self.parse, headers=headers)

Sessions

To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The value of session can be any integer. Sessions expire 15 minutes after the last usage.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'session_number': '123'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text

# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + '&session_number=123']

Geographic Location

Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the APIs geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.

To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.

For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.

Business and Enterprise Plan users can geotarget their requests to the following 12 countries (Startup Plan can only use US geotargeting) by using the country_code in their request:

Other countries are available to Enterprise customers upon request.

• API Request
• SDK
• Proxy Mode

import requests
                                                        payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'country_code': 'us'}
                                                        r = requests.get('http://api.scraperapi.com', params=payload)
                                                        print r.text

                                                        # Scrapy users can simply replace the urls in their start_urls and parse function
                                                        # ...other scrapy setup code
                                                        start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + 'country_code=us']

                                                        def parse(self, response):
                                                        # ...your parsing logic here
                                                        yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url + 'country_code=us', self.parse)

Premium Residential/Mobile Proxy Pools

Our standard proxy pools include millions of proxies from over a dozen ISPs, and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is only available to users on the Business plan or higher.

Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API calls against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API calls against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.

We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API calls against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'premium': 'true'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text

# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + 'premium=true']

def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url + 'premium=true', self.parse)

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

• Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows or Linux) user agent. Note: This is the default behaviour. Not setting the parameter will have the same effect.
• Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.

Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'device_type': 'mobile'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text

# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + 'device_type=mobile']

def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url + 'device_type=mobile', self.parse)

Autoparse

For select websites the API will parse all the valuable data in the HTML response and return it in JSON format. To enable this feature, simply add autoparse=true to your request and the API will parse the data for you. Currently, this feature works with Amazon, Google Search and Google Shopping.

• API Request
• SDK
• Proxy Mode

import requests
payload = {'api_key': 'APIKEY', 'url':'https://www.amazon.com/dp/B07V1PHM66', 'autoparse': 'true'}
r = requests.get('http://api.scraperapi.com', params=payload)
print r.text

# Scrapy users can simply replace the urls in their start_urls and parse function
# ...other scrapy setup code
start_urls = ['http://api.scraperapi.com?api_key=APIKEY&url=' + url + 'autoparse=true']

def parse(self, response):
  # ...your parsing logic here
  yield scrapy.Request('http://api.scraperapi.com/?api_key=APIKEY&url=' + url + 'autoparse=true', self.parse)

Account Information

If you would like to monitor your account usage and limits programmatically (how many concurrent requests you’re using, how many requests you’ve made, etc.) you may use the /account endpoint, which returns JSON.

Note: the requestCount and failedRequestCount numbers only refresh once every 15 seconds, while the concurrentRequests number is available in realtime.

• API Request
• SDK

import requests
payload = {'api_key': 'APIKEY'}
r = requests.get('http://api.scraperapi.com/account', params=payload)
print r.text

Getting Started

Using ScraperAPI is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the url you want to scrape.

API Key & Authentication

ScraperAPI uses API keys to authenticate requests. To use the API you need to sign up for an account and include your unique API key in every request.

Making Requests

You can use the API to scrape web pages, API endpoints, images, documents, PDFs or other files just as you would any other URL. Note: there is a 2MB limit per request.

There are three ways in which you can send GET requests to ScraperAPI:

• Via our API endpoint http://api.scraperapi.com?
• Via our NodeJS SDK
• Via our proxy port http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip"

const request = require('request-promise');
request(`http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip`)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true"

const request = require('request-promise');
request(`http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&render=true`)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"

const request = require('request-promise');
request(`http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us`)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

npm install scraperapi-sdk --save

// remember to install the library: npm install scraperapi-sdk --save
const scraperapiClient = require('scraperapi-sdk')('APIKEY')
scraperapiClient.get('http://httpbin.org/ip')
.then(response => {
 console.log(response)
})
.catch(error => {
 console.log(error)
})

// remember to install the library: npm install scraperapi-sdk --save
const scraperapiClient = require('scraperapi-sdk')('APIKEY')
scraperapiClient.get('http://httpbin.org/ip', {render: true})
.then(response => {
 console.log(response)
})
.catch(error => {
 console.log(error)
})

// remember to install the library: npm install scraperapi-sdk --save
const scraperapiClient = require('scraperapi-sdk')('APIKEY')
const params = {
 render: true,
 country_code: 'us'
}
scraperapiClient.get('http://httpbin.org/ip', params)
.then(response => {
 console.log(response)
})
.catch(error => {
 console.log(error)
})

"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"

const request = require('request-promise');
options = {
 method: 'GET',
 url: 'http://httpbin.org/ip',
 proxy:`http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001`
}
request(options)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

"http://scraperapi.render=true:APIKEY@proxy-server.scraperapi.com:8001"

"http://scraperapi.render=true.country_code=us:APIKEY@proxy-server.scraperapi.com:8001"

Making POST/PUT Requests

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

// Replace POST with PUT to send a PUT request instead
options = {
 method: 'POST',
 url: 'http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/post',
 body: JSON.stringify({foo: 'bar'}),
 headers: {
 'Content-Type': 'application/json',
 },
}

request(options)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

//For form data
options = {
 method: 'POST',
 url: http://api.scraperapi.com/?api_key=${api_key}&url=http://httpbin.org/post,
 form: {foo: 'bar'},
 headers: {
 'Content-Type': 'application/x-www-form-urlencoded',
 },
}

request(options)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

API Status Codes

The API will return a specific status code after every request depending on whether the request was successful, failed or some other error occurred. ScraperAPI will retry failed requests for up to 60 seconds to try and get a successful response from the target url before responding with a 500 error indicating a failed request.

Note: To avoid timing out your request before the API has had a chance to complete all retries, remember to set your timeout to 60 seconds.

In cases where a request fails after 60 seconds of retying, you will not be charged for the unsuccessful request (you are only charged for successful requests, 200 and 404 status codes).

Errors will inevitably occur so on your end make sure you catch these errors. You can configure your code to immediately retry the request and in most cases it will be successful. If a request is consistently failing then check your request to make sure that it is configured correctly. Or if you are consistently getting bans messages from an anti-bot then create a ticket with our support team and we will try to bypass this anti-bot for you.

If you receive a successful 200 status code response from the API but the response contains a CAPTCHA then contact our support team and they will add it to our CAPTCHA detection database. Once included in our CAPTCHA database the API will treat it as a ban in future and automatically retry the request.

Below are the possible status code you will receive:

Customise API Functionality

ScraperAPI enables you to customise the APIs functionality by adding additional parameters to your requests. The API will accept the following parameters:

Rendering Javascript

If you are crawling a page that requires you to render the javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser. To render javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is only available on the Business and Enterprise plans.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&render=true')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Custom Headers

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

options = {
 method: 'GET',
 url: 'http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/anything&keep_headers=true',
 headers: {
 Accept: 'application/json',
 'X-MyHeader': '123',
 },
}
request(options)
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Sessions

To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The value of session can be any integer. Sessions expire 15 minutes after the last usage.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&session_number=123')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Geographic Location

Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the APIs geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.

To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.

For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.

Business and Enterprise Plan users can geotarget their requests to the following 12 countries (Startup Plan can only use US geotargeting) by using the country_code in their request:

Other countries are available to Enterprise customers upon request.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&country_code=us')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Premium Residential/Mobile Proxy Pools

Our standard proxy pools include millions of proxies from over a dozen ISPs, and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is only available to users on the Business plan or higher.

Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API calls against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API calls against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.

We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API calls against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&premium=true')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

• Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows or Linux) user agent. Note: This is the default behaviour. Not setting the parameter will have the same effect.
• Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.

Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&device_type=mobile')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Autoparse

For select websites the API will parse all the valuable data in the HTML response and return it in JSON format. To enable this feature, simply add autoparse=true to your request and the API will parse the data for you. Currently, this feature works with Amazon, Google Search and Google Shopping.

• API Request
• SDK
• Proxy Mode

const request = require('request-promise');

request('http://api.scraperapi.com/?api_key=APIKEY&url=https://www.amazon.com/dp/B07V1PHM66&autoparse=true')
.then(response => {
 console.log(response)
})
.catch(error => {
 console.log(error)
})

Account Information

If you would like to monitor your account usage and limits programmatically (how many concurrent requests you’re using, how many requests you’ve made, etc.) you may use the /account endpoint, which returns JSON.

Note: the requestCount and failedRequestCount numbers only refresh once every 15 seconds, while the concurrentRequests number is available in realtime.

• API Request
• SDK

const request = require('request-promise');

request('http://api.scraperapi.com/account?api_key=APIKEY')
 .then(response => {
 console.log(response)
 })
 .catch(error => {
 console.log(error)
 })

Getting Started

Using ScraperAPI is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the url you want to scrape.

API Key & Authentication

ScraperAPI uses API keys to authenticate requests. To use the API you need to sign up for an account and include your unique API key in every request.

Making Requests

You can use the API to scrape web pages, API endpoints, images, documents, PDFs or other files just as you would any other URL. Note: there is a 2MB limit per request.

There are three ways in which you can send GET requests to ScraperAPI:

• Via our API endpoint http://api.scraperapi.com?
• Via our PHP SDK
• Via our proxy port http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip"

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true"

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

gem install scraperapi

# remember to install the library: composer require scraperapi/sdk
<?php
 $client = new ScraperAPI\Client("APIKEY");
 $result = $client->get("http://httpbin.org/ip")->raw_body;
 print($result);

# remember to install the library: composer require scraperapi/sdk
<?php
 $client = new ScraperAPI\Client("APIKEY");
 $result = $client->get("http://httpbin.org/ip", ["render" => true])->raw_body;
 print($result);

# remember to install the library: composer require scraperapi/sdk
<?php
 $client = new ScraperAPI\Client("APIKEY");
 $result = $client->get("http://httpbin.org/ip", ["render" => true], [“country_code” => “us”])->raw_body;
 print($result);

"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"

<?php
 $ch = curl_init();
 curl_setopt($ch, CURLOPT_URL, "http://httpbin.org/ip");
 curl_setopt($ch, CURLOPT_PROXY, "http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001");
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
 curl_setopt($ch, CURLOPT_HEADER, FALSE);
 curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
 curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
 $response = curl_exec($ch);
 curl_close($ch);
 var_dump($response);

"http://scraperapi.render=true:APIKEY@proxy-server.scraperapi.com:8001"

"http://scraperapi.render=true.country_code=us:APIKEY@proxy-server.scraperapi.com:8001"

Making POST/PUT Requests

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/anything";

# POST/PUT Requests
$postData = ["foo" => "bar"];
$postData = json_encode($postData);
$headers = [
 "Content-Type: application/json"
];

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData); //Post Fields
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$response = curl_exec($ch);
curl_close($ch);

print_r($response);


#Form POST Request
$postData = ["foo" => "bar"];
$postData = json_encode($postData);
$headers = [
 'Content-Type: application/x-www-form-urlencoded; charset=utf-8',
];

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$response = curl_exec($ch);
curl_close($ch);

print_r($response);

API Status Codes

The API will return a specific status code after every request depending on whether the request was successful, failed or some other error occurred. ScraperAPI will retry failed requests for up to 60 seconds to try and get a successful response from the target url before responding with a 500 error indicating a failed request.

Note: To avoid timing out your request before the API has had a chance to complete all retries, remember to set your timeout to 60 seconds.

In cases where a request fails after 60 seconds of retying, you will not be charged for the unsuccessful request (you are only charged for successful requests, 200 and 404 status codes).

Errors will inevitably occur so on your end make sure you catch these errors. You can configure your code to immediately retry the request and in most cases it will be successful. If a request is consistently failing then check your request to make sure that it is configured correctly. Or if you are consistently getting bans messages from an anti-bot then create a ticket with our support team and we will try to bypass this anti-bot for you.

If you receive a successful 200 status code response from the API but the response contains a CAPTCHA then contact our support team and they will add it to our CAPTCHA detection database. Once included in our CAPTCHA database the API will treat it as a ban in future and automatically retry the request.

Below are the possible status code you will receive:

Customise API Functionality

ScraperAPI enables you to customise the APIs functionality by adding additional parameters to your requests. The API will accept the following parameters:

Rendering Javascript

If you are crawling a page that requires you to render the javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser. To render javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is only available on the Business and Enterprise plans.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Custom Headers

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&keep_headers=true";
$headerArray = array(
 "Content-Type: application/json",
 "X-MyHeader: 123"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headerArray);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Sessions

To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The value of session can be any integer. Sessions expire 15 minutes after the last usage.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&session_number=123";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Geographic Location

Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the APIs geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.

To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.

For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.

Business and Enterprise Plan users can geotarget their requests to the following 12 countries (Startup Plan can only use US geotargeting) by using the country_code in their request:

Other countries are available to Enterprise customers upon request.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&country_code=us";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Premium Residential/Mobile Proxy Pools

Our standard proxy pools include millions of proxies from over a dozen ISPs, and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is only available to users on the Business plan or higher.

Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API calls against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API calls against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.

We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API calls against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&premium=true";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

• Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows or Linux) user agent. Note: This is the default behaviour. Not setting the parameter will have the same effect.
• Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.

Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&device_type=mobile";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Autoparse

For select websites the API will parse all the valuable data in the HTML response and return it in JSON format. To enable this feature, simply add autoparse=true to your request and the API will parse the data for you. Currently, this feature works with Amazon, Google Search and Google Shopping.

• API Request
• SDK
• Proxy Mode

<?php
$url = "http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&autoparse=true";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Account Information

If you would like to monitor your account usage and limits programmatically (how many concurrent requests you’re using, how many requests you’ve made, etc.) you may use the /account endpoint, which returns JSON.

Note: the requestCount and failedRequestCount numbers only refresh once every 15 seconds, while the concurrentRequests number is available in realtime.

• API Request
• SDK

<?php
$url = "http://api.scraperapi.com/account?api_key=APIKEY";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
print_r($response);

Getting Started

Using ScraperAPI is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the url you want to scrape.

API Key & Authentication

ScraperAPI uses API keys to authenticate requests. To use the API you need to sign up for an account and include your unique API key in every request.

Making Requests

You can use the API to scrape web pages, API endpoints, images, documents, PDFs or other files just as you would any other URL. Note: there is a 2MB limit per request.

There are three ways in which you can send GET requests to ScraperAPI:

• Via our API endpoint http://api.scraperapi.com?
• Via our Ruby SDK
• Via our proxy port http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip"

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip"
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true"

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :render => true
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :render => true,
 :country_code => “us”
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

gem install scraperapi

# remember to install the library: gem install scraperapi
require "scraper_api"
client = ScraperAPI::Client.new("APIKEY")
result = client.get("http://httpbin.org/ip").raw_body
puts result

# remember to install the library: gem install scraperapi
require "scraper_api"
client = ScraperAPI::Client.new("APIKEY")
result = client.get("http://httpbin.org/ip", render: true).raw_body
puts result

# remember to install the library: gem install scraperapi
require "scraper_api"
client = ScraperAPI::Client.new("APIKEY")
result = client.get("http://httpbin.org/ip", render: true, country_code: “us”).raw_body
puts result

"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"

require 'httparty'
HTTParty::Basement.default_options.update(verify: false)
response = HTTParty.get('http://httpbin.org/ip', {
 http_proxyaddr: "proxy-server.scraperapi.com",
 http_proxyport: "8001",
 http_proxyuser: "scraperapi",
 http_proxypass: "APIKEY"
})
results = response.body
puts results 

"http://scraperapi.render=true:APIKEY@proxy-server.scraperapi.com:8001"

"http://scraperapi.render=true.country_code=us:APIKEY@proxy-server.scraperapi.com:8001"

Making POST/PUT Requests

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'

## Replace POST with PUT to send a PUT request instead
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/anything"
}

uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

website_content = Net::HTTP.post(uri, { "foo" => "bar"}.to_json, "Content-Type" => "application/json")
print(website_content.body)

## For form data
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/anything"
}

uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

req = Net::HTTP::Post.new(uri)
req.set_form_data('foo' => 'bar')

website_content = Net::HTTP.start(uri.hostname, uri.port) {|http|
 http.request(req)
}

print(website_content.body)

API Status Codes

The API will return a specific status code after every request depending on whether the request was successful, failed or some other error occurred. ScraperAPI will retry failed requests for up to 60 seconds to try and get a successful response from the target url before responding with a 500 error indicating a failed request.

Note: To avoid timing out your request before the API has had a chance to complete all retries, remember to set your timeout to 60 seconds.

In cases where a request fails after 60 seconds of retying, you will not be charged for the unsuccessful request (you are only charged for successful requests, 200 and 404 status codes).

Errors will inevitably occur so on your end make sure you catch these errors. You can configure your code to immediately retry the request and in most cases it will be successful. If a request is consistently failing then check your request to make sure that it is configured correctly. Or if you are consistently getting bans messages from an anti-bot then create a ticket with our support team and we will try to bypass this anti-bot for you.

If you receive a successful 200 status code response from the API but the response contains a CAPTCHA then contact our support team and they will add it to our CAPTCHA detection database. Once included in our CAPTCHA database the API will treat it as a ban in future and automatically retry the request.

Below are the possible status code you will receive:

Customise API Functionality

ScraperAPI enables you to customise the APIs functionality by adding additional parameters to your requests. The API will accept the following parameters:

Rendering Javascript

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Do not use this feature in order to avoid blocks. We handle header rotation and optimisation internally so letting us set the headers internally will increase your success rates.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/anything",
 :keep_headers => true
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
req = Net::HTTP::Get.new(uri)
req['Accept'] = 'application/json'
req['X-MyHeader'] = '123'
website_content = Net::HTTP.start(uri.hostname, uri.port) {|http|
 http.request(req)
}
print(website_content.body)

Custom Headers

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/anything",
 :keep_headers => true
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
req = Net::HTTP::Get.new(uri)
req['Accept'] = 'application/json'
req['X-MyHeader'] = '123'
website_content = Net::HTTP.start(uri.hostname, uri.port) {|http|
 http.request(req)
}
print(website_content.body)

Sessions

To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The value of session can be any integer. Sessions expire 15 minutes after the last usage.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :session_number => 123
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Geographic Location

Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the APIs geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.

To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.

For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.

Business and Enterprise Plan users can geotarget their requests to the following 12 countries (Startup Plan can only use US geotargeting) by using the country_code in their request:

Other countries are available to Enterprise customers upon request.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :country_code => “us”
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Premium Residential/Mobile Proxy Pools

Our standard proxy pools include millions of proxies from over a dozen ISPs, and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is only available to users on the Business plan or higher.

Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API calls against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API calls against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.

We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API calls against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :premium => true
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

• Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows or Linux) user agent. Note: This is the default behaviour. Not setting the parameter will have the same effect.
• Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.

Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :device_type => “mobile”
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Autoparse

For select websites the API will parse all the valuable data in the HTML response and return it in JSON format. To enable this feature, simply add autoparse=true to your request and the API will parse the data for you. Currently, this feature works with Amazon, Google Search and Google Shopping.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY",
 :url => "http://httpbin.org/ip",
 :autoparse => true
}
uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Account Information

If you would like to monitor your account usage and limits programmatically (how many concurrent requests you’re using, how many requests you’ve made, etc.) you may use the /account endpoint, which returns JSON.

Note: the requestCount and failedRequestCount numbers only refresh once every 15 seconds, while the concurrentRequests number is available in realtime.

• API Request
• SDK
• Proxy Mode

require 'net/http'
require 'json'
params = {
 :api_key => "APIKEY"
}
uri = URI('http://api.scraperapi.com/account')
uri.query = URI.encode_www_form(params)
website_content = Net::HTTP.get(uri)
print(website_content)

Getting Started

Using ScraperAPI is easy. Just send the URL you would like to scrape to the API along with your API key and the API will return the HTML response from the url you want to scrape.

API Key & Authentication

ScraperAPI uses API keys to authenticate requests. To use the API you need to sign up for an account and include your unique API key in every request.

Making Requests

You can use the API to scrape web pages, API endpoints, images, documents, PDFs or other files just as you would any other URL. Note: there is a 2MB limit per request.

There are three ways in which you can send GET requests to ScraperAPI:

• Via our API endpoint http://api.scraperapi.com?
• Via one of our SDKs (only available for some programing languages)
• Via our proxy port http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip"

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true"

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&render=true";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

"http://api.scraperapi.com?api_key=APIKEY&url=http://httpbin.org/ip&render=true&country_code=us"

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&render=true&country_code=us";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

install the library: https://search.maven.org/artifact/com.scraperapi/sdk/1.0

// remember to install the library: https://search.maven.org/artifact/com.scraperapi/sdk/1.0
import com.scraperapi
ScraperApiClient client = new ScraperApiClient("APIKEY");
 client.get("http://httpbin.org/ip")
 .result();

ScraperApiClient client = new ScraperApiClient("APIKEY");
 client.get("http://httpbin.org/ip")
 .render(true)
 .result();

ScraperApiClient client = new ScraperApiClient("APIKEY");
 client.get("http://httpbin.org/ip")
 .render(true)
 .premium(true)
 .result();

"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"

try {
 String apiKey = "APIKEY";
 String proxy = "http://scraperapi:" + apiKey + "@proxy-server.scraperapi.com";
 URL server = new URL("https://httpbin.org/ip");
 Properties systemProperties = System.getProperties();
 systemProperties.setProperty("http.proxyHost", proxy);
 systemProperties.setProperty("http.proxyPort", "8001");
 HttpURLConnection httpURLConnection = (HttpURLConnection) server.openConnection();
 httpURLConnection.connect();
 String readLine = null;
 int responseCode = httpURLConnection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(httpURLConnection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

"http://scraperapi.render=true:APIKEY@proxy-server.scraperapi.com:8001"

"http://scraperapi.render=true.country_code=us:APIKEY@proxy-server.scraperapi.com:8001"

Making POST/PUT Requests

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "YOURAPIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

API Status Codes

The API will return a specific status code after every request depending on whether the request was successful, failed or some other error occurred. ScraperAPI will retry failed requests for up to 60 seconds to try and get a successful response from the target url before responding with a 500 error indicating a failed request.

Note: To avoid timing out your request before the API has had a chance to complete all retries, remember to set your timeout to 60 seconds.

In cases where a request fails after 60 seconds of retying, you will not be charged for the unsuccessful request (you are only charged for successful requests, 200 and 404 status codes).

Errors will inevitably occur so on your end make sure you catch these errors. You can configure your code to immediately retry the request and in most cases it will be successful. If a request is consistently failing then check your request to make sure that it is configured correctly. Or if you are consistently getting bans messages from an anti-bot then create a ticket with our support team and we will try to bypass this anti-bot for you.

If you receive a successful 200 status code response from the API but the response contains a CAPTCHA then contact our support team and they will add it to our CAPTCHA detection database. Once included in our CAPTCHA database the API will treat it as a ban in future and automatically retry the request.

Below are the possible status code you will receive:

Customise API Functionality

ScraperAPI enables you to customise the APIs functionality by adding additional parameters to your requests. The API will accept the following parameters:

Rendering Javascript

If you are crawling a page that requires you to render the javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser. To render javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is only available on the Business and Enterprise plans.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&render=true";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

Custom Headers

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

Note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you need to get results for mobile devices, use the device_type parameter to set the user agent header for your, instead of setting your own.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/anything&keep_headers=true";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection httpURLConnection = (HttpURLConnection) urlForGetRequest.openConnection();
 httpURLConnection.setRequestProperty("Content-Type", "application/json");
 httpURLConnection.setRequestProperty("X-MyHeader", "123");
 httpURLConnection.setRequestMethod("GET");
 int responseCode = httpURLConnection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(httpURLConnection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

Sessions

To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The value of session can be any integer. Sessions expire 15 minutes after the last usage.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&session_number=123";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

Geographic Location

Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the APIs geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.

To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.

For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.

Business and Enterprise Plan users can geotarget their requests to the following 12 countries (Startup Plan can only use US geotargeting) by using the country_code in their request:

Other countries are available to Enterprise customers upon request.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&country_code=us";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

Premium Residential/Mobile Proxy Pools

Our standard proxy pools include millions of proxies from over a dozen ISPs, and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is only available to users on the Business plan or higher.

Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API calls against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API calls against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.

We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API calls against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&premium=true";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }

} catch (Exception ex) {
 ex.printStackTrace();
}

Device Type

If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.

• Set device_type=desktop to have the API set a desktop (e.g. iOS, Windows or Linux) user agent. Note: This is the default behaviour. Not setting the parameter will have the same effect.
• Set device_type=mobile to have the API set a mobile (e.g. iPhone or Android) user agent.

Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&device_type=mobile";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }

} catch (Exception ex) {
 ex.printStackTrace();
}

Autoparse

For select websites the API will parse all the valuable data in the HTML response and return it in JSON format. To enable this feature, simply add autoparse=true to your request and the API will parse the data for you. Currently, this feature works with Amazon, Google Search and Google Shopping.

• API Request
• SDK
• Proxy Mode

try {
 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/ip&autoparse=true";
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }
} catch (Exception ex) {
 ex.printStackTrace();
}

Account Information

If you would like to monitor your account usage and limits programmatically (how many concurrent requests you’re using, how many requests you’ve made, etc.) you may use the /account endpoint, which returns JSON.

Note: the requestCount and failedRequestCount numbers only refresh once every 15 seconds, while the concurrentRequests number is available in realtime.

• API Request

try {

 String apiKey = "APIKEY";
 String url = "http://api.scraperapi.com/account?api_key=" + apiKey;
 URL urlForGetRequest = new URL(url);
 String readLine = null;
 HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
 conection.setRequestMethod("GET");
 int responseCode = conection.getResponseCode();
 if (responseCode == HttpURLConnection.HTTP_OK) {
 BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
 StringBuffer response = new StringBuffer();
 while ((readLine = in.readLine()) != null) {
 response.append(readLine);
 }
 in.close();
 System.out.println(response.toString());
 } else {
 throw new Exception("Error in API Call");
 }

} catch (Exception ex) {
 ex.printStackTrace();
}