ScraperAPI has acquired Traject Data

Read the announcement
Contact Sales
Login

Real-time web scraping for your LangChain agents

Your agent’s page-fetch tool shouldn’t die on a 403. ScraperAPI’s LangChain integration adds web scraping to your agent in two lines of code. Get clean text, markdown, or structured JSON from JS-heavy, geo-targeted, or anti-bot-protected pages. Feed data straight into your agent or RAG pipeline, all with just two lines of code.

Start a free trial

No credit card required. 5,000 free API credits.

Up to 99.99% success rate

40M+ proxies

2 lines of code

What you can build with LangChain and ScraperAPI

Once your agent’s browse tool stops returning block pages, these are the kinds of LangChain agents builders ship.

Web research agent

Pull company data, job listings, and contact signals from SERP and the pages behind them. Your agent enriches records with structured data from each site while you focus on outreach.

RAG pipeline on live data

Scrape web pages for a LangChain RAG pipeline and load clean text into your vector database, so retrieval runs on current data instead of a stale crawl.

Lead-generation agent

Pull company data, job listings, and contact signals from directories and sites. Your agent enriches records with structured JSON while you focus on outreach.

Price and SERP monitor

Pull structured Google Search results and Amazon product data on a schedule, so your agent tracks prices and rankings without parsing pages.

Everything your LangChain agents need

Your LangChain agent needs to open any URL, get structured data back, and avoid running scraping infrastructure. The ScraperAPI tool delivers all three through native tool-calling.

A browse tool with real unblocking

  • Opens any URL: JS-rendered, geo-targeted, or anti-bot-protected
  • JS rendering and residential proxies are parameters, not a rewrite
  • 40M+ proxies, CAPTCHA handling, and up to a 99% success rate

Structured Google and Amazon, ready for reasoning

  • Google Search and Amazon product data as JSON fields
  • Clean text, markdown, or raw HTML for any other URL
  • Feed structured results straight into your chain or RAG pipeline

Native tool-calling, nothing to host

  • pip install langchain-scraperapi, then tools=[ScraperAPITool()]
  • Plugs into LangChain's own tool-calling. No scraping infrastructure to run or maintain, even when the target site changes

See It in Action

From basic chatbot to web scraping LangChain agent in under five minutes.

python snippet
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_scraperapi.tools import ScraperAPITool

# Add web access to your agent in two lines
tools = [ScraperAPITool()]
llm = ChatOpenAI(model="gpt-4o", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a job search assistant. Use ScraperAPITool to browse and read live job listings."),
    ("human", "{input}"),    MessagesPlaceholder(variable_name="agent_scratchpad"),
])
Without ScraperAPI x LangChain
  • Gets blocked by bot detection
  • Fails on JavaScript-heavy sites
  • Requires complex proxy setup
  • Returns raw pages to untangle
  • Hours of debugging
With ScraperAPI x LangChain
  • Opens any URL, even protected ones
  • Renders JavaScript-heavy pages
  • No infrastructure needed
  • Clean, LLM-optimized data
  • 2 lines of code
Start a free trial

Why 10,000+ developers choose ScraperAPI for LangChain

Your agent stops failing mid-run

When a chain dies on a 403 or summarizes a block page by accident, the output is wrong. ScraperAPI’s browse tool opens protected, JS-heavy pages so your agent keeps running.

Your workflows keep running, even on protected sites

An up to 99% success rate keeps your LangChain agents running on sites like Amazon and Google. When a site updates its anti-bot protection, ScraperAPI adapts automatically.

Structured data your agent can reason over

ScraperAPI returns clean markdown and structured JSON, so your agent gets the data rather than the page, and spends fewer tokens reasoning over it.

Two lines of code, nothing to host

Run pip install langchain-scraperapi, add tools=[ScraperAPITool()], and your agent has web access. No proxies to configure and no scraping infrastructure to maintain.

Pay only for data that comes back clean

No charges for failed requests. ScraperAPI handles IP rotation, JavaScript rendering, and CAPTCHA bypass behind the scenes, so you only pay for successful results.

Works with any LLM and the rest of your stack

The tools work with any LLM: OpenAI, Anthropic, Cohere, or local models. The same web access also runs under LlamaIndex, MCP, and the CLI.

Part of your AI stack

The LangChain integration runs on the same web access as the rest of ScraperAPI’s agent tools.

LlamaIndex

For RAG and indexing

Add web access to your LlamaIndex agents in two lines. Pull any URL as clean text or markdown, plus structured JSON from Amazon, Google, eBay, Walmart, and Redfin, ready to chunk, embed, and load into your vector store.

Learn more

MCP server

For MCP clients

Give any MCP client access to live web data. Pull Amazon and Walmart prices, SERP results, or read any public URL. Your agent gets the data, not the page.

Learn more

CLI

For the terminal

Reach any page from your terminal in one command: raw HTML, text, markdown, or parsed JSON. See the exact credit cost before you run it. Feed the clean output straight to an LLM, RAG pipeline, or training set.

Learn more

Frequently asked questions about data for AI agents

Run pip install langchain-scraperapi, then add tools=[ScraperAPITool()] to your agent. In two lines your LangChain agent can open any URL and search Google and Amazon, with proxies, rendering, and retries handled for you.

WebBaseLoader and other lightweight loaders fetch a URL directly, so they fail on JavaScript-heavy and anti-bot-protected pages and hand back raw HTML. ScraperAPI routes through 40M+ proxies with rendering and CAPTCHA handling, reaches pages a basic loader can’t, and returns clean text, markdown, or structured JSON.

Yes. Use ScraperAPI to scrape live web pages, get back clean markdown or structured JSON, and load it into your vector database so retrieval runs on current data. It’s a common pattern for scraping data for RAG and for building a LangChain RAG pipeline on the open web.

Yes. The same web access is available for LlamaIndex as well as LangChain. If you’re weighing LangChain vs LlamaIndex, ScraperAPI works the same way in both — open any URL and get LLM-ready output back.

Raw HTML, clean text, markdown, or structured JSON. Google Search and Amazon come back as structured fields; any other URL returns text, markdown, or HTML your agent can reason over directly.

Route requests through ScraperAPI instead of fetching pages directly. It rotates 40M+ proxies, renders JavaScript, and handles CAPTCHAs automatically, with render and premium available as parameters when a target site needs them.

No. ScraperAPI is a hosted web scraping API. You pip install the tool and call it — there are no proxies, headless browsers, or parsers to run or maintain, even when the target site changes.

Start with 5,000 free API credits to test your agent. After that, upgrade to a paid plan — and you only pay for successful requests.

Give your LangChain agent web access in two lines

Add the ScraperAPI tool to your LangChain project and pull clean, structured web data into your agents in minutes.

Start a free trial

No credit card required. 5,000 free API credits to start.