> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-lsapi-1779323176-451247e.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider integration

> Integrate with the Spider document loader using LangChain JavaScript.

[Spider](https://spider.cloud/?ref=langchainjs) is the [fastest](https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md#benchmark-results) crawler. It converts any website into pure HTML, markdown, metadata or text while enabling you to crawl with custom actions using AI.

## Overview

Spider allows you to use high performance proxies to prevent detection, caches AI actions, webhooks for crawling status, scheduled crawls etc...

This guide shows how to crawl/scrape a website using [Spider](https://spider.cloud/) and loading the LLM-ready documents with `SpiderLoader` in LanghChain.

## Setup

Get your own Spider API key on [spider.cloud](https://spider.cloud/).

## Usage

Here's an example of how to use the `SpiderLoader`:

Spider offers two scraping modes `scrape` and `crawl`. Scrape only gets the content of the URL provided while crawl gets the content of the URL provided and crawls deeper following subpages.

```typescript theme={null}
import { SpiderLoader } from "@langchain/community/document_loaders/web/spider";

const loader = new SpiderLoader({
  url: "https://spider.cloud", // The URL to scrape
  apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env.
  mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages
  // params: {
  //   // optional parameters based on Spider API docs
  //   // For API documentation, visit https://spider.cloud/docs/api
  // },
});

const docs = await loader.load();
```

### Additional parameters

See the [Spider documentation](https://spider.cloud/docs/api) for all the available `params`.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/javascript/integrations/document_loaders/web_loaders/spider.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>