Skip to main content
Spider Cloud hosts your Scrapy project. You connect a git repository, we build it into an image, and your spiders run on our machines - with logs, items, stats, schedules and delivery attached to every run. The part you cannot get elsewhere: ScrapeUnblocker routing is built in. A spider that gets blocked on another hosting platform runs here without you writing a single line of proxy, header or fingerprint code. You choose per run how much of the crawl goes through the unblocking API - nothing, only the requests the spider asks for, or everything.
Open it in your dashboard: app.scrapeunblocker.com/dashboard/spiders. Spider Cloud is enabled per account - if the page says Not authorized, contact us to have it turned on.

​
What you need

A Scrapy project in git

A scrapy.cfg at the repository root, spiders under a spiders/ package, and a requirements.txt if you have dependencies beyond Scrapy.

A ScrapeUnblocker account

The key on your account is attached to each project automatically, so routed runs are billed to you and nobody else.
Nothing about the project has to change to run here. No SDK, no decorators, no platform-specific base class - the same repository you run with scrapy crawl locally is the one we build.

​
How it fits together

​
What runs your spider

Every job is a throwaway container with a strict shape, and a few consequences are worth knowing before your first run:
  • Your item pipelines can be switched off for a run. This is on by default, so trial-running a spider that normally writes to your production database touches nothing.
  • FEEDS has no effect. The container filesystem is discarded when the job ends, so feed exports would write into nothing. Items reach storage through the platform instead, and you read or deliver them from there.
  • The telnet console is always off.
  • Spiders written for older Scrapy still work. Scrapy 2.13 replaced start_requests() with async def start(), and by 2.17 the old hook is gone with no error - a spider written the old way would crawl nothing and exit successfully. We detect that at load time and adapt the spider, then say so in the log.
  • A job that fetched nothing is reported as failed, not as a green run with no data.

​
Spider Cloud vs the rest of ScrapeUnblocker

Use Spider Cloud

You already have (or want to write) a Scrapy project, and you want it hosted, scheduled and unblocked without maintaining machines.

Use the API

You are building an application and want to call getPageSource, serpApi or getImage from your own code, wherever it runs.

Use the No-code area

You want a dataset emailed to you and would rather not write a scraper at all.

Use the Scrapy integration

You have a Scrapy project that runs somewhere else and only want the unblocking part.

​
Next steps

Getting started

From nothing to your first successful run, in order.

ScrapeUnblocker routing

The three routing modes, what each one costs, and how a spider marks a request.