Jobs

A job is one run of one deploy. It carries its own routing mode, Scrapy settings and spider arguments, so a quick test and a nightly schedule can behave completely differently on the same code.

​

Starting a job

​

Run options

Open Run options above the spider list, or inside the schedule editor for a scheduled run.

​

Grouped runs

Normally one spider means one container and one concurrency slot. Grouped puts several spiders in one container, sharing one Twisted reactor. This exists because per-spider containers are the wrong shape for a project with hundreds of small spiders: a thousand light spiders need hundreds of gigabytes as separate containers but only a few gigabytes sharing one reactor. The trade-off is a shared blast radius - a crash at reactor level takes the whole group with it - which is why it is opt-in rather than the default. A grouped job’s items belong to the job, not to any one spider in it. That is why per-spider item counts on the Spiders tab say “+N grouped runs” separately instead of dividing the same items between every spider that shared the job.

​

Pipeline isolation

With Keep the project’s own pipelines off switched on, your ITEM_PIPELINES are removed for that run and only the platform’s own item sink remains. A spider cannot write to S3, a database or an API even if its code says to. This is what makes it safe to trial-run a copy of a production spider here while the real one keeps running elsewhere. Turn it off when you want the project to write where its code writes - and remember that a destination delivers your items either way, from outside the container.

​

Job states

Press Stop on a live job to end it. A job that was never dispatched is simply dropped from the queue.

​

What makes a job failed

A green job with no data is the worst outcome on a paid platform, so a run is reported as failed when:

• a spider never started - a component that failed to initialise, a database middleware with nowhere to connect. Scrapy would still exit 0 here;
• a spider logged a critical error;
• a spider closed for an unexpected reason. Reaching a limit you asked for - an item cap, a page cap, a CLOSESPIDER_TIMEOUT - is not one of those and stays a clean finish;
• the run made no successful requests at all.

Jobs also die on their ceilings: each container has a memory limit and a hard time limit, and a spider that exceeds memory is killed rather than allowed to take the machine with it.

​

Inspecting a job

Inspect opens three views. All three refresh every five seconds while the job is moving, and stop as soon as it is not.

​

Logs

The spider’s own output. Scrapy’s stats dump at the end of a run lands here too, which is where the su_cloud/* routing counters can be read.

​

Items

The scraped items, as JSON, while the job is still running. The runner writes items to storage in large chunks, so what is readable trails what the spider has scraped - the header shows both numbers and names the difference as “still buffered on the runner”. Paging is by cursor: Load more appends the next page and never re-reads what you have already seen. A page is a window onto the data, not an export - for the whole dataset, use a destination. An empty list always comes with a reason, because “not written out yet” and “nothing was scraped” look identical otherwise and only one of them is alarming:

​

Stats

What the crawl actually did, refreshed by the spider every 30 seconds. Three of these are routinely misread, so they are named for exactly what they count: There is deliberately no success rate anywhere on this screen. A redirect that was followed counts as two responses, so no honest percentage exists. While a job is running these numbers are a snapshot, not a final tally, and the screen says so.

​

Delivery

If the project has destinations, the Delivered column shows what happened when the finished job’s items were handed over - 1 240 → mydb.items, or the error if it failed. A delivery failure does not change the job’s own outcome: the crawl did succeed, and the items are still in storage.

​

What the container can and cannot do

• The filesystem is discarded when the job ends. FEEDS is therefore switched off with a warning in the log; items reach storage through the platform instead.
• The telnet console is off, always.
• The network is restricted. Containers cannot reach cloud metadata endpoints or private address ranges, so a spider cannot be pointed at internal infrastructure - ours or anyone else’s.
• No platform credentials are inside. Object storage keys and delivery credentials live outside the container, which is why delivery runs from our side rather than from your pipeline.

​

Next steps