v1, v2, v3 - and they are immutable: deploying again builds a new version rather than replacing one. Every job records the deploy it ran on, so “which code produced this data” is always answerable from the job row alone.
Connecting the repository
Both paths live on the Deploys tab, under Where the code comes from.- GitHub App
- Any other git host
Connect GitHub takes you to GitHub, where you tick the repositories we may read. Coming back, pick one from the dropdown and press Save.We store the installation id and nothing else. The token used to clone is minted for each build and expires within the hour, and it can only see the repositories you ticked. Disconnect forgets the installation on our side; the App stays installed on GitHub until you remove it there.Only the person who owns the installation can attach it, so nobody can point their project at your repositories by guessing an id.
scrapy.cfg at its root.
Choosing what to build
The Branch to build control offers the repository’s branches, fetched from GitHub at that moment. Nothing is cached: branches are created and deleted while people work, and a remembered list would keep naming branches that no longer exist while looking authoritative. Next to it, any ref can be typed by hand. A tag or a commit sha will never appear in a branch list, so the free-text field is the way to build one - and it is also the fallback whenever the list cannot be filled. If listing fails, the reason is shown under the field and typing still works:
A ref typed into Branch to build applies to that build only - the badge says one-off - and the project stays on its saved branch. Change the saved branch in the block above when you want it to stick.
What a build does
1
Clone
A shallow clone of the ref you asked for. History is discarded - it is not part of what gets built.
2
Check
The root must hold a
scrapy.cfg. Without one the build stops with “no scrapy.cfg found - is this the root of a Scrapy project?”.3
Build
A Dockerfile is generated for you. If
requirements.txt exists, it is installed in its own layer, so changing spider code later does not reinstall your dependencies. The project is then copied in and the image is left running as an unprivileged user.4
List the spiders
We run
scrapy list inside the finished image and record the names. This is what lets a job for a spider that does not exist be refused up front, rather than by starting a container so it can fail.What each build records
The Builds table on the Deploys tab shows:
The ref and the commit answer different questions and neither substitutes for the other:
develop is what a person recognises, while the commit is the only thing that reproduces the build. Where the ref reads not recorded, the build predates us keeping it, or it followed the repository’s default branch - whose name we are never told. It is said in words rather than guessed at from the project’s current branch, which would be a guess that reads as a fact.
Pinning a build
By default a run uses the newest deploy, and a schedule set up months ago keeps following the newest - which is what people expect from automation they have stopped thinking about. Under Run options → Deploy version you can pin a schedule to a specific build that is known to work. Set it back to Always the latest deploy to unpin.Advanced: your own Dockerfile
If the generated Dockerfile cannot cover your case, a project may ship its own. It has to buildFROM the platform base image, and a Dockerfile that runs as root, asks for privileged mode, or references the Docker socket is refused - each of those breaks an assumption the sandbox depends on. Contact us for the current base image tag before taking this path.
When a build fails
Next steps
Jobs
Run a spider from the build and watch what it does.
Schedules
Pin a schedule to a build, or let it follow the newest.