Technical Guides

How to Run Screaming Frog SEO Spider on a VPS (Step-by-Step Guide)

July 9, 2026 · 5 min read

Screaming Frog crawls hundreds of thousands of URLs — far too heavy for a laptop. Here is how to run it on a VPS with a GUI, schedule regular crawls and audit sites at scale.

Screaming Frog SEO Spider is the industry-standard website crawler. Point it at a URL and it will systematically crawl the site, mapping out every page, identifying broken links, duplicate content, missing titles, redirect chains, and hundreds of other SEO issues. For a small site, running it on your laptop is fine. For a large site — 50,000, 200,000 or a million URLs — the crawl takes hours and hammers your machine.

That is where a VPS becomes the right tool. A server with dedicated CPU and RAM can run Screaming Frog headlessly or with a virtual display, crawl large sites without freezing your workstation, and run scheduled audits that your SEO team can check any time. This guide walks through running Screaming Frog on a VPS step by step.

Why run Screaming Frog on a server at all?

There are three practical reasons:

Scale. Large crawls are memory-hungry. A modern site crawl can need 8–16 GB of RAM just to hold the URL queue and crawl data. Your laptop was never designed for that.

Automation. Scheduled crawls — nightly, weekly — give you a continuous audit trail. SEO issues get caught the week they appear, not the quarter after.

Always-on availability. The crawler keeps running whether you are at your desk or not, and the results live on a server your whole team can access.

Choosing the right VPS for crawling

Screaming Frog is more about RAM and I/O than raw CPU. A sensible baseline:

4 GB RAM for small-to-medium sites and occasional crawls

8–16 GB RAM for large sites, deeper crawls and regular scheduled audits

NVMe storage so the crawl database and logs write quickly without slowing the crawl

A stable connection and dedicated resources — crawling thousands of pages means thousands of simultaneous outbound requests

PakiWebHost VPS hosting plans scale from 4 GB to 128 GB RAM on NVMe storage, which comfortably covers everything from a quick single-site audit to an agency crawling dozens of client domains on a schedule.

Installing Screaming Frog on a VPS

Screaming Frog ships a Linux version of its Java application. The basic approach on Ubuntu or Debian:

Install Java. Screaming Frog requires a recent Java runtime — install OpenJDK 17 or newer.

Download the Linux package for Screaming Frog and unpack it on the server.

Run it with a display. Screaming Frog is a GUI application. For occasional interactive use, install a lightweight desktop (XFCE) plus a VNC server so you can see the interface from your local machine.

Or run it headlessly. For automation, launch the crawl from the command line with a crawl configuration file and let it write the report to disk.

Option A: run Screaming Frog with a graphical interface

If you want the full Screaming Frog UI on your VPS:


# Install a lightweight desktop and VNC server (Debian/Ubuntu)

apt install xfce4 xfce4-goodies tigervnc-standalone-server

# Start VNC on display :1

vncserver :1 -geometry 1600x1000

Then connect to server_ip:1 with any VNC client and launch Screaming Frog inside the session. You get the full interface — crawl tabs, filters, export options — running on server resources instead of your laptop.

This is the right approach when you want to see the crawl and interact with it: tweak config, re-crawl, inspect the broken-link tab live.

Option B: run Screaming Frog headlessly (automation)

For scheduled audits you do not need a GUI at all. Screaming Frog's command-line mode runs the same engine and writes a report:


screamingfrog --crawl https://example.com --headless \

  --save-crawl --output_folder /var/seo-reports/example --overwrite

Key flags worth knowing:

--crawl URL — the site to crawl

--headless — run without the graphical interface

--save-crawl / --crawl-config — save or reuse a crawl configuration

--output_folder — where the report and exported files are written

--export — export specific tabs (crawl, inlinks, redirects, hreflang) automatically

Combined with cron, this becomes a self-running SEO monitoring system.

Scheduling regular crawls with cron

Once headless crawling works, schedule it:


# Crawl every Sunday at 2am and log the output

0 2 * * 0 /usr/local/bin/crawl-site.sh >> /var/log/screamingfrog.log 2>&1

A small wrapper script can crawl several client sites in sequence, export the reports, and optionally email the team or post a summary to a chat webhook when each crawl finishes. Over time this produces a history of crawl exports — invaluable for showing an SEO client the progress of fixes between audits.

Practical tips for large crawls

Respect the site. Configure a reasonable crawl speed and respect robots.txt; an aggressive crawl is a quick way to get your server IP rate-limited.

Use crawl configurations. Save a config per site — exclusions, custom headers, JS rendering settings — so every scheduled crawl is consistent.

Mind your own resources. While the crawler runs, it uses memory and outbound bandwidth. Schedule heavy crawls off-peak.

Store reports separately. Keep exports on disk (or an object store) so historical reports survive server reinstalls.

Monitor the crawler. A simple cron health check that confirms the last crawl finished lets you notice a silent failure in days, not months.

The bottom line

Screaming Frog on a VPS is the difference between an occasional manual audit and a continuous SEO monitoring system. The setup — Java, the app, either VNC for interactive use or headless mode for automation, and a cron schedule — takes an afternoon. After that, large sites get crawled on server-grade RAM and NVMe, reports accumulate automatically, and your team always knows the current state of every site you look after.

Start small with a PakiWebHost VPS sized to your largest site, and scale the RAM as your crawl ambitions grow.

Ready to launch faster, more reliable hosting?

Get NVMe-powered hosting with free migrations, free SSL and 24/7 support — plans from $1.99/mo.

View Hosting Plans Talk to Us

Frequently Asked Questions

For small and medium sites, 4 GB RAM is workable. Large sites with hundreds of thousands of URLs benefit from 8–16 GB, since the crawler holds the URL queue and crawl data in memory. PakiWebHost VPS plans scale from 4 GB to 128 GB RAM, so you can match the server to your largest crawl.

Yes. Screaming Frog has a headless command-line mode that runs the same crawl engine and writes reports without opening a GUI. This is the recommended way to run scheduled, automated crawls on a server.

Crawling respects the site's robots.txt and terms of service. For sites you own or manage — the typical use case for SEO audits — it is standard practice. For third-party sites, keep crawl speed reasonable and honour robots.txt to avoid being blocked.

No. Screaming Frog needs Java, significant RAM and the ability to run long-lived processes, all of which shared hosting restricts. A VPS with root access and guaranteed resources is the appropriate environment.