Automated CSP Violation Crawler: Deployment Summary

We’ve now deployed an automated CSP violation crawler that proactively checks every page on our managed sites for CSP violations — catching problems before real visitors do, and running on a weekly schedule without any manual intervention. The crawling script runs in Window’s Powershell and follows up on our earlier posts about Content Security Policy (CSP) monitoring,

What It Does

The crawler (built with Node.js and Playwright) works in three steps:

• Reads each site’s robots.txt to locate its sitemap (following sitemap index files automatically if a site uses multiple)
• Visits every page listed in the sitemap using a real, headless Chromium browser
• Listens for the browser’s native CSP violation event on each page — the same mechanism that powers live violation reporting — and records anything blocked

Because it uses an actual browser engine rather than a simple HTTP fetch, it catches violations exactly as a real visitor’s browser would encounter them, including from JavaScript-driven content, third-party embeds, and lazy-loaded images.

Problems Solved Along the Way

Getting this running reliably across a dozen-plus production sites surfaced a handful of real infrastructure issues worth documenting:

Incomplete SSL certificate chain

One site’s certificate bundle was missing an intermediate certificate. Browsers and lenient HTTP clients quietly tolerated this by fetching the missing certificate automatically, but Node’s strict TLS validation did not, surfacing a chain problem that had been silently affecting other strict clients as well.

Bot-detection false positives

Our servers run a User-Agent based filter to block malicious scrapers and bots. Early crawler runs were blocked by this same filter, since it correctly treats generic headless-browser traffic as suspicious. The fix was to give the crawler a distinct, self-identifying User-Agent and explicitly allowlist it — while leaving the filter in place (and even strengthened) for everything else. This same investigation actually turned up a real anomaly: automated traffic from a datacenter IP that was spoofing an old desktop browser’s identity on ordinary page requests, while its true headless engine leaked through in a CSP violation report generated internally by the browser itself.

Lazy-loaded content and slow-to-settle pages

Early crawls were missing violations tied to images that only load once scrolled into view, and were timing out entirely on pages with persistent background network activity (trackers, chat widgets) that never technically go “idle.” Both were fixed by switching the crawler’s wait condition to the page load event rather than full network silence, and adding a scripted scroll-through after each page load to trigger lazy content before reading results.

Automated CSP Violation Crawler Current Status

As of the most recent full run, 12 of 13 managed domains crawled completely clean — nearly 1,000 pages checked with zero CSP violations found. The one remaining domain is excluded only because it doesn’t yet have a sitemap generated; it will be added to the rotation once that’s in place.

Automation

The crawler now runs automatically every Sunday at 3:00 AM via Windows Task Scheduler, well outside normal traffic hours. Each run produces:

• A dated report folder, preserving history rather than overwriting the previous run
• A per-domain JSON report with full violation detail
• A summary CSV for a quick at-a-glance view across all sites
• A full log capturing anything that failed, with the underlying error preserved for troubleshooting

Combined with the live CSP violation reporting endpoint covered in earlier posts, this gives us both real-time visibility (from actual visitor traffic) and proactive, scheduled coverage (from the crawler) — without relying on visitors to be the ones who discover a broken policy first.