A PHP Package for Concurrent Website Crawling: spatie/crawler v9 Explained

TamilTech AI summary

Spatie released crawler v9, a popular open-source PHP package that lets you crawl websites concurrently with Guzzle promises instead of one page at a time. It adds a clean CrawlResponse object with handy methods like status(), wasRedirected(), redirectHistory(), dom(), and body(), plus event hooks such as onCrawled() and observer classes for bigger apps. You can tightly control scope with helpers like internalOnly(), depth(), and foundUrls(), which makes SEO audits, broken-link checks, and sitemap work much faster and cleaner. Install it with Composer (PHP 8.1+), and it works great in Laravel or plain PHP projects. If you need a solid, well-maintained crawler in 2026, this package with its 5,000+ GitHub stars is definitely worth adding to your toolkit.

• What is spatie/crawler and what does it do?
• What's new in spatie/crawler version 9?
• Can spatie/crawler handle JavaScript-rendered pages?
• How do I test my crawler without hitting live websites?

AI-assisted summary, checked by the TamilTech editorial team.