Turn off images or use Adblock rules to make rendering faster.Like other headless browsers you can tell Scrapy Splash to do certain actions before returning the HTML response to your spider. Scrapy Splash is a light weight browser that spins up a HTTP server and which you render pages with by sending urls to request over its HTTP API.Īt this point, Scrapy Splash is a bit outdated, having being overtaken by Playwright and Puppeteer headless browsers, but it still is a very capable headless browser for web scraping. Next, up is scrapy-splash which was developed by many of the core Scrapy developers. If you would like to learn more about Scrapy Playwright then you are check out our Scrapy Playwright Guide, or the scrapy-playwright documentation. However, it is possible to run it with WSL (Windows Subsystem for Linux) Note: As of writing this guide, the only major drawback to Scrapy Playwright is that doesn't work with Windows. Wait for elements to load before returning response.Scrapy-playwright allows you to use the all the Playwright functionality you will ever need when scraping a website.

Puppeteer docker download free