Scrapy middleware to handle javascript pages using seleniumbase driver.

Based on scrapy-selenium library, scrapy-seleniumbase was made for those who want to make use of Seleniumbase Driver convenient features.

Installation

pip install git+https://github.com/Quartz-Core/scrapy-seleniumbase

Configuration

1. Provide keyword arguments for Driver in dict. For example:

   SELENIUMBASE_DRIVER_KWARGS = {
   "browser": "chrome",
   "uc": True,
   "headless": True,
   "block_images": True,
   }

2. Add the SeleniumBaseMiddleware to the downloader middlewares:

   DOWNLOADER_MIDDLEWARES = {
       'scrapy_seleniumbase.SeleniumBaseMiddleware': 800
   }

Usage

Use the scrapy_seleniumbase.SeleniumBaseRequest instead of the scrapy built-in Request like below:

from scrapy_seleniumbase import SeleniumBaseRequest

yield SeleniumBaseRequest(url=url, callback=self.parse_result)

The request will be handled by seleniumbase, and the request will have an additional meta key, named driver containing the seleniumbase driver with the request processed.

def parse_result(self, response):
    print(response.request.meta['driver'].title)

For more information about the available driver methods and attributes, refer to the selenium python documentation (all vanilla selenium driver methods are available) and seleniumbase documentation (look for "driver" specific methods, located at the end of the page).

The selector response attribute work as usual (but contains the html processed by the selenium driver).

def parse_result(self, response):
    print(response.selector.xpath('//title/@text'))

Additional arguments

The scrapy_selenium.SeleniumBaseRequest accept 5 additional arguments:

wait_time / wait_until

When used, webdriver will perform an Explicit wait before returning the response to the spider.

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    wait_time=10,
    wait_until=EC.element_to_be_clickable((By.ID, 'someid'))
)

screenshot

When used, webdriver will take a screenshot of the page and the binary data of the .png captured will be added to the response meta:

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    screenshot=True
)

def parse_result(self, response):
    with open('image.png', 'wb') as image_file:
        image_file.write(response.meta['screenshot'])

script

When used, webdriver will execute custom JavaScript code.

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    script='window.scrollTo(0, document.body.scrollHeight);',
)

driver_methods

When used, seleniumbase webdriver will execute methods, provided as strings in a list, before returning page's html.

def start_requests(self):
    for url in self.start_urls:
        yield SeleniumRequest(
                        url=url,
                        driver_methods=['''.find_element("xpath","some_xpath").click()'''])
)