Sneakpeek - is a platform to author, schedule and monitor scrapers in an easy, fast and extensible way. It's the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis.
- Horizontally scalable
- Robust scraper scheduler and priority task queue
- Multiple storage implementations to persist scrapers' configs, tasks, logs, etc.
- JSON RPC API to manage the platform programmatically
- Useful UI to manage all of your scrapers
- Scraper IDE to enable you developing scrapers right in your browser
- Easily extendable via middleware
Here's a demo project which uses Sneakpeek framework.
You can also run the demo using Docker:
docker run -it --rm -p 8080:8080 flulemon/sneakpeek-demoOnce it has started head over to http://localhost:8080 to play around with it.
For the full documentation please visit sneakpeek-py.readthedocs.io
Please take a look at our contributing guidelines if you're interested in helping!
- Headful and headless browser engines middleware (Selenium and Playwright)
- SQL and AmazonDB storage implementation
- Advanced monitoring for the scrapers' health