About

Scrapy Demo Project
for A Data Pro summer Internship Burgas 2022

Install

requirements:

• python >= 3.9
• poetry

git clone [email protected]:simeonmartev/crawl_licitatie.git
cd crawl_licitatie
poetry install

Crawl

list spiders

cd crawl_licitatie
scrapy list

crawl

poetry shell
cd crawl_licitatie
scrapy crawl api_spider

Useful links

• HTTP reference @ MDM
• HTML reference @ MDM
• JSON
• Xpath reference @ MDM
• XPath Tutorial
• Scrapy Docs
• Chrome DevTools
• Firefox DevTools
• SQLite: SQL database engine
• Peewee is a simple and small ORM
• JSON Schema is a vocabulary that allows you to annotate and validate JSON documents
• ipdb IPython debugger
• Visual Studio Code a source-code editor
• Data Collection From Open Web presentation