Crowl is developped with Python3, and uses Scrapy for core crawling functionnalities.
You'll find here the list of available documentation topics.
If you can't find what you are looking for, please read the contributing guidelines and check the list of open issues.
Be sure to check tags and release notes to keep track of what's changed.
Here's the list of available topics:
How to install Crowl
Learn how to set Crowl up on your computer.
Crowl configuration file
Every Crowl configuration option detailed.
Links extraction
How to extract links with Crowl, and what kind of data you will get.
Content extraction
How to extract page content with Crowl.
Output: using CSV
Configure Crowl to save data to CSV files.
Output: using MySQL
Configure Crowl to save data to a MySQL database.
Resume a crawl
Stop and resume a crawl.
