Crowl is developped with Python3, and uses Scrapy for core crawling functionnalities.

You'll find here the list of available documentation topics.
If you can't find what you are looking for, please read the contributing guidelines and check the list of open issues.

Here's the list of available topics:

How to install Crowl
Learn how to set Crowl up on your computer.

Links extraction
How to extract links with Crowl, and what kind of data you will get.

Content extraction
How to extract page content with Crowl.

Output: using MySQL
Configure Crowl to save data to MySQL.

Get Connected