Crowl

The open-source SEO crawler

What is Crowl?

A crawler made by SEOs for SEOs. And this means a lot.

Free and open-source

Crowl is distributed under the GNU GPL v3. This means you can use, distribute and modify the source code for private or commercial use, as long as you share your code under the same licence. This also means we do not offer any warranty.

Designed by SEOs

Most of the people developping this crawler are professionnal SEOs, experts in the technical aspects of the job. We've been using crawlers on a daily basis for years, and we know what to expect from such a tool.

Generic AND customizable

Crowl is quite a basic crawler yet, but still evolving!
We aim at providing state-of-the-art functionality and customization, for a perfect fit whichever kind of website you're working on.

Community-based

Our goal is to provide a smart and efficient tool for all SEOs. Feel free to join our gang of un-paid volunteers!
How to contribute

Python & Scrapy

Crowl is developped using Python, and Scrapy. We chose this language both because we like it and it is widely used. If you can code, please give us a hand!
How to contribute

Open roadmap

We have a loooong list of features in mind for this project. However, your ideas and opinions are welcome to help us prioritize the next feature to develop.
View the backlog and contribute

Features

Get Started

Set things up

Crowl works best with Python 3.6+ and MySQL 5.5+.
We also recommend you use pyenv.

Create a MySQL user with the ability to create databases, then clone the git repository.

Once you downloaded Crowl source-code, create a copy of config.sample.ini, fill-in the fields and save it as config.ini.

You can then install Python dependencies by executing this command in your terminal:


    pip install -r requirements.txt
                     

And that should be all!

Launch your first crawl

To launch your first crawl with basic settings, simply indicate the start URL and the project name:


    python crowl.py -u https://www.crowl.tech/ -b crowltech  
                    

Crowl will create a MySQL database to store the crawled data.
Check out the docs for more configuration options.
Enjoy!

Get Connected