crawly

# Crawly

[![Module Version](https://img.shields.io/hexpm/v/crawly.svg)](https://hex.pm/packages/crawly)
[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/crawly/)
[![Total Download](https://img.shields.io/hexpm/dt/crawly.svg)](https://hex.pm/packages/crawly)
[![License](https://img.shields.io/hexpm/l/crawly.svg)](https://github.com/elixir-crawly/crawly/blob/master/LICENSE)
[![Last Updated](https://img.shields.io/github/last-commit/elixir-crawly/crawly.svg)](https://github.com/elixir-crawly/crawly/commits/master)

## Overview

Crawly is an application framework for crawling web sites and
extracting structured data which can be used for a wide range of
useful applications, like data mining, information processing or
historical archival.

## Requirements

1. Elixir `~> 1.14`
2. Works on GNU/Linux, Windows, macOS X, and BSD.


## Quickstart
0. Create a new project: `mix new quickstart --sup`
1. Add Crawly as a dependencies:

   ```elixir
   # mix.exs
   defp deps do
       [
         {:crawly, "~> 0.17.2"},
         {:floki, "~> 0.33.0"}
       ]
   end
   ```
2. Fetch dependencies: `$ mix deps.get`
3. Create a spider

   ```elixir
    # lib/crawly_example/books_to_scrape.ex
    defmodule BooksToScrape do
      use Crawly.Spider

      @impl Crawly.Spider
      def base_url(), do: "https://books.toscrape.com/"

      @impl Crawly.Spider
      def init() do
        [start_urls: ["https://books.toscrape.com/"]]
      end

      @impl Crawly.Spider
      def parse_item(response) do
        # Parse response body to document
        {:ok, document} = Floki.parse_document(response.body)

        # Create item (for pages where items exists)
        items =
          document
          |> Floki.find(".product_pod")
          |> Enum.map(fn x ->
            %{
              title: Floki.find(x, "h3 a") |> Floki.attribute("title") |> Floki.text(),
              price: Floki.find(x, ".product_price .price_color") |> Floki.text(),
              url: response.request_url
            }
          end)

        next_requests =
          document
          |> Floki.find(".next a")
          |> Floki.attribute("href")
          |> Enum.map(fn url ->
            Crawly.Utils.build_absolute_url(url, response.request.url)
            |> Crawly.Utils.request_from_url()
          end)

        %Crawly.ParsedItem{items: items, requests: next_requests}
      end
    end
   ```

    **New in 0.15.0 :**

    > It's possible to use the command to speed up the spider creation,
    so you will have a generated file with all needed callbacks:
    `mix crawly.gen.spider --filepath ./lib/crawly_example/books_to_scrape.ex --spidername BooksToScrape`


4. Configure Crawly

   By default, Crawly does not require any configuration. But obviously you will need a configuration for fine tuning the crawls:
   (in file: `config/config.exs`)

   ```elixir

    import Config

    config :crawly,
      closespider_timeout: 10,
      concurrent_requests_per_