A crawler demonstrates the capabilities of version 1.0 of the Scrapy framework [1] running under Python 2.7 [2]. Scrapy is an open source framework for extracting data from websites. It recursively crawls through HTML documents and follows all the links it finds.

In the spirit of HTML5, the test created in this article is designed to reveal non-semantic markup on websites. The crawler counts the number of words used per page, as well as the number of characteristic tag groups (Table 1), saving the results along with the URL in a database.

To install the required packages, I used the Debian 8 Apt package manager:

apt-get install python-pip libxslt1-dev python-dev python-lxml

The packages include the Python package manager (Pip), the libxslt library along with the header files, the Python header files, and the Python bindings for libxml and libxslt. Because Debian 8 comes with Python 2.7 and libxml pre-installed, you can install Scrapy as follows:

pip install scrapy

Unlike Apt, Pip installs as the latest Scrapy version for Python 2.7 from the Python package index [3].

Test Run

To begin, open an interactive session in the Scrapy shell by entering scrapy shell (Figure 1). Next, send a command to the Scrapy engine to tell the on-board downloader to read the German Linux-Magazin homepage through an HTTP request and transfer the results to the response object (Figure 2):

Figure 1: In the Scrapy shell, you can test commands interactively.

Figure 2: The on-board downloader bundles the Linux-Magazin site into a response object.

fetch('http://www.linux-magazin.de')

Figure 3 demonstrates in detail how the components of the Scrapy architecture work together. This illustration makes it clear that the engine does not talk directly to the downloaders but first passes the HTTP request to the scheduler (Figure 3, top). The downloader middleware (Figure 3, center right) modifies the HTTP request before deployment. CookiesMiddleware, which is enabled by default, stores the cookies from the queried domain, whereas RobotsTxtMiddleware suppresses the retrieval of documents blocked for crawlers by the robots.txt [5] file on the web server.

Figure 3: The Scrapy engine delegates tasks to different components, like the spider, the item pipelines, and middleware [4]. Twisted, an event-driven network framework, works in the background.

Tracker

Scrapy evaluates the document components by interactively querying and investigating them via the response object, as shown in Figure 2. The selection is made either with the help of CSS selectors [6], as in jQuery, or with XPath expressions [7], as in XSLT. For example, first enter the command,

response.xpath('//title/text()').extract()

as shown in Figure 2 to call the xpath() method. The //title subexpression first selects all <title> tags from the HTML document, and /text() selects the text nodes that follow. The extract() method transfers the results set to the Python list:

[u'Home \xbb Linux-Magazin']

Using the expression

len(response.xpath('//a/@href').extract())

you can extract the values of the <href> attributes of all <a> tags in a list. Their length is discovered by the Python len() function; in this case, there are 215 (see Figure 2).

Getting Started

A sample application can be compiled with a little knowledge of Scrapy. The command

scrapy startproject mirror

lays the foundations by creating a matching directory structure (Listing 1). The application reaches the user after changing to the mirror project directory. Empty files with the name __init__.py are purely technical in nature [8]. The spider and pipeline classes can be found in subdirectories of the same name. Scrapy stores the results in the results directory and the associated reports in the reports directory.

A few listings will be added to the skeleton project later. The mirror/utils.py file from the last line of Listing 1 stores the helper functions. Listing 2 shows the contents of this file.

The global settings for the project are also in Python format and belong in mirror/settings.py (Listing 3). Scrapy thus creates the variables in the first three lines, their capitalization is reminiscent of constants in C, although they are not available in Python.

Line 1 stores the name Scrapy sends to the requested web server instead of the browser identifier in the header of the HTTP requests. Lines 2 and 3 are inherent in the Scrapy system and require no change. The variables that follow store application-specific constants.