Aggregating data with Portia
Route Planner
You can use the Crawling slide control to manage Slybot's behavior. This lets you define whether Slybot observes the predefined access rules set by the webmaster in robots.txt
(Respect Nofollow
). If you want the crawler to ignore certain pages, just add them to the drop-down list Configure follow-up and
exclude patterns. Slybot only follows links that match the regular expression in the top box. Similarly, Slybot follows any link that matches the regular expression in the bottom box. If you check Overlay blocked links, you see the links the spider would currently follow, or not follow, in the preview on the left (Figure 3).

Portia keeps track of the sections of a page you previously marked in light blue in a template. The Extraction slider lists all currently existing templates. A click on one of the identifiers opens the corresponding template in selection mode. You can now use this to add more selections in the normal way. You can also edit the individual selected elements in the Annotations slider.
Sometimes users only select the desired text roughly with the mouse and refine the selection in the Extractors slider using a regular expression. Clicking on New extractor adds the expression to the corresponding field; Slybot applies it to the cut-out text and returns only the results.
Family Growth
Portia refers to the now complete configuration as a Spider, Slybot then later uses the settings stored in it to harvest pages which have the same structure as the one currently on view. A spider usually applies to a specific domain; you can use the web application to create more spiders. Portia groups these in projects. For example, you could create a project named News that contains a spider with all the settings for theregister.co.uk and for slashdot.org (Figure 4).

By default, Portia creates a project named New_Project. To discover the spiders in a project, click on the project name in the top left corner. The sidebar then lists the spiders; clicking on the sidebar shows the settings they store. You can add a new spider by clicking on the project name and then calling a new URL. By default, each spider is assigned the domain name. The sidebar lists all your projects when you click on the Home icon.
Endurance
Portia stores all your projects in subdirectories below slyd/data/projects
; the directory names are also the project names. To see which spiders live in new_project
, type this at the command line:
portiacrawl slyd/data/projects/new_project
Now, to finally grab the information off the sites, you simply call portiacrawl
with the required spider:
portiacrawl slyd/data/projects/new_project en.wikipedia.com
The output in JSON format is sent to standard output; the -o
parameter redirects the data to a file:
portiacrawl slyd/data/projects/new_project en.wikipedia.com \ -o output.txt
portiacrawl
is really just a wrapper script for Slybot. Its documentation can be found online [4].
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
News
-
Fedora 39 Beta is Now Available for Testing
For fans and users of Fedora Linux, the first beta of release 39 is now available, which is a minor upgrade but does include GNOME 45.
-
Fedora Linux 40 to Drop X11 for KDE Plasma
When Fedora 40 arrives in 2024, there will be a few big changes coming, especially for the KDE Plasma option.
-
Real-Time Ubuntu Available in AWS Marketplace
Anyone looking for a Linux distribution for real-time processing could do a whole lot worse than Real-Time Ubuntu.
-
KSMBD Finally Reaches a Stable State
For those who've been looking forward to the first release of KSMBD, after two years it's no longer considered experimental.
-
Nitrux 3.0.0 Has Been Released
The latest version of Nitrux brings plenty of innovation and fresh apps to the table.
-
Linux From Scratch 12.0 Now Available
If you're looking to roll your own Linux distribution, the latest version of Linux From Scratch is now available with plenty of updates.
-
Linux Kernel 6.5 Has Been Released
The newest Linux kernel, version 6.5, now includes initial support for two very exciting features.
-
UbuntuDDE 23.04 Now Available
A new version of the UbuntuDDE remix has finally arrived with all the updates from the Deepin desktop and everything that comes with the Ubuntu 23.04 base.
-
Star Labs Reveals a New Surface-Like Linux Tablet
If you've ever wanted a tablet that rivals the MS Surface, you're in luck as Star Labs has created such a device.
-
SUSE Going Private (Again)
The company behind SUSE Linux Enterprise, Rancher, and NeuVector recently announced that Marcel LUX III SARL (Marcel), its majority shareholder, intends to delist it from the Frankfurt Stock Exchange by way of a merger.