Smart research using Elasticsearch
More, Please
Websites often offer readers links to articles about similar topics. Using Elasticsearch, the free search engine, is one way to find related documents instantly and automatically.
When people rummage around on the StackOverflow website looking for advice on programming questions, they can find list of links to related topics in the Related section (Figure 1). This helps users if the first search result didn't show what they expected or the located resource is insufficient. According to Gormley and Tong [1], Elasticsearch [2] [3], the free search engine, generates these links on the website in real time from the growing and very impressive collection of 10 million StackOverflow contributions.
Artificial Brain
This isn't actual intelligence at work, because computers still find it difficult to understand the content of a document, meaning they can't find documents with related content. In fact, the algorithm used is based on simple nitpicking – it combines values for word frequency and derives a score from those values.
Elasticsearch uses an inverted index for this task; this index is a complete list of individual words that appear in any of the documents that have been added to the search engine so far. It remembers which document each word has been found in and can therefore instantly output a list of documents for a search term.
For example, if a user is looking for the term perl, Elasticsearch will immediately find the doc-1 document in the inverted index from Figure 2 and present this (one hopes) accurate search result.
When searching for two words (e.g., linux and cpan), two documents come into consideration, but because doc-2 only contains one term, whereas doc-3 contains both, the algorithm gives doc-3 a higher relevance score. In a match list sorted by descending score, doc-3 is then at the very top and is more likely to match the user's expectations.
What Is Relevant?
Not all words are equally important. For example, the word file understandably comes up in quite a large number of documents on computer topics. If the user searches for file linux cpan, doc-2 and doc-3 provide two matches each, but because linux is more significant for one document than file, the algorithm rates linux cpan higher than cpan file and gives doc-3 preference.
The Tf-idf score [4] determines how important a word is within a document. It gives a high value to those words that are prevalent in one document but do not occur too frequently in other documents competing for a high score (i.e., words that underpin the uniqueness of the document). A word's relevance value increases with the number of times the word appears in the document (this is known as term frequency, TF) and decreases if the word also appears in many other documents in the collection (IDF, inverse document frequency).
Searching for the Same
To find documents in the database that are similar to document x, Elasticsearch first extracts all relevant words from x, then forms a search query using these words and returns the results. Elasticsearch performs this search for similar documents using the more_like_this
query [5] command with very little programming required. However, all the relevant documents must be added to the index beforehand. I'll be using the official Perl client Search::Elasticsearch from CPAN for this.
Listing 3 (described later) wades through a directory of text files. These are stories from my Usarundbrief.com blog that I extracted from the home-grown content management system via another Perl script. Each text file corresponds to a blog entry – 877 messages accumulated over almost 20 years, which is why the directory contains 877 files. Listing 1 shows the shell output [6].
Listing 1
Feed Data
$ ls idx | wc -l 877 $ mlt-index Added 10-cents-for-a-grocery-bag.txt Added a-job-for-angelika.txt Added absurd-and-funny-american-tv-shows.txt ...
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.
-
OSI and LPI Form Strategic Alliance
With a goal of strengthening Linux and open source communities, this new alliance aims to nurture the growth of more highly skilled professionals.