Highly Parallel Programming with Apache Spark
Tutorials – Apache Spark
Churn through lots of data with cluster computing on Apache's Spark platform.
As a society, we're creating more data than ever before. We're monitoring everything from the planet's weather to the performance of our computers, and we're storing all this information. But how do you process all this data? On a single machine, you can get a few terabytes of disk space and a few hundred gigabytes of memory (at least, you can if your pockets are deep enough), but how do you churn through a petabyte of raw ones and zeros? Basically, you're going to need more than one computer, and you're going to look for a method of running your programs on many machines at the same time: Apache Spark [1].
Before you run off and buy a rack of servers, slow down! We're going to start by introducing Spark on a single machine. Once you've mastered the basics, you can scale up.
Spark is a data processing engine that is often used with Hadoop for managing large amounts of data in a highly distributed manner. If you move forward with Spark, you're probably going to end up with a complete Hadoop setup; however, that's also getting ahead of ourselves. We can start Spark as a standalone service on a single computer.
The first thing you need is Java v1.8. On Ubuntu and derivatives, enter:
apt install openjdk-8-jre
On other systems, take a look at your package manager for details.
The second thing you need is Spark itself. You can download Spark from the Apache website [2]. I used the latest version at the time of writing (2.1.1 Pre-build for Hadoop 2.7). Download this version and extract the downloaded file.
Open a terminal and navigate to the folder that you've just extracted. From there, you should be able to run the following command:
bin/pyspark
This command will start Spark and drop you into a PySpark shell. Spark isn't just a programming environment, it's a way of scheduling jobs across a cluster of machines. This is true even in this case when we're just running on a cluster of one machine. There's a web interface to this server on port 4040, so point your web browser to localhost:4040 to see a list of what's running (Figure 1). At the moment, it won't show much, but this interface is useful for keeping an eye on your Spark server as you have more machines and jobs.
Spark isn't specific to Python. In fact, Scala is the default language, but there are tools for many languages, and Python is my language of preference. PySpark is a tool for submitting Python jobs to the Spark cluster.
Take a look at a simple PySpark program:
import re words = sc.textFile("/usr/share/dict/words") def check_regex(word): return re.match('^[bean][bean][bean][bean]$',word) out = words.filter(check_regex) out.take(5)
Resilient Distributed Datasets (RDDs) are the core concept of Spark. They're basically data structures that are stored across all the machines in the cluster so that any operation can be easily parallelized across all the machines. Each RDD is resilient because it can't change. Any operation on an RDD creates a new one while leaving the old one intact.
Our really simple code here takes the words file from your machine (if it's not at this location, you can download a words file from the Linux Voice site [3]), points your program to the downloaded file), and builds an RDD, with each item in the RDD being created from a line in the file. Essentially, what we have now is an array containing one entry for each word in the English language.
RDDs aren't regular arrays, though. They have methods that allow you to work on them. In this example, I will use the filter
method, which takes a function as its argument, and this function is run on every item in the RDD to create a new RDD. I will also use the take
method, which gets us a sample from the RDD; in this case, the argument 5
means I want five items from the RDD.
The sc
referenced in the program is SparkContext
, which serves as an entry point for Spark functionality. If you're trying to use Spark from outside the PySpark shell, you'll need to import SparkContext
and set it up. Take a look at the documentation for details of how to set up SparkContext
, as it's a bit different depending on how you're running Spark.
This simple program returns up to five words containing just the letters b, e, a, and n (Figure 2). It is, admittedly, not a particularly impressive program, but it gets us started with Spark.
Another core Spark concept is DataFrames. DataFrames are very similar to RDDs except for the fact that they have a schema. In the previous example, each entry in the RDD has a single element, a word, but this doesn't have to be the case. RDDs and DataFrames often contain complex sets of data, and setting them against a schema allows you to make more structured queries.
A schema is basically a table that we want the data to fit into. As you'll see soon, we have to define a name for the columns in the table as a Python list (in this case, the list ['word']).
Take a look at the following:
words_df = words.map(lambda x: (x, )).toDF(['word']) out = words_df.filter("word like '%ing'") out.take(5)
This code follows on from the previous code block and needs to run in the same PySpark session. The first line does two things; first, it uses the map
method of the RDD to wrap the words up as tuples. This is important because DataFrames need schemas, and the schema can't be just a single value; it has to be a list, tuple, or row. The second part of the first line builds a new DataFrame from this map, and the DataFrame has a schema with a single column called word
.
Again we use the filter
method to pull out some data we're interested in. This time, however, we're not passing a function but a string. Database users amongst you will recognize the syntax as the same as the where
clause in an SQL statement. We're asking for every row where the column word is like '%ing'
, and the percent sign matches any text in SQL, so this means every row where the word ends in ing
.
Now you've run a few tasks in Spark, you can go back to your browser at localhost:4040 and you should see several completed tasks that have run on your Spark cluster.
This has been a very whirlwind tour of the basics of Apache Spark. Hopefully, you now understand what Spark is and how you can program in it. See the Apache Spark website for examples, documentation, and other information on using Spark (Figure 3). The real advantage of Spark is when you're dealing with massive datasets. You can create DataFrames on the fly and query them efficiently across massive clusters of computers. It's not unusual to have Spark clusters with well over a terabyte of RAM, where huge datasets can sit and be processed without ever hitting the disk, leading to really powerful analyses taking place incredibly quickly.

Infos
- Apache Spark: http://spark.apache.org
- Download Spark: http://spark.apache.org/downloads.html
- Download words file: http://www.linuxvoice.com/words
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
Kubuntu Focus Announces XE Gen 2 Linux Laptop
Another Kubuntu-based laptop has arrived to be your next ultra-portable powerhouse with a Linux heart.
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.
-
openSUSE Leap 15.5 Beta Now Available
The final version of the Leap 15 series of openSUSE is available for beta testing and offers only new software versions.
-
Linux Kernel 6.2 Released with New Hardware Support
Find out what's new in the most recent release from Linus Torvalds and the Linux kernel team.