Parallel shell with pdsh
Shell Games
The most fundamental tool needed to administer a cluster is a parallel shell, which allows you to run the same command on a series of nodes. In this article, we look at pdsh.
A parallel shell allows you to run the same command on designated nodes in the cluster, so you don't have to log in to each node to run the command. This tool can be useful in many ways, but I like to use it when performing administrative tasks, such as:
- Checking the versions of particular software packages on each node
- Checking the OS version on all nodes
- Checking the kernel version on all nodes
- Searching the system logs on each node (if you don't store them centrally)
- Examining the CPU usage on each node
- Examining local I/O (if the nodes are doing local I/O)
- Checking whether any nodes are swapping
- Spot-monitoring the compute nodes
The complete list of possible tasks is extensive, but anything you want to do on a single node can be done on a large number of nodes using a parallel shell tool.
If you try to use a parallel shell on a 50,000-node cluster, however, the time skew could be large enough to make the results meaningless. Although certain techniques can allow the use of parallel commands on a large number of nodes, parallel shells are better used on a modest number of nodes or to gather information on slowly varying data. Parallel shells are even great for administering instances in the cloud on something like Amazon Web Services (AWS).
Many parallel shells are available – including DSH [1], PyDSH [2], PPSS [3], PSSH [4], pdsh [5], PuSSH [6], sshpt [7], and mqsh [8] – and each tool has its pros and cons. (Note: I have not tested all of these tools, so I can't vouch for them.) Several of these tools are written in Python, which has become a very popular tool for devops.
In this article, I'll select one of the parallel shells to illustrate its possibilities. Other tools are fairly similar, with some syntactical differences and various sets of features. The tool I'm going to talk about here is pdsh
.
Introduction to pdsh
Pdsh is arguably one of the most popular parallel shell tools. The most recent version on SourceForge as of writing this article is 2.26, dated 2011-05-01. Code development appears to have moved to Google code. The most recent version there is 2.29, updated February 2013. I'll be using that version in this article.
Pdsh is very interesting because it allows you to run commands on multiple nodes using only ssh
. The client nodes only need ssh
, which is generally present on systems, and you don't need to install any extra software on the compute nodes – you just need ssh
. However, you need the ability to SSH to any node without a password ("passwordless SSH").
Building and Installing pdsh
Building and installing pdsh is really simple if you've built code using GNU's autoconfigure before. The steps are quite easy:
./configure --with-ssh --without-rsh make make install
This puts the binaries into /usr/local/
, which is fine for testing purposes. For production work, I would put it in /opt
or something like that – just be sure it's in your path.
You might notice that I used the --without-rsh
option in the configure
command. By default, pdsh uses rsh
, which is not really secure, so I chose to exclude it from the configuration. In the output in Listing 1, you can see the pdsh rcmd modules (rcmd is the remote command used by pdsh). Notice that the "available rcmd modules" at the end of the output lists only ssh and exec. If I didn't exclude rsh, it would be listed here, too, and it would be the default. To override rsh and make ssh the default, you just add the following line to your .bashrc
file:
Listing 1
rcmd Modules
export PDSH_RCMD_TYPE=ssh
Be sure to "source" your .bashrc
file (i.e., source .bashrc
) to set the environment variable. You can also log out and log back in. If, for some reason, you see the following when you try running pdsh,
$ pdsh -w 192.168.1.250 ls -s pdsh@home4: 192.168.1.250: rcmd: socket: Permission denied
then you have built it with rsh. You can either rebuild pdsh without rsh, or you can use the environment variable in your .bashrc
file, or you can do both.
First pdsh Commands
To begin, I'll try to get the kernel version of a node by using its IP address:
$ pdsh -w 192.168.1.250 uname -r 192.168.1.250: 2.6.32-431.11.2.el6.x86_64
The -w
option means I am specifying the node(s) that will run the command. In this case, I specified the IP address of the node (192.168.1.250). After the list of nodes, I add the command I want to run, which is uname -r
in this case. Notice that pdsh starts the output line by identifying the node name.
If you need to mix rcmd modules in a single command, you can specify which module to use in the command line,
$ pdsh -w ssh:laytonjb@192.168.1.250 uname -r 192.168.1.250: 2.6.32-431.11.2.el6.x86_64
by putting the rcmd module before the node name. In this case, I used ssh and typical ssh syntax.
A very common way of using pdsh is to set the environment variable WCOLL
to point to the file that contains the list of hosts you want to use in the pdsh command. For example, I created a subdirectory PDSH
where I create a file hosts
that lists the hosts I want to use:
[laytonjb@home4 ~]$ mkdir PDSH [laytonjb@home4 ~]$ cd PDSH [laytonjb@home4 PDSH]$ vi hosts [laytonjb@home4 PDSH]$ more hosts 192.168.1.4 192.168.1.250
I'm only using two nodes: 192.168.1.4 and 192.168.1.250. The first is my test system (like a cluster head node), and the second is my test compute node. You can put hosts in the file as you would on the command line separated by commas. Be sure not to put a blank line at the end of the file because pdsh will try to connect to it. You can put the environment variable WCOLL
in your .bashrc
file:
export WCOLL=/home/laytonjb/PDSH/hosts
As before, you can source your .bashrc file, or you can log out and log back in.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.