Creating low-cost "learning clusters" for HPC

Crazy Eddie's Used Systems

Up to this point, I've mostly focused on the software aspect of clusters – programming and cluster tools. However, the first two letters of HPC stand for "high" and "performance," which means understanding and tuning the hardware for the cluster. Toward this end, you need to find some hardware to build your own cluster.

An easy way to get started is to find some older hardware that is no longer being used. Friends and family are good sources, and you can sometimes even find good deals at yard sales, although "buyer beware" is a truth when you are in someone's garage buying a used PC. A number of websites also sell reasonable used systems. (Google is your friend, but be sure you read reviews before buying from a website.) A few stores also sell used hardware, but their prices can be a bit on the high side because you usually get some sort of warranty with them.

If you choose older hardware, I'll assume you want to use them without the virtualization I discussed previously. The HPC world calls this using "bare metal." Although you can use just one system to learn about cluster programming, just as you could use your laptop or desktop, as explained in the first section, ideally, you should have access to two or more systems and a basic switch. Figure 1 shows an outline of a really basic cluster with three compute nodes. (I've shown three compute nodes here, but you can get by with just one.)

Figure 1: Generic cluster layout.

The most important item is the Head Node, also called the master node; it is the brains of the cluster. In this particular setup, it also contains some storage for users, as shown by the disk to the right of the head node. The head node also contains the cluster tools used to provision other nodes, called compute nodes, and to run and control applications.

Note that the head node has two network connections. The first is to the Outside World, which is the connection into the cluster and, often, network device eth0. The second is from the head node to the cluster switch, which is a private network, so many times it will have an address of 192.168.x.x (don't forget that the head node has two Ethernet interfaces). This network is the only way the head node accesses the cluster nodes. You can use a single compute node or several. The only real requirements is that they can run a reasonable distribution.

The details on how you create a cluster from a head node, a switch, and a compute node, are usually found with the documentation of the cluster tool you are learning to use. Although it's beyond the scope of this article to cover them, look around, because a number of articles discuss how to set up and configure hardware for the various cluster tools.

Low-Cost Build-Your-Own

The use of older, used systems borrowed or bought from friends and family is a good way to get started learning about clusters and "hacking" the hardware, but sometimes there really is nothing like designing your own systems, so you have control of the hardware and know precisely what is in each node. This means going shopping.

When learning about clusters, my aim is to control the costs as much as possible. The prices I'll be quoting are from Newegg [16] at the time of writing (your mileage may vary), although you might be able to find cheaper prices. Also, just because I'm listing these items does not mean I endorse the associated companies (or the website). I just looked for the least expensive products in the various categories.

Table 1 lists the components for a simple head node with a reasonable amount of memory (4GB) and a dual-core processor. Note that these prices don't include tax and delivery (those vary too much to be estimated accurately). I included a DVD burner in the head node, but this is not strictly necessary. Overall, roughly US$  250 will get you a dual-core head node.

Table 1

Dual-Core Node

Name

Description

Quantity

Price (US$)

Extended Price

AMD A4-3300 processor

2.5GHz dual-core processor with AMD Radeon HD 6410D graphics

1

$39.99

$39.99

ASRock A55M-V2 FM1 motherboard

MicroATX Socket FM1 motherboard; 2x240-pin DDR3 2400+ (OC)/1866/1600/1333/1066/800 memory; 16GB maximum memory, dual-channel; 1 PCIe 2.0x16 slot, 1 PCI slot; 2x SATA 3Gbps connectors, RAID-0/1/10; Realtek 8105E GigE NIC

1

$42.99

$42.99

1866MHz dual-channel memory

2x 2GB = 4GB total memory

1

$39.99

$39.99

ZION XON-720P mATX/ITX slim desktop case

300W power supply

1

$34.99

$34.99

250GB SATA 3.0Gbps disk

1

$59.99

$59.99

DVD +/-RW burner

1

$14.99

$14.99

GigE network card

1

$15.00

$15.00

Total

$247.94

A compute node version of the head node doesn't need a DVD burner or the second NIC, bringing the price down to US$  218 per node. If you want to be more daring and go diskless, then the price is US$  158 for a compute node.

Out of curiosity, I also configured a quad-core server to get an idea how much the extra cores cost. I used the same basic configuration as the dual-core, but I used the least expensive quad-core processor I could find. With the extra cores, I doubled the memory to 8GB using 4 DIMM slots instead of two, so each core has 2GB of memory (same as the dual-core system). Table 2 shows the configuration I used. Roughly US$  370 will get you a quad-core head node.

Table 2

Quad-Core Head Node

Name

Description

Quantity

Price (US$)

Extended Price

AMD A6-3650 processor

2.6GHz quad-core processor with AMD Radeon HD 6530D graphics

1

$69.99

$69.99

ASRock A75 PRO4-M FM1 motherboard

microATX Socket FM1 motherboard; 4x240-pin DDR3 2400+ (OC)/1866 (OC)/1600 (OC)/1333/1066/800 memory; 32GB maximum memory, dual-channel; 2 PCIe 2.0x16 slots (x16, x4), 2 PCI slots; 5x SATA 6Gbps connectors, RAID-0/1/10; Realtek 8111E GigE NIC

1

$74.99

$74.99

1600MHz dual-channel memory

4x4GB = 16GB total memory

2

$49.99

$99.98

ZION XON-720P mATX/ITX slim desktop case

300W power supply

1

$34.99

$34.99

250GB SATA 3.0Gbps disk

1

$59.99

$59.99

DVD +/-RW burner

1

$14.99

$14.99

GigE network card

1

$15.00

$15.00

Total

$369.93

For the compute nodes, you don't need the second NIC or the DVD burner, resulting in a price of US$  340. If you go diskless, you can get the price down to US$  280.

You can get a small five-port GigE switch for less than US$  20.00. Throw in some Ethernet cables at about US$  2 per cable, and you get the following system totals:

  • Five-node dual-core processor cluster: US$  1,150 @ US$  115/core
  • Five-node dual-core processor, diskless cluster: US$  910 @ US$  91/core
  • Five-node quad-core processor cluster: US$  1,760 @ US$  88/core
  • Five-node quad-core processor, diskless cluster: US$  1,520 @ US$  76/core

These pricing games can be fun, but in all seriousness, they give you an idea how much money you would need to build your five-node cluster. (I limited it to five total nodes because of the size of the GigE switch.) Building your own systems means you know exactly what is in the cluster and how to tune it for the best possible performance.

Summary

As you discover HPC computing, you will likely want to learn about using HPC clusters. The two fundamental things to learn about clusters are writing applications and running or administering clusters. In this article, I walked through some options, starting with building and running applications on what you have today – your laptop or desktop computer – and highlighting options for further investigation.

One of the quickest and easiest ways to really get started is to use virtualization, which is sort of the antithesis of HPC: Virtualization takes a single system and makes it appear to be several separate systems; HPC tries to take several systems and make them appear to be a single system. You can use your laptop or desktop to run a number of VMs and then use them to learn about HPC applications or administering clusters. In doing so, you not only learn about HPC, you learn about virtualization for Linux, which is not a bad skill to have.

The next step up from using VMs on your laptop or desktop is to use VMs from a cloud provider such as Amazon. Cloud VMs give you a great deal of freedom in choosing the level of performance, as well as the number of VMs, you want. It also gives you another kind of freedom: Once you are done with the system, you just turn it off and it goes away. Cloud VMs can also be fairly cost effective because you only pay for the time you use them. However, you also don't have the benefit of owning the hardware and learning more about the details of configuring it and tuning applications for it.

Finally, I went over some options for your own hardware, starting with used or older systems, which can give you a very quick start configuring, managing, and running a cluster using bare metal. Of course, nothing is as fun as building the systems yourself, so I covered the option of building a cluster with a head node, a private network, and one to five compute nodes. Going this route allows you to understand what truly goes into the system, so you can tune it to run applications or tune the applications for the hardware.

If you are brand new to HPC, I would suggest first learning how to program for clusters, because this helps you understand how parallel applications work and gives you a much better understanding of clusters.

I recommend choosing the easiest route; spinning up some VMs on your laptop or desktop is not a bad way to go because you learn about virtualization while you are learning about HPC. Once you have written an application or two and understand how they work, and if you are interested in learning how to build or manage clusters, you should move on and experiment with cluster tools. Again, the easiest way to get started is to use VMs on your laptop or desktop. I know some cluster tool developers that do this every day, illustrating the usefulness of such an approach, even though nothing beats building your own hardware.

The Author

Jeff Layton is an Enterprise Technologist for HPC at Dell. He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales (but never during working hours).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • HPC Cluster Basics

    The beginning for high-performance computing is understanding what you are trying to achieve, the assumptions you make to get there, and the resulting boundaries and limitations imposed on you and your HPC system.

  • StarCluster

    Cloud computing has become a viable option for highperformance computing. In this article, we discuss the use case for cloud-based HPC, introduce the StarCluster toolkit, and show how to build a custom machine image for compute nodes.

  • Rocks Clustering

    Rocks offers an easy solution for clustering with virtual machines.

  • CoreOS

    CoreOS is an operating system for the cloud that relies on Docker and also lets you build compute clusters out of the box.

  • PelicanHPC

    Crunch big numbers with your very own high-performance computing cluster.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News