Web Workers

A few years ago, I developed a commercial botnet, also known as a sniper. This sniper botnet automatically purchased vehicles from a website. It also arranged for payment and scheduled shipping to my client, a very reputable automotive dealership. Although the first live test was unsuccessful, I quickly found the problem and composed an email to indicate that we were ready to try again. The email read something like this…

Dear Client, The snipers are in place and they're going to fire at cars today, starting at noon. Hopefully they'll hit something today …!

Just before I sent the message, I recalled talk of government interception and screening of email [1] [2]. I wisely reworded the message with less inflammatory language. Such is my life as a botnet developer.

Colorful language is just one of the obstacles botnet developers face, but beyond their bad reputation and unfortunate name, commercial botnets create true opportunities for business people and developers. In this article, I describe my automobile purchasing project, explore the importance of commercial botnets, and detail how open source tools, including Linux, are important in the world of botnet development and hosting.

Bad Reputation

Please don't paint the wrong picture of me when I say that I am a botnet developer. I am, in fact, an upstanding and well-published member of the IT community, with a background in security. I am also someone who develops software that delivers demonstrable value to clients. The primary way I generate that value is by automating online tasks. Examples of tasks I automate include verifying online prices, checking websites for accuracy, verifying identities, or procuring items for resale. I've even done work for a foreign government. Over 15 years and numerous projects, I have found the best way to automate online tasks is by distributing the load with a botnet.

No matter how well my botnets are received, I constantly battle the bad reputation of other people's botnets. When most people hear the word "botnet," they visualize stealthy viruses and spyware executing on compromised computers. Most of this reputation is well deserved. For example, according to a Microsoft security report published earlier this year [3], a single botnet, Conficker, infected an estimated 7 million computers worldwide. File-sharing websites were the primary carrier for the infection. The symptoms included disabled security services. The prognosis for infected computers wasn't good. If left untreated, the Conficker virus could eventually take full control of the infected computer, making it a zombie slave to a master server somewhere in the cloud. Once infected, the computers could be used for a variety of nefarious purposes, such as launching DDoS (Distributed Denial of Service) attacks or committing identity fraud.

Although the intent and distribution methods are different, the Conficker botnet shares many similarities to the "White Hat" botnets I write for commercial purposes, but the distinction is important. Not every botnet is something that should be removed by anti-virus software. In reality, a small percentage of botnets are actually quite legitimate and useful, especially to a business that is looking for a competitive advantage.

The Sniper Botnet

Although I have developed many commercial botnets, the sniper project is a particularly nice example. Unlike the nasty Conficker, this bot had no ill intent, and it was possibly the only way to solve a real business problem. Now, for the first time, I've been granted permission to talk about the project.

One day, an automotive client called with a problem. He explained that auto dealers used a special secured website to purchase returned rental vehicles (or for readers in the UK, retired cars for hire). These cars were high-value vehicles for automotive dealerships because they had low mileage and were late-model versions of a popular brand. Every automotive dealership in The United States wanted vehicles like these in their resale inventory, and everyday, several hundred of these vehicles were added to the website. Of those offered for sale, about a half-a-dozen vehicles always piqued the interest of nearly every automotive dealership in the country. These vehicles were particularly popular because they had special equipment, were discounted, or were a favorite model.

My client's problem was that, although he could view the cars for sale on the website, he wasn't able to purchase vehicles until a specific time, and when that time arrived, the servers got so much traffic from competing dealerships that it was difficult to even make a bid, let alone purchase anything. My client had discovered a great source for desirable used cars but was unable to purchase any of them. He had a business problem in need of a solution and asked for my help. I suspected that a custom botnet could solve his problem.

How Commercial Botnets Work

Botnets like the auto sniper share three common parts:

• A central server.
• A number of client machines (which I refer to as harvesters).
• A web portal for administering the botnet.

Figure 1 shows how these elements are interrelated.

Figure 1: Anatomy of a commercial botnet.

The Central Server and Web Portal

As with any botnet, the central server was the heart and soul of the auto sniper. It supported a web portal that replaced the frustrating interface on the original website. Instead of competing with everybody else to purchase vehicles at a precise moment, my client now leisurely selected vehicles he wanted to buy at the botnet's web portal. The central server then assigned these cars to individual harvester computers. Later in the day, my client used the web portal to discover which cars had been purchased by the botnet.

Harvesters

We deployed between one and 12 harvesters for each car we wanted to buy. The harvesters were distributed around the country. Some computers ran one harvester, and others had multiple instances of harvesters running simultaneously. In all cases, the harvesters ran on computers that we owned (unlike nefarious botnets like Conficker that prey on other people's computers). Once assigned to a vehicle on the sale (target) website, the harvester would verify that the car was still active in the sale and lock onto the web server's clock. As the sale grew near, it would more closely synchronize with the sale server's clock. During the synchronization process, each harvester also gauged the amount of server latency and estimated how many people were using the website.

Depending on how many people were estimated to be buying cars on the website, each harvester would send precisely timed requests to purchase its assigned vehicle. On particularly busy days, the request was made as early as five seconds before sale time. In most cases, however, the request to purchase a car was made closer to the actual sale. Once a sale was confirmed, the harvester would pay for the vehicle and arrange for shipping to my client at his dealership.

Most of the botnet's individual attempts to purchase cars failed, but collectively, the botnet had a purchase success rate of more than 95 percent. This rate is remarkable if you consider that, before using this botnet, the success rate was so dismal that it wasn't even worth trying to buy the cars. Using this sniper botnet, my client was almost assured of getting any vehicle he wanted from the sale website. In fact, I had to continually temper my client's enthusiasm and remind him "not to get greedy" by attempting to purchase too many cars in a single day.

The sniper botnet ran for more than a year and purchased millions of dollars of cars with very little need for maintenance. We did, however, at one point discover that we were competing with another automotive dealership that also used some automated methods for purchasing vehicles from the same website. This lead to an inevitable competition with the other programmers. It also lead to the clock synchronization techniques described earlier. Once clock synchronization was implemented and tuned, our botnet again dominated the competition, even the competition from the other botnet!

Under the Hood

Although the internal operation of the botnet might sound complicated, the process of developing any botnet is drastically simplified when tasks are viewed individually and the correct tools are used. Following are some code snippets that demonstrate how writing an effective commercial botnet isn't as hard as it looks. One thing that makes development easier is the set of open source tools I've developed for this purpose, which you can find online [4]. The libraries include functions for parsing and downloading web pages, as I'll explore next. They also include code that will aid many other tasks, like resizing images, writing spiders, and communicating with email servers.

Downloading Web Pages

The harvesters for the sniper botnet not only had to download web pages from the sale server, they also had to request tasks from the central server. In either case, they were essentially making HTTP requests from web servers for web pages. If you use the library LIB_http [4], the code to download a web page is quite simple. The code shown below in Listing 1 shows an example.

01 <?php
02 include('LIB_http.php');                        // include the http code from library
03 $target = 'http://www.schrenk.com';             // specify a target website
04 $referrer = '';                                 // specify a referrer, if needed
05 $result = http_get($target, $referrer);
06 $raw_data = $result['FILE'];
07 ?>
08 @KE

Listing 1 uses LIB_http, which makes extensive use of libcurl4 [5], the PHP implementation of cURL. The cURL project is a cross-platform toolset for accessing resources in a variety of protocols (not just HTTP). cURL also automatically takes care of obstacles like cookie management, page forwarding, encryption, and other low-level tasks.

Parsing Data

Once you've downloaded a web page, you'll need to parse the important stuff from the surrounding text that is of little interest to your harvester. Take, for example, the instructions coming from the central server, which are essentially specialized web pages. For this demonstration, assume the data the harvester receives from the central server looks like the XML depicted in Listing 2.

01 <xml>
02     <task>purchase</task>
03     <target>somewebsite.com</target>
04     <username>clientname</username>
05     <password>supersecret</password>
06     <vehicles>
07         <vin>192KF8Q2MSJE8T921</vin>
08         <vin>192KF8IWECOWIMCE3</vin>
09         <vin>192KF0032HJS002AS</vin>
10     </vehicles>
11 </xml>

Although there are many ways to parse XML documents, the library LIB_parse is used in Listing 3 to demonstrate how you can use the same technique to parse a variety of web information.

01 <?php
02 include('LIB_parse.php');                        // include the parsing code from library
03
04 $task      = return_between($raw_data, "<task>", "</task>", EXCL);
05 $target    = return_between($raw_data, "<target>", "</target>", EXCL);
06 $username  = return_between($raw_data, "<username>", "</username>", EXCL);
07 $password  = return_between($raw_data, "<password>", "</password>", EXCL);
08 $vin_array = parse_array($raw_data, "<vin>", "</vin>");
09 ?>

The example in Listing 3 shows two of the many useful functions in LIB_parse. The return_between() function simply returns data found between two endpoints. The EXCL passed as the last parameter in the function specifies that the returned sample is exclusive of the endpoints. If you also wanted the endpoints returned with the data, the example would have used INCL instead. The other function shown in Listing 3 is parse_array(). This function creates an easy-to-use array of all characters found between every instance of the two endpoints. In this case, those endpoints are the XML tags that encase the Vehicle Identification Numbers. In regular web pages, the parse_array() function might return table cells, hyperlinks, or all references to images, as shown in Listing 4.

01 <?php
02 include('LIB_parse.php');                       // include the parsing code from library
03
04 $td_array   = parse_array($raw_data, "<td", "</td>"); // array of table cells
05 $img_array  = parse_array($raw_data, "<img", ">");    // array of images
06 $link_array = parse_array($raw_data, "<a", "</a>");   // array of links
07 ?>

Form Submission

Another task required in the sniper botnet was the ability to submit a form. Form submission is how vehicles were purchased and payment and shipping arranged. Again, with the help of LIB_http, this was an easy task. There are actually two ways of submitting a form, depending on what the form handler is expecting. You could make a GET method request by tacking a query string on the end of the URL, or you could make the POST method request, as shown in Listing 5.

01 <?php
02 include('LIB_http.php');                        // include the parsing code from library
03
04 // initialize the form elements as an array
05 $form_array['session']  = "12301231#e0sds0122342";
06 $form_array['user']     = $username;
07 $form_array['password'] = $password;
08 $form_array['Login']    = "login";
09 $form_handler   = "http://www.WebbotsSpidersScreenScrapers.com/form_analyzer.php";
10 $referrer       = "";
11
12 $submission_result = http_post_form($form_handler, $referrer, $form_array);
13 $web_page          = $submission_result['FILE'];
14 ?>

Programming and Hosting Environments

I develop harvesters and the harvester servers in either a WAMP or LAMP environment, where Ubuntu is my Linux distribution of choice. (I have no particular technical preference for Ubuntu; it's mostly a matter of familiarity.) I've written bots in Perl, Java, Visual Basic, and even Tcl/Tk. For the last 12 years, however, I've programmed exclusively in PHP. In addition to the simplicity of the language, PHP interfaces well with libcurl [5], which makes downloading web pages easy. In addition to the open source tools I described earlier, the only other tool I use is a text editor, usually jEdit [6].

Over the years, I've deployed my projects in various production environments. Initially, my production servers were all based on FreeBSD. Later, I moved to cheap, hosted Linux environments, where the operative word is "cheap." My experience is that hosted solutions costing as little as UK£ 3.00 or US$ 5.99 a month are more than adequate for hosting harvester servers. The reason I can get by with minimal hosting is that because my botnets provide a very task-specific benefit, they have few human users. Lately, I've been enjoying the flexibility and cost advantages of cloud-based Linux solutions, primarily in Amazon's EC2, where Ubuntu is the standard distribution.

Benefits of the Botnet Architecture

The classic botnet architecture offers many benefits. The most obvious is that it is easy to scale the capacity of the architecture by simply adding harvesters. The prospect of hosting harvesters in a cloud environment is especially attractive for this reason alone. Multiple harvesters also provide opportunities for redundancy and let you use multiple IP addresses. As seen in the example of the sniper botnet, botnet technology leads to obvious competitive advantages. In the case of the example, my client was vastly more competitive than other automotive dealerships that didn't deploy a botnet.

You might be wondering whether applying botnets to business problems is fair. In the case of the example – a car-buying botnet – the alternative was to fill a room with sales and administrative personnel and instruct each of them to repeatedly refresh their browsers until the cars they were assigned were available for purchase. Using the human approach, the same amount of server resources (or more) were expended, but at a higher cost and with a poorer result for my client. The one difference is that my botnet approach used less labor.

With the botnet, the website that offered the cars still made their sales. In fact, the botnet might have even saved the seller money because the process was automated. As such, opportunities for error in payment or shipping were drastically reduced. For the year that project ran, the botnet never caused an error that cost the sales website any money or lost time.

The true danger in dismissing nontraditional ideas such as commercial botnets is the risk of discouraging developers from using their skills in innovative ways.

Living in the Shadows

Even when everything is legal and aboveboard, it is still advantageous for botnet developers not to show all their cards. The only reason I'm using an actual example in this article is because the application's natural life came to an end years ago, and I got specific permission from my client to write about this project; otherwise, I'm quite tight-lipped about the projects I develop. The reason to effectively "live in the shadows" is to hide any clues that I'm developing technology to solve a problem that everyone else handles traditionally. This policy is best because you never know who reads server logs, and you never know who knows your client's competitors.

Although many botnets depend on proxies to cloak their path, I think the best way to develop stealthy botnets is to simulate human activity as closely as possible. Ways to make a botnet look human include:

• inserting random delays;
• limiting interaction with the target website;
• using multiple user accounts (if possible or necessary); and
• running your botnet during peak usage periods.

The last point is often overlooked, but if you want your botnet to look like a person, it's best to run it when everyone else is using the targeted website. Running a botnet during peak periods will also make the log entries of your server less conspicuous.

Botnet Development as a Career Path

My life as a software developer got interesting once I started writing bots. My entry into this area came from an interest in network security, which quickly lead to my interest in automated web clients. Webbots were my ticket out of the cubical jungle, and I started a period of my life that produced a worn-out passport, publishing contracts, speaking opportunities around the world, and media appearances, including the BBC in the UK and California Public Radio in the US. Perhaps the biggest reward, though, is the expression I bring to my clients' faces after I solve a business problem using existing technologies in interesting and unexpected ways.