Watchful Seller

If I ever get around to writing my memoirs, I am going to include a lengthy chapter about my life's motto, which is: "Anything you don't constantly monitor is guaranteed to go belly up when you least expect it." True to this motto, I wrote a script this month to receive an email immediately when one of my eBay customers has left feedback about a transaction.

Simply Scraping

As a quick and dirty solution, and to first avoid having to register as a developer with eBay, I wrote a screen-scraper that extracts the current feedback score from the slew of HTML on the feedback page, before saving the counter and raising the alarm during subsequent runs if the value has increased.

When I click on my username on eBay.com, the browser lands on the feedback page in Figure 1. Selecting view-source in the browser shows the HTML code from Figure 2, and a text search for the string 362 (the current feedback score) shows that this number is found in a markup tag of the mbg-l class:

( 362<img src="...

Figure 1: The score for previously received customer comments is on eBay's feedback page – in this case, it is 362.

Figure 2: The feedback score is found in an HTML element of the mbg-l class: here, 362.

An XPath processor such as HTML::TreeBuilder::XPath can easily retrieve the content of this tag. The query

/html/body//span[@class="mbg-l"]

locates all span elements in the HTML body that have a class attribute with a value of mbg-l. The double slash in the expression indicates that the requested elements can exist at an arbitrary nesting depth beneath the HTML body tag.

The XPath query then spits out a string such as ( 362) (.... From there, with a regular expression in Perl, extracting the score is child's play. Listing 1 does exactly this in the feedback_fetch function in line 70, and then saves the score that was found in a cache file. Then, on the next run, it compares the value with the one obtained at that point, and fires off an email if the number has increased [1].

01 #!/usr/local/bin/perl -w
02 use strict;
03 use Sysadm::Install qw(:all);
04 use LWP::Simple ;
05 use HTML::TreeBuilder::XPath;
06 use Log::Log4perl qw(:easy);
07
08 my $nick      = "";
09 my $ebay_url  =
10   "http://feedback.ebay.com" .
11   "/ws/eBayISAPI.dll?ViewFeedback2" .
12   "&userid=$nick";
13
14 my( $home )   = glob "~";
15 my $data_dir  = "$home/logs";
16 my $cache     =
17   "$data_dir/ebay-feedback.cache";
18 my $log_file  =
19   "$data_dir/ebay-feedback.log";
20
21   # mail prefs
22 my $mailer    = "/usr/bin/mail";
23 my $mail_to   = '';
24
25 mkd $data_dir if !-d $data_dir;
26
27 Log::Log4perl->easy_init( {
28   level => $DEBUG,
29   file => ">>$log_file" } );
30
31 my $last_feedback;
32
33 if( -f $cache ) {
34   $last_feedback = slurp $cache;
35 }
36
37 my $feedback = feedback_fetch();
38
39 if( !defined $last_feedback or
40   $last_feedback != $feedback ) {
41
42   $last_feedback ||= 0;
43
44   INFO "New feedback: $feedback";
45   INFO "Sending mail to $mail_to";
46
47   open PIPE,
48     "| $mailer -s 'New Ebay Feedback: " .
49     "$feedback' $mail_to";
50
51   print PIPE <<EOT;
52 Ebay feedback changed: It's $feedback now
53 and was $last_feedback yesterday:
54
55     $ebay_url
56
57 Greetings!
58
59 Your faithful Ebay feedback scraper.
60 EOT
61   close PIPE;
62
63   blurt $feedback, $cache;
64
65 } else {
66   INFO "Feedback unchanged ($feedback).";
67 }
68
69 ###########################################
70 sub feedback_fetch {
71 ###########################################
72   INFO "Fetching $ebay_url";
73
74   my $content = get $ebay_url;
75
76   if( !defined $content ) {
77     ERROR "Fetching $ebay_url failed";
78     return undef;
79   }
80
81   my $tree= HTML::TreeBuilder::XPath->new;
82   $tree->parse( $content );
83
84   my( $text ) = $tree->findvalue(
85       '/html/body//span[@class="mbg-l"]');
86
87   if( $text =~ /\s*(\d+)/ ) {
88     return $1;
89   }
90
91   ERROR "Pattern in page not found";
92   return undef;
93 }

Email at Your Command

Writing and reading the cache data, which saves the feedback score confirmed during the last run as its only value, is handled by the slurp and blurt functions. These are from the CPAN Sysadm::Install module, which exports them in line 3. To retrieve the eBay page, the script uses the slimmed-down LWP::Simple module, whose get method simply runs an HTTP request for the specified URL and, if successful, returns the content of the page that was found.

On Linux, it is often very easy to send mail with the /usr/bin/mail utility; the CPAN Mail::DWIM module is recommended for those wanting more flexibility. Because I already replaced /usr/bin/mail on my Linux box at home with a Perl script that can cope with my ISP's requirements, I left things as is. Figure 3 shows the email that landed in my Gmail inbox.

Figure 3: A message informs the user about the newly arrived feedback from a buyer.

Before firing it up, the parameters in the script header still need to be tuned to match the local conditions. In line 8, the $nick variable receives the name assigned to the eBay user, whose feedback counter will be monitored by ebay-feedback. The URL is valid for US eBay accounts, whereas UK accounts run on ebay.co.uk instead. Finally, in the $mail_to variable, line 23 requires the email address of the user to whom the alerts should be sent.

Monitoring the Monitor

How can you monitor whether the script is still functioning, or whether eBay has changed its page layout? Listing 1 records all the operations for that purpose via Log::Log4perl in the log file ~/logs/ebay-feedback.log. Additionally, for instance, a daily Nagios script could trawl through the data that show up there based on a pattern such as "OK" under the current date, and sound the alarm if it is not found."