Screen scraping with Colly in Go

goquery: jQuery for Go

But oops, if you look closer, the span element contains not one but three lines. The first one, "8-12", shows the wave height I was searching for, and the second one contains <sup>FT</sup> to add feet as the unit of length next to it in the browser. The third line contains a "+", which indicates that the waves could be a bit higher than indicated.

How can the query processor separate these three lines? I didn't find anything helpful on this topic in the Colly documentation, but luckily Colly uses the goquery language internally, which is very similar to jQuery. Starting with a structure of HTMLElement type found in Colly, you can quickly extract to the corresponding goquery structure via its DOM attribute.

The corresponding goquery documentation [4] states that the Contents() function splits the text from the discovered element into its three components, and a subsequent call to Slice(0,1) then extracts the first bit.

As usual in Go, the index numbers for a slice always refer to the first (inclusive) and the last elements (exclusive). The following Each() in line 15 grabs the one and only result and calls the provided callback function with the selection at hand. The callback then uses s.Text() to extract the wave height in feet as a string, and we're in business.

High Waves

The following call to the compiled binary

$ ./surfline
8-12

thus reveals that the waves are 8 to 12 feet tall (or 2.4 to 3.6 meters if you prefer). That is a little beyond my capabilities, forcing me to leave my surfboard at home today for safety reasons and ensuring timely production of future columns. But tomorrow is another day!

Infos

  1. "Programming Snapshot: Protectli" by Mike Schilli, Linux Magazine, issue 208, March 2018, pp. 46-49, http://www.linux-magazine.com/Issues/2018/208/Servile-Guardian/(language)/eng-US
  2. Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/208/
  3. Colly documentation:http://go-colly.org/docs/
  4. goquery documentation: https://godoc.org/github.com/PuerkitoBio/goquery

The Author

Mike Schilli works as a software engineer in the San Francisco Bay area, California. Each month in his column, which has been running since 1997, he researches practical applications of various programming languages. If you email him at mailto:mschilli@perlmeister.com he will gladly answer any questions.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Plan Your Hike

    The hiking and cycling app komoot saves your traveled excursion routes. Mike Schilli shows you how to retrieve the data with Go.

  • At a Glance

    Using extensions in Go and Ruby, Mike Schilli adapts the WTF terminal dashboard tool to meet his personal needs.

  • Under the Hood

    Screen scrapers often fail when confronted with complex web pages. To keep his scraper on task, Mike Schilli remotely controls the Chrome browser using the DevTools protocol to extract data, even from highly dynamic web pages.

  • Simile

    The Simile project jump starts the semantic web with a collection of tools for extending semantic information to existing websites.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News