Command Line

Tutorial

Article from Issue 198/2017
Author(s):

Not only is it possible to browse the web from the CLI, it's mightily useful, too, as we demonstrate in this article.

A few months ago, I was stranded at an airport – my plane was delayed, so I had a couple of hours to kill. Fortunately, it was past beer-o-clock (i.e., 1800 hours), so I headed to the bar and then cracked open my laptop for some Reddit-browsing antics. But there was one big problem: The airport's Internet connection was slow.

I'm talking horrendously slow here – it was like using some 300-baud modem from the 1980s. I pretty much couldn't do anything on the Internet, and I saw other grumbling passengers desperately trying to reconnect multiple times in hope of a teensy bit of better bandwidth. They soon gave up.

Because I'm a turbo-geek, though, I came up with a solution. Somehow, I needed to drastically reduce my bandwidth but still be able to surf the web. I tried disabling images and JavaScript in Firefox; this resulted in a slight improvement, but browsing was still agonizingly slow. And then it struck me: I could SSH into a Raspberry Pi I have at home and browse from there.

Yes, thanks to a couple of rather excellent text-mode browsers, I was soon browsing Reddit threads and Wikipedia pages at lightning speed, despite my terrible connection. The actual job of browsing was being done on the Pi, so only the screen changes were being sent over the SSH connection to my laptop – and those required hardly any bandwidth. Because no images were involved, everything ran buttery smooth.

So, in this tutorial, we're going to look at some ways to navigate the web and grab content at the command line. This may be useful in scenarios such as the one I described; moreover, these techniques could come in useful if you're logged in to a remote server and need to do some browsing from there (e.g., to download something) or even want to watch a video (see the "YouTube from the Command Line" box for details). And, of course, browsing the web from the command line wins you oodles of geek points as well. So, let's get started.

YouTube from the Command Line

Ever seen a YouTube video that you'd like to keep permanently on your hard drive? Copyright issues aside, there are browser extensions that provide extra links – but often these extensions are bundled with adware or malware. A more elegant (and Linux-y) way is to use a command-line tool: youtube-dl.

This lets you specify a YouTube URL and then grab the video data for offline use; it also works with many other video-hosting websites, such as Dailymotion, Streamable, and LiveLeak. The program is available in the package repositories of most popular Linux distros – if you can't find it, though, or you want the very latest version, you can get it from the youtube-dl website [1].

Then, you can download a video by providing the URL, for example:

youtube-dl https://www.youtube.com/watch?v=rfl9KQb_HVk

However, you'll probably want some more control over the exact format of the file that you're downloading. To achieve this, add the --list-formats parameter:

youtube-dl --list-formats https://www.youtube.com/watch?v=rfl9KQb_HVk

With this, youtube-dl probes YouTube to find out which formats are available for the specified video and then lists them with numbers. You can then repeat the command, specifying the appropriate number after the -f parameter:

youtube-dl --f 32 https://www.youtube.com/watch?v=rfl9KQb_HVk

There are many other options in youtube-dl for logging into video sites before downloading, performing post-processing on audio or video (e.g., recoding in a different format), and much more. You can even create templates for output file names – that's especially useful if you want to create compilations of music, for instance. For full details on all the options, check out the program's documentation [2].

The Web in Your Terminal

There are many text-mode web browsers out there, and performance and layout differ significantly. No text-mode browser can compare to a graphical one in terms of advanced layout features, of course – so don't expect a web page's fancy CSS animations and floating div prettiness to shine through inside a terminal window. However, some browsers are much better than others. If you've been in the Unix world for a while, you've probably heard of the Lynx browser; this dates back to 1992, so it's almost as old as the World Wide Web itself. Lynx isn't bad, but there are much better alternatives – we're going to focus on two here.

The first is ELinks [3]. Grab it via your distro's package manager, or on a Debian-based distro, use:

sudo apt-get install elinks

Now you can simply start it in a terminal by entering the program's name followed by a website:

elinks http://elinks.or.cz

There you have it: the ELinks website, rendered in plain text, like in Figure 1. In the top-right corner of the window, you'll see the page title, and to navigate through links on the page, use the up and down cursor keys. You can see the address of the currently highlighted link in the status bar at the bottom; use Enter (or the right cursor key) to navigate to the link, and the left cursor key to go back in your history. Try it out by hitting the down cursor key a few times to select the "Linux From Scratch" link, and then press Enter to access it. Keep an eye on the status bar which shows you progress as a page is loading. Once you're done, hit left to return to the ELinks site.

Figure 1: ELinks on its first start – all in black and white. We can easily fix this, though.

Figure 1 shows ELinks in its default configuration, which is meant to provide basic browsing facilities for running on almost every Unix-like OS, on every terminal type, and over every connection. But we can make it a lot prettier. For starters, we can change the box at the top of the ELinks page from dashes and plus characters into a proper box using better text characters by going to the options in the menu.

But where is the menu? It's not visible by default, but hit F10 to bring it up. (If that doesn't work on your setup for some reason, press Esc followed by Enter). Use the cursor keys to go to the Setup menu and then Terminal options, and then hit Enter. A dialog box will pop up with a bunch of options: Use the down cursor keys and space to select VT 100 frames and 265 colors, and then keep going down to the OK button at the bottom and press Enter.

Suddenly, ELinks is a lot more colorful, and boxes are drawn properly as well. If you open up the menu again, you'll see that it's rendered better, too. Note that the settings aren't stored automatically – to do that, go to Setup | Save options.

Kicking the Tires

Now let's try ELinks with a slightly more complicated web page. To go to another URL, simply type g to open a dialog box and enter an address – try entering www.openbsd.org. You'll see the results like in Figure 2, and if you open that OpenBSD website in a regular graphical browser like Firefox, you'll see that ELinks does a pretty decent job of maintaining the layout. Sure, no images are displayed, but the site is more than usable if you just want to get textual information from it.

Figure 2: Here's ELinks after a bit of customization, showing the OpenBSD website. Pretty good rendering, for text mode!

From here, you can start exploring other websites to see how well they render. As mentioned, it's best to not expect too much, and sites that rely on complicated JavaScript will have problems. But you may be surprised at how well ELinks handles many text-oriented websites. If you just need to pluck some information from a wiki page, it's great.

ELinks has various features that you'd expect in graphical browsers as well: history, bookmarks, the ability to view a page's source code, and more. Check out the View and Tools menus for these – and if you want to download a link rather than visit it, press d when the link is highlighted. Finally, when you want to quit and return to the command line, hit q.

So that's ELinks – it's great, but it's always a good idea to have a couple of tools in your armory for tasks like this. There's another great text-mode browser that's not quite as user-friendly as ELinks (with its menus and dialog boxes), but actually renders some pages even better: w3m [4]. This was originally developed as a "pager" – that is, a tool for viewing HTML documents on demand, like when they're attached to an email, but it has developed into a dedicated browser in its own right.

To run it, install the w3m and, if it's available, the w3m-img package as well. On a Debian-based distro (e.g., on a Raspberry Pi running Raspbian), you can use this command:

sudo apt-get install w3m w3m-img

Now start it by providing a website:

w3m www.openbsd.org

Here you can see how w3m attempts to render the OpenBSD website, like in Figure 3. (Compare it with how ELinks renders the same site.) You'll note that in its default configuration, w3m is more colorful than ELinks, so you don't need to tweak any settings straightaway. Navigation is quite different, though: You can use the Tab key to move the cursor to the next link in a page or use Shift+Tab to move the cursor to the previous link.

Figure 3: This is w3m, showing the OpenBSD site. It's not quite as true to the original as with ELinks, but w3m is better for some other sites.

Alternatively, you can use your arrow keys to move the cursor around freely (or H, J, K, and L in vi-style), and once the cursor has landed on a link, tap Enter to follow that link. To go back in your browser history – that is, return to the previous page, hit B (note the capital letter). And to quit the browser, press Q.

To go to another website, press U, which opens up a prompt at the bottom of the screen. Simply type in the address and hit Enter. If you're reading a long page, you can use the space bar to jump down a page at a time – this is where w3m acts like a pager, similar to the good old less command.

W3m even supports tabbed browsing: Press Shift+T to open the current URL in a separate tab, and then use Shift along with the square bracket keys – [and ] – to navigate between tabs. To close a tab without terminating the whole browser, press Ctrl+q. (See the "w3m and Reddit" box for more options.)

w3m and Reddit

Reddit is (usually!) a good source for news and discussion about a zillion topics, and the free and open source communities are well represented too. W3m is especially good for browsing Reddit discussions (Figure 4), but if you go to a normal Reddit page (like www.reddit.com/r/linux/), you'll notice that the text-mode browser struggles to recreate the layout. Navigate to a comment link and you'll see that you have to scroll past lots of unrelated content before you can see the comments – and then the indentation of them, which is essential to following Reddit discussions, doesn't work properly either.

Figure 4: W3m is great for browsing Reddit discussions – just make sure you add ".mobile" to the end of the subreddit URL.

Fortunately, there's a solution to this. Reddit provides a few alternative views for older or more limited web browsers. Simply add .mobile on to the end of a subreddit, and Reddit will send back much simpler HTML – so for the Linux subreddit, go to www.reddit.com/r/linux/.mobile. Go to one of the comments links (e.g., 83 comments), then scroll down, and you'll see that it's much easier to follow discussions.

Reddit has its share of trolls and memes, but some of the discussions are stimulating, and if you're ever stuck in a super low-bandwidth situation, you can keep yourself busy for hours by browsing various subreddits.

Pretty as a Picture

OK, let's now take text-mode browsing to the max: We're going to display images. Yes, this is possible, and no, it ain't pretty. In fact, there aren't many reasons why you'd want to do this, unless your X server has gone down completely but you still need to look at cat pictures on the Internet – but it's fun to try out nonetheless. First of all, you need to grab a package called caca-utils – if you can't find it as a standalone package, try installing libcaca as well. This is an ASCII-art library – ASCII being plain text – which converts images into the appropriate numbers, letters, and colors for displaying inside a terminal window.

Now restart w3m and navigate to an image file. For this example, we're going to use a JPEG file [5]. W3m will automatically launch a viewer from the caca-utils package, and the result should look something like the screenshot in Figure 5. Can you make that out? It's supposed to be a dog, and the white block in the top-left is some text. If you squint you can just about make out that it's a dog – obviously, the text in the image is too low-definition, though.

Figure 5: We can even view images in text mode, like this dog from a meme pic. (Can you make it out?)

In the image viewer, try pressing D and A to turn on dithering and anti-aliasing; your mileage will vary though, depending on the image. If you really, absolutely must see an image in text mode, however, you can always reduce the font size in the terminal and then make the terminal much bigger, so that there are more rows and columns of characters. This provides you with more "pixels," thereby making the image (slightly) easier to recognize. To exit the image viewing and return to w3m, press q.

If you want to save an image in w3m rather than view it, press a – you'll then be prompted to confirm the filename. (Indeed, you can use the same keybinding to download any link to a file for storing locally.) You can then view the image like so:

cacaview <filename>

There are some other utilities in the caca-tools package that are worth playing around with as well. Try entering cacademo to see what other funky ASCII-art effects are available – use Esc to quit, and then try running it again for a different randomly-selected demo. Another program to try is cacafire, which as the name suggests creates a fire effect roaring up from the bottom of the terminal. It's quite impressive to see what's possible with mere text characters – indeed, the cacafire program makes a rather cool screen saver when you're accumulating geek points!

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Use Reddit on the Move with Diode Reddit Browser
  • Adriane

    The Adriane audio desktop system delivers Linux to sightimpaired users.

  • Ask Klaus!

    Klaus Knopper is the creator of Knoppix and co-founder of LinuxTag expo. He works as a teacher, programmer, and consultant. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: klaus@linux-magazine.com

  • Social Aid

    Fortunately, Facebook is not the only place you can chat: When sys admin columnist Charly Kühnast visits the web chat center Reddit, he often goes without a web browser and uses the RTV tool, which can even display photos and videos on plain text consoles.

  • FOSSPicks

    Graham looks at Krita 4.0, FreeTube 0.2.0, OpenSnitch, Yoda, Citybound, GZDoom, and much more!

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News