Automate your web logins

Script Limitations

The xdotool and xte utilities are great for simple web page automation when the HTML form items are sequential and no special decision making is required. Unfortunately, I found that when I tried to book a park time on the weekend, I started to see some limitations (Figure 4). During busy times, if I tried to book by time, xte and xdotool could not determine whether the time slot was taken. A simple workaround would be to search for the first Available or Not Busy slot, but this doesn't allow you to pick times you like.

Figure 4: Limitations with keystroke automation.

For projects that require some logic (like choosing a good time from a list of times), Selenium with Python is an excellent fit.

Selenium with Python

Selenium [4] is a portable framework for testing web applications, with client-server tools and an IDE. The Selenium WebDriver component (available for Firefox, Google Chrome, Internet Explorer, Safari, Opera, and Edge) sends commands from client APIs directly to a browser. Client APIs are available for C#, Go, Java, JavaScript, PHP, Python, and Ruby. The Selenium Downloads page [5] has details on installation of the WebDriver scripts.

To install the Linux 32-bit Selenium driver (geckodriver) for Firefox, enter:

tar -xvzf geckodriver-v0.24.0-linux32.tar.gz
chmod +x geckodriver
sudo mv geckodriver /usr/local/bin

To install the Selenium library for Python, enter:

pip install selenium

The big difference between the xte or xdotool utility and Selenium is that Selenium can access the HTML code of the selected web page directly.

Log In with Selenium and Python

As for xte and xdotool, you need to do some background manual work before writing the script. Once the required web page is open, you can use the Web Developer Inspector tool to examine HTML code. To access the Inspector, Select Tools | Web Developer | Inspector from the top menubar or use the shortcut Ctrl+Shift+C.

For the Netflix sign-in example, the Email or phone number and Password inputs are needed (Figure 5). When the Inspector is open, items selected on the web page are highlighted in the Inspector pane. In this example, the Email or phone number entry uses id="id_userLoginId", and the password entry uses id="id_password". Listing 3 shows the Python code that signs in to Netflix.

Listing 3

Netflix Sign-In with Selenium

01 #
02 # - automate Netflix Login
03 #
04 from selenium import webdriver
06 url=""
07 email=""
08 pwd="my_password"
10 browser = webdriver.Firefox()
12 browser.get(url)
14 # wait for page to refresh
15 browser.implicitly_wait(10)
17 username = browser.find_element_by_id('id_userLoginId')
18 username.send_keys(email)
20 password = browser.find_element_by_id('id_password')
21 password.send_keys(pwd)
23 password.submit()
25 print("Login Complete")
Figure 5: Finding input field IDs with Inspector.

When a web page is called, it's important to give the page some time to refresh. The implicity_wait(10) call (line 15) waits up to 10 seconds for a Selenium query to complete.

HTML items can be found by either ID (find_element_by_id()) or by name (find_element_by_name()). A Selenium object needs to be created before initiating any action on it. Line 17 finds and then creates a username object from ID 'id_userLoginId'. The send_keys() method is used to pass text strings to <input> tags (lines 18 and 21). Calling the submit() method on any input object will send all the form data as a request to the web server (line 23).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Perl: Web Regression Test

    Testing complex web applications does not necessarily mean investing in expensive, proprietary tools such as Test Director or Silk Performer. Selenium is for free; it can remotely control any major browser, and it is programmable in Perl.

  • Workspace: Text Expander

    A couple of utilities and a dash of Bash scripting are all you need to roll out a simple yet flexible text expander.

  • Tutorial – Devilspie2

    Stop battling your window manager to position things as you like – make scripts do all the hard work!

  • Kitchen Kiosk

    Create a kiosk display from an old eReader to show data culled from Home Automation, Raspberry Pis, and Arduinos.

  • Servile Guardian

    What is making the lights on the router flicker so excitedly? An intruder? We investigate with pfSense on a Protectli micro appliance and a screen scraper to email the information.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More